Title: Statistical and Computational Trade-off in Multi-Agent Multi-Armed Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-Agent Multi-Armed Bandits

Multi-Armed Bandits

Regret Minimization
Abstract: We study the problem of regret minimization in Multi-Agent Multi-Armed Bandits (MAMABs) where the rewards are defined through a factor graph. We derive an instance-specific regret lower bound and characterize the minimal expected number of times each global action should be explored. Unfortunately, this bound and the corresponding optimal exploration process are obtained by solving a combinatorial optimization problem with a set of variables and constraints exponentially growing with the number of agents. We approximate the regret lower bound problem via Mean Field techniques to reduce the number of variables and constraints. By tuning the latter, we explore the trade-off between achievable regret and complexity. We devise Efficient Sampling for MAMAB (ESM), an algorithm whose regret asymptotically matches the corresponding approximated lower bound. We assess the regret and computational complexity of ESM numerically, using both synthetic and real-world experiments in radio communications networks.

Statistical and Computational Trade-off in Multi-Agent Multi-Armed Bandits

Abstract

We study the problem of regret minimization in Multi-Agent Multi-Armed Bandits (MAMABs) where the rewards are defined through a factor graph. We derive an instance-specific regret lower bound and characterize the minimal expected number of times each global action should be explored. This bound and the corresponding optimal exploration process are obtained by solving a combinatorial optimization problem whose set of variables and constraints exponentially grow with the number of agents, and cannot be exploited in the design of efficient algorithms. Inspired by Mean Field approximation techniques used in graphical models, we provide simple upper bounds of the regret lower bound. The corresponding optimization problems have a reduced number of variables and constraints. By tuning the latter, we may explore the trade-off between the achievable regret and the complexity of computing the corresponding exploration process. We devise Efficient Sampling for MAMAB (ESM), an algorithm whose regret asymptotically matches the approximated lower bounds. The regret and computational complexity of ESM are assessed numerically, using both synthetic and real-world experiments in radio communications networks.

1 Introduction

The stochastic Multi-Agent Multi-Armed Bandits (MAMABs) [35, 2, 3] is a combinatorial sequential decision-making problem that generalizes the classical stochastic MAB problem by assuming that ( i ) a global action is defined by actions individually selected by a set of agents, and ( ii ) the reward function is defined through a factor graph, which defines inter-dependencies between agents. This reward structure arises naturally in applications where agents interact in a graph with the need to coordinate towards a common goal. MAMABs can model a wide range of real-world problems, from wind farm control [2, 37] to radio communication networks parameters optimization (see Fig. 1).

Despite the wide spectrum of their potential applications, MAMABs are extremely hard to solve, even when the reward function is known. The main challenge stems from the combinatorial structure of the action set (there are K N possible global actions, where N is the number of agents and K is the number of actions per agent). This issue is exacerbated in the learning setting where the reward function has to be inferred. In this work, we study the regret minimization problem in MAMABs, and more specifically, the trade-off between statistical efficiency (the learner aims at achieving low regret), and computational efficiency (she will typically have to solve combinatorial optimization problems over the set of possible global actions while learning).

Alexandre Protiuere KTH

Contributions. We present statistically and computationally efficient algorithms for MAMABs. Our algorithms enjoy (in the worst case) regret guarantees scaling as ρK d log( T ) , where K is the number of actions per agent, ρ and d are the number of factors and the maximal degree of the graph defining the reward function. This scaling illustrates the gains one may achieve by exploiting the factor graph structure: without leveraging it, the regret would scale as K N log( T ) . Our algorithms have controllable computational complexity and can be applied in large-scale MAMABs. More precisely, our contributions are as follows.

1) Regret lower bound. We derive a regret lower bound satisfied by any algorithm. The bound is defined through a convex program (the lower bound problem ), whose solution provides an optimal exploration strategy. Unfortunately, because of the factored reward structure, this optimization problem contains an exponential number of variables and constraints, and is hard to use in practice.

2) Approximations of the lower bound problem. We devise approximations of the lower bound problem by combining variable and constraint reduction techniques inspired by methods in the probabilistic graphical model literature [39, 20]. To reduce the number of variables, we propose ( i ) locally tree-like approximation, a tight relaxation for MAMAB instances described by acyclic factor graphs, and ( ii ) Mean Field (MF) approximation for general graphs. The MF approximation yields an upper bound of the regret lower bound, scaling as ρK d log( T ) (where T is the time horizon).

Both approximations yield lower bound problems with a polynomial number of variables and exponential number of constraints (in N ). To reduce the number of constraints, we propose a technique that leverages an ordering of the m smallest gaps and a Factored Constraint Reduction (FCR) method to represent the exponentially many constraints in a compact manner. The corresponding optimization problems have a reduced number of variables and constraints. By tuning the latter, we may explore the trade-off between the achievable regret and the complexity of computing the corresponding exploration process.

3) The ESM algorithm. Based on this approximation, we devise Efficient Sampling for MAMABs (ESM), an algorithm whose regret provably matches our approximated regret lower bound. The algorithm trades off statistical and computational complexity by performing exploration as prescribed by the solution of the approximated lower bound problem. We test the performance of ESM numerically on both synthetic experiments and learn to coordinate the antenna tilts in a radio communication network. In both sets of experiments, ESM can solve problems with a large number of global actions in a statistical and computationally efficient manner.

2 Related Work

Our work belongs to the framework of structured regret minimization in MABs, which encompasses a large variety of reward structures such as linear [21], unimodal [9], Lipschitz [11], etc. For general structured bandits, [8] propose Optimal Sampling for Structured Bandits (OSSB), a statistically optimal algorithm, i.e., matching the regret lower bound. The algorithm is computationally inefficient when applied to the MAMABs combinatorial structure. Our algorithm is inspired by OSSB, but relies on approximated lower bound problems to trade-off statistical and computational complexity.

Afew studies investigate MAMABs with the same factored reward structure as ours [35, 2, 37]. These works focus on devising algorithms with regret guarantees using methods based on, e.g., Upper Confidence Bound (UCB) [35, 2] or Thompson Sampling (TS) [37]. For example, Stranders et al. [35] propose HEIST, an UCB-type algorithm whose asymptotic regret scales as O ( K N ∆ max / ∆ min log( T )) , where ∆ min and ∆ max are the minimal and maximal gaps, respectively. The MAUCE algorithm from Bargiacchi et al., [2] improves over [35] yielding asymptotic regret O ( ρ 2 K d ∆ 2 max / ∆ 2 min log( T )) . Our worst approximation improves of a factor ∆ max w.r.t. this bound, a quantity that typically scales with ρK d (see App. M).

There is a large body of work [24, 10, 12, 13, 38] investigating regret minimization in the (linear) combinatorial semi-bandit feedback setting. Although our model can be interpreted as a particular instance of this setting (see App. E for details), the MAMAB combinatorial structure has never been explicitly considered in this context. The closest related work is [12], in which the authors study a regret lower bound problem with an exponentially large number of variables and constraints. They leverage [12, Assumption 6] to compactly represent the lower bound optimization problem and propose a gradient-based procedure to solve it in polynomial time. Unfortunately, for MAMABs, the

above-mentioned assumption only holds for rewards described by acyclic factor graphs (see App. M). We propose computationally efficient approximations valid for any factor graph while retaining statistical tightness in the case of acyclic factor graphs.

3 Problem Setting

We consider the generic MAMAB model with factored structure introduced in [2]. The model is defined by the tuple ⟨S , A , r ⟩ , where:

S = [ N ] ≜ { 1 , . . . , N } is a set of N agents;

r is the reward function mapping the global action to the collected reward.

A = × i ∈ [ N ] A i is a set of global actions, which is the Cartesian product over i of the set A i of actions available to the agent i . We assume w.l.o.g. that |A i | = K , for all i ∈ [ N ] , and define A ≜ |A| = K N ;

Rewards and their factor-graph representation. We model the collected rewards. There are ρ possibly overlapping groups of agents ( S e ) e ∈ [ ρ ] , with S e ⊆ S and |S e | = N e . The local reward generated by group e depends on group actions a e ≜ ( a i ) i ∈S e ∈ A e ≜ × i ∈S e A i only. More precisely, each time a e is selected, the collected local rewards are i.i.d. copies of a random variable r e ( a e ) ∼ N ( θ e ( a e ) , 1 / 2) . Rewards collected in various groups are independent. The global reward for action a is then r ( a ) = ∑ r ( a ) , a random variable with expectation θ ( a ) = ∑ θ ( a )

The number of possible group actions in group e is A e ≜ |A e | = K N e , and we define A ≜ ∑ e ∈ [ ρ ] A e . The reward function can be represented using a factor graph [39]. Factor graphs are bipartite graphs with two types of node: N action nodes , one for each agent, and ρ factor nodes , one for each group. An edge between a factor r e and an agent i exists if the action a i selected by the agent i is an input of r e : i ∈ S e . Fig. 1 shows an example of a factor graph modeling interference in a radio communication network.

e ∈ [ ρ ] e e e ∈ [ ρ ] e e . ˜

1

r

2

3

r

4

r

1

r

3

2

r

5

4

1

r

2

3

r

4

r

1

r

3

2

5

r

5

4

5

Sequential decision process. The decision maker sequentially selects global actions based on the history of previous observations and receives a set of samples of the local rewards associated to the various groups. Specifically, in each round t ≥ 1 , the decision maker selects a global action a t = ( a t, 1 , . . . , a t,N ) and observes the local rewards r t = ( r t, 1 , . . . , r t,ρ ) from each group. The global action a t +1 is selected based on the history of observations H t = ( a s , r s ) s ∈ [ t ] . This type of interaction is known as semi-bandit feedback.

Regret minimization. The goal is to devise an algorithm π = ( a t ) t ≥ 1 , i.e., a sequence of global actions a t ∈ A selected in each round t ≥ 1 , that minimizes the regret up to time T ≥ 1 , defined as

where a ⋆ θ ∈ arg max a ∈A θ ( a ) denotes the best global action . Throughout the paper, we assume that a ⋆ θ is unique and we use a ⋆ and a ⋆ θ interchangeably. We define the gap of a sub-optimal global action a by ∆( a ) = θ ( a ⋆ ) -θ ( a ) .

4 Regret Lower Bound

To derive instance-specific regret lower bounds, we restrict our attention to the class of uniformly good algorithms: An algorithm π is uniformly good if for any θ , ∀ α > 0 , we have that R π ( T ) = o ( T α ) .

Theorem 4.1. The regret of any uniformly good algorithm satisfies for any θ , lim inf T →∞ R π ( T ) log( T ) ≥ C ⋆ θ , where C ⋆ θ is the value of the following convex optimization problem

̸

̸

The proof of this result leverages classical change-of-measure arguments [25] (see App. A.1 for details). If v ⋆ denotes the solution of the lower bound optimization problem, then for a = a ⋆ θ , v ⋆ a log( T ) can be interpreted as the asymptotic expected number of times the sub-optimal action a is explored under a uniformly good algorithm minimizing regret. We conclude this section by reformulating (1) using group variables ˜ v . Introduce the marginal cone :

The set contains group variables ˜ v = (˜ v e ) e ∈ [ ρ ] where ˜ v e = (˜ v e,a e ) a e ∈A e .

Lemma 4.2. For any , is the value of the following convex optimization problem

̸

Again, if the solution of (2) is ˜ v ⋆ , then for any e ∈ [ ρ ] and a e ∈ A e , ˜ v ⋆ e,a e log( T ) can be interpreted as the asymptotic expected number of times the group action a e is selected under an optimal algorithm when it explores, i.e., when the global action a = a ⋆ θ .

5 Lower Bound Approximations

As suggested above, if we are able to solve (1) and hence obtain v ⋆ , the latter specifies the optimal exploration process. From there, we could devise an algorithm with minimal regret [8]. Unfortunately, solving (1) is an extremely hard task, even for relatively small problems. Indeed, the problem has K N variables and K N constraints, and using general-purpose solvers, e.g., based on the interiorpoint method, would require poly ( K N ) log(1 /ε ) floating-point operations [33]. To circumvent this difficulty, we present approximations of the lower bound problem with a reduced number of variables and constraints. We will then leverage these approximations to design efficient algorithms.

5.1 Variable reduction

To reduce the number of variables, we apply approximation techniques inspired by methods in the probabilistic graphical model literature [39]. In Sec. 5.1.1, we first propose a locally tree-like reduction, yielding an optimization problem whose value C L θ exactly matches the true lower bound C ⋆ θ for MAMABs with acyclic factor graphs (see App. J for a formal definition and examples). For graphs containing cycles however, we have C L θ < C ⋆ θ , and hence for those graphs, it is impossible to devise an algorithm based on this reduction (such an algorithm would lead to a regret C L θ log( T ) , which contradicts the lower bound).

Instead, for general graphs, we propose in Sec. 5.1.2 the ψ -mean-field reduction, an approximation based on a local decomposition inspired by Mean Field (MF) methods [39]. The ψ -mean field reduction leads to an optimization problem whose value C MF θ provably upper bounds C ⋆ θ , and hence that can be used to devise an algorithm with regret approaching C MF θ log( T ) .

̸

5.1.1 Locally tree-like reduction

This reduction imposes local consistency constraints between group variables ˜ v and local variables local variables w = ( w i ) i ∈ [ N ] , where w i = ( w i,a i ) a i ∈A i ∈ R K ≥ 0 . Define the local cone as:

where the notation a e ∼ a i means that the i th element of a e equals a i . The locally tree-like approximation, presented in the next lemma, is obtained by replacing ˜ V by ˜ V L in (2).

Lemma 5.1. For any θ with rewards described by an acyclic factor graph, we have that C ⋆ θ = C L θ , where C L θ is the value of the following convex optimization problem:

̸

The proof is presented in App. A.2. This approximation reduces the number of variables from K N to ˜ A + KN . The lemma states that, for acyclic factor graphs, the locally tree-like approximation (3) is tight, i.e., C L θ = C ⋆ θ . Unfortunately, for general graphs, we have that C L θ < C ⋆ θ (a direct consequence of [39, Prop. 4.1]), and hence it is impossible to devise algorithms based on this approximation.

5.1.2 ψ -Mean-Field reduction

Our ψ -MF reduction is loosely inspired by MF approximation methods in graphical models [39]. It consists in decomposing global variables v as a function ψ of the local variables w = ( w i ) i ∈ [ N ] . Specifically, the ψ -MF reduction introduces the following set of constraints: v a = ψ a ( w ) , ∀ a = a ⋆ θ , where ψ a : R KN ≥ 0 → R ≥ 0 . Let V ψ = { v ∈ R A ≥ 0 : ∃ w ∈ R KN ≥ 0 , v a = ψ a ( w ) , ∀ a = a ⋆ θ } , and define the ψ -MF marginal cone as

̸

̸

We get the ψ -MF approximation, C ψ -MF θ by replacing ˜ V by ˜ V ψ -MF in (2).

Lemma 5.2. For any θ , ψ , we have that C ⋆ θ ≤ C ψ -MF θ , where C ψ -MF θ is the value of the optimization problem:

̸

Clearly, the tractability of the problem C ψ -MF θ depends on the choice of ψ . A natural choice would be ψ a ( w ) = ∏ i ∈ [ N ] w i,a i , as proposed, e.g., in approximate inference methods [39]. However, this choice leads to a non-convex program (see App. B). The following lemma proposes a choice of ψ which leads to a convex program over local variables w only.

̸

Lemma 5.3. Let ψ a ( w ) = ∑ i ∈ [ N ] w i,a i , ∀ a = a ⋆ . Then C ψ -MF θ is the value of the following convex optimization problem:

̸

̸

where f e,a e ( w ) = K N -|S e | ∑ i ∈S e ,a i ∈A i w i,a i + K N -|S e |-1 ∑ i ̸∈S e w i,a i . Furthermore, it holds that C ψ -MF θ ≤ ρ ∆ -2 min ∑ e ∈ [ ρ ] ,a e ∈A e ( θ ( a ⋆ e ) -θ e ( a e )) , where ∆ min = min a = a ⋆ ∆( a ) .

The proof is presented in App. A.3. The lemma provides a worst-case scaling of C ψ -MF θ : it scales at most as ˜ A = ∑ e ∈ [ ρ ] K |S e | (remember that if we were considering a MAMAB as a standard bandit problem, the latter would have K N arm and hence a regret scaling exponentially in N ). The number of variables involved in (5) is KN . The quantities ( f e,a e ( w )) e ∈ [ ρ ] ,a e ∈A e are group quantities interpreted as the group variables ˜ v e,a e , and uniquely determined by local variables w . In the following, we use the notation C MF θ to represent C ψ -MF θ for the function ψ defined in Lemma 5.3.

5.2 Constraint reduction

The remaining challenge is to reduce the number of constraints in (3), (4) or (5). For each global action a , the constraint writes ∑ e ∈ [ ρ ] ˜ v -1 e,a e ≤ ∆( a ) 2 . The major issue is the non-linearity of the function appearing in the constraints w.r.t. group actions a e . Upon inspection, it appears that the heterogeneity in the gaps (generally ∆( a ) = ∆( b ) for a = b ) is causing the non-linearity. To address this problem, we present, in the following lemma, a family of approximations leveraging an ordering of the first m smallest gaps. For m ∈ [ K N ] , let a ( m ) be the m th best global action and, for m ∈ [ K N -1] , let ∆ m = θ ( a ⋆ θ ) -θ ( a ( m +1) ) be the m th minimal non-zero gap (with ties breaking arbitrarily).

̸

̸

Lemma 5.4. Let m ∈ [ K N -1] , and ⋄ ∈ { L , MF } . Let C ⋄ θ ( m ) be the value of the convex program:

̸

̸

Then, for any ⋄ ∈ { L , MF } , m ∈ [ K N -2] , we have C ⋄ θ ( m +1) ≤ C ⋄ θ ( m ) , C ⋆ θ ≤ C ⋄ θ ( m ) , and by definition C ⋄ θ ( K N -1) = C ⋄ θ .

The proof is reported in App. F. Clearly, C ⋄ θ ( m ) has still |A| constraints (7)-(8). However, as the gap ∆ m used in (8) is constant, these constraints are now a linear sum of terms depending on group actions a e . For constraints with this type of structure, there exists an efficient and provably equivalent representation. The procedure yielding this representation, which we refer to as FCR, is based on a generalization of the popular Factored LP algorithm described in [19, 20] for Factored Markov Decision Processes (FMDPs).

For the sake of brevity, we briefly describe the procedure below and postpone its detailed exposition to App. G. FCR is inspired by the Variable Elimination (VE) procedure in graphical models [14]. It iteratively eliminates constraints from (8), according to an elimination order O . The elimination procedure induces an elimination graph , which encodes dependencies between constraints as we perform elimination. As shown in the following lemma, the number of constraints is exponential in the degree A O of the elimination graph induced by the order of elimination O .

Lemma5.5. There exists a procedure which, given the constraints in (8) returns a provably equivalent constraint set of size O ( NK A O +1 ) .

Although for general graphs finding an ordering O minimizing A O is an NP -hard problem [14], for specific graphs there are orderings yielding A O ≪ N . For example, these orderings yield A O = 2 for line or star factor graphs, and A O = 3 for ring factor graphs, independently of the number of agents N (see Fig. 3 and refer to App. J for details). Solving C ⋄ θ ( m ) , requires computing the first m +1 best global actions and the m minimal gaps. To solve this task, the elimm -opt algorithm [15] has complexity O (( m +1) NK A O +1 ) (see App. G). Fig. 2 shows an illustration of the trade-off between statistical and computational complexity. Note that ESM is meant to be applied when m does not grow exponentially in N . In practice, we observed that selecting m = ˜ A yields a good trade-off between statistical complexity and computational complexity.

6 The ESM algorithm

In this section, we present ESM, an algorithm whose regret matches (asymptotically) our approximated lower bounds. The algorithm is inspired by OSSB [8]. It ensures that sub-optimal actions are sampled as prescribed by the solution of the approximated optimization problems C MF θ ( m ) or C L θ ( m ) : each group e must explore each group action a e for ˜ v e,a e log( T ) times, and selecting the action yielding the largest estimated reward for the remaining rounds. As the lower bounds depend on the unknown parameter θ , it has to be estimated.

O

(1)

O

(

KN

A

O

)

O

(

mNK

A

O

)

O

(

K

N

)

C

/star

θ

C

MF

θ

(

m

)

C

MF

θ

(1)

ρ

˜

A

∆

∆

2

min

Computational complexity

Statistical complexity

O

(1)

O

( ˜

ANK

A

O

)

O

(

K

N

)

C

/star

θ

C

MF

θ

( ˜

A

)

ρ

˜

A

∆

∆

2

min

(

m

= 1)

(

m

= ˜

A

)

Computational complexity

Statistical complexity

Generally, the estimation of θ would require evaluating an exponentially large number of components, i.e., θ ( a ) , ∀ a ∈ A . Instead, by leveraging the factored reward structure, we can simply focus on estimating group parameters ( θ e ) e ∈ [ ρ ] . We define the estimate at time t , group e , and action a e as ˆ θ t,e,a e = 1 N t,e,a e ∑ s ∈ [ t ]: a s,e = a e r s,e where N t,e,a e = ∑ s ∈ [ t ] ✶ { a s,e = a e } is the number of times action a e is selected for group e . We also define ˆ θ t = ( ˆ θ t,e,a e ) e ∈ [ ρ ] ,a e ∈A e , and N t = ( N t,e,a e ) e ∈ [ ρ ] ,a e ∈A e .

Algorithm 1 ESM( A 0 , ε, γ , ⋄ , m ) Sample each group actions in A 0 once and update ( N T 0 , ˆ θ T 0 ) ; s T 0 = 0 ----- ▷ Initialization for t = T 0 , . . . , T do ( (˜ v t,e ) e ∈ [ ρ ] ) ← Solve C ⋄ ˆ θ t ( m ) if N t,e,a e ≥ (1 + γ )˜ v t,e,a e log( t ) , ∀ e ∈ [ ρ ] , a e ∈ A e then ▷ Exploitation a t = a ⋆ ˆ θ t s t = s t - 1 else s t = s t - 1 +1 if min e ∈ [ ρ ] ,a e ∈A e N t,e,a e ≤ εs t then ▷ Estimation a t ∈ A 0 : a t,e ′ = b e ′ with ( e ′ , b e ′ ) ∈ arg min e,a e N t,e,a e else ▷ Exploration a t ∈ A : a t,e ′ = b e ′ with ( e ′ , b e ′ ) ∈ arg min e,a e N t,e,a e ˜ v t,e,a e Update ( N t,e,a t,e , ˆ θ t,e,a t,e ) e ∈ [ ρ ]

The pseudocode of ESM is presented in Alg. 1. It takes as inputs two exploration parameters ε, γ > 0 , an exploration set A 0 ⊂ A , the approximation parameter m ∈ [ K N -1] , and ⋄ ∈ { MF , L } depending on the targeted regret lower bound approximation. The parameters ε, γ > 0 impact the amount of exploration performed by ESM. When decreasing both these parameters the exploration of ESM also decreases. After an initialization phase, the algorithm alternates between three additional phases as described below.

Initialization. In the initialization phase, we select actions from A 0 to ensure that each group action is sampled at least once. The set A 0 ⊆ A is chosen in such a way that it covers all possible group actions, i.e., A 0 is such that ∀ e ∈ [ ρ ] , ∀ a e ∈ A e , ∃ b ∈ A 0 : b e = a e . In App. I, we present an efficient routine to select A 0 . Let T 0 = inf { t ≥ 0 : N t,e,a e > 0 , ∀ e ∈ [ ρ ] , a e ∈ A e } be the length of the initialization phase. For t ≤ T 0 we select a t ∈ A 0 : a t,e ′ = b e ′ with ( e ′ , b e ′ ) ∈ arg min e,a e N t,e,a e (with ties breaking arbitrarily), i.e., we select a global action containing the most under-explored group action. This choice ensures that T 0 ≤ ˜ A . For t > T 0 , the algorithm solves the approximated lower bound optimization problem C ⋄ ˆ θ t ( m ) and alternates between exploitation, exploration, and estimation.

Exploitation. If N t,e,a e ≥ (1 + γ )˜ v t,e,a e log( t ) , ESM enters the exploitation phase: it selects the best empirical action a ⋆ ˆ θ t = arg max a ∈A ∑ e ∈ [ ρ ] ˆ θ t,e,a e . Generally, computing a ⋆ ˆ θ t requires a max operation over an exponential number of actions a ∈ A . Fortunately, due to the factored structure, we can implement the max operation efficiently through a VE procedure [39] (see App. G).

Estimation. If not enough information has been gathered, ESM enters an estimation phase, where it selects the least explored group action similarly to the initialization phase. This ensures that the certainty equivalence holds, i.e., that ˆ θ t is estimated accurately.

Exploration. Otherwise, the algorithm enters the exploration phase and selects actions as suggested by the solution of C ⋄ ˆ θ t ( m ) . More precisely, we select a global action a t ∈ A which contains a group action a t,e ′ = b e ′ that minimizes the following ratio where e ′ and b e ′ are the group index and group action which minimize N t,e,a e ˜ v t,e,a e .

Upper bound. We establish that the ESM algorithm achieves a regret, matching the approximate lower bound C ⋄ θ ( m ) log( T ) , asymptotically as T →∞ . The proof is given in App. D.

Theorem 6.1. Let ε < 1 / |A 0 | . For any m ∈ [ K N -1] , we have that

lim sup T →∞ R π ( T ) log( T ) ≤ C MF θ ( m ) ξ ( ε, γ ) , for π = ESM( A 0 , ε, γ, MF , m ) , for any θ ,

lim sup T →∞ R π ( T ) log( T ) ≤ C L θ ( m ) ξ ( ε, γ ) , for π = ESM( A 0 , ε, γ, L , m ) , for any θ described by acyclic factor graphs,

where ξ is a function such that lim ( ε,γ ) → (0 , 0) ξ ( ε, γ ) = 1 .

7 Experiments

In this section, we present numerical experiments to assess the performance of our algorithm. We propose two sets of experiments: ( i ) a set of synthetic MAMABs with different graph topologies, and ( ii ) an industrial use-case from the radio communication domain: antenna tilt optimization . The code for the synthetic experiments and the additional experiments presented in App. K is available at this link.

7.1 Synthetic Experiments

Problem instances. We consider the factor graphs depicted in Fig. 3. The expected rewards are selected uniformly at random in the interval [0 , 10] . In our experiments, select N = 5 and K = 3 . We execute our experiments for N sim = 5 independent runs. Following previous work [8], we select γ = 0 , and ε = 0 . 01 . The elimination order is chosen as O = [ N ] . We implement the solver for the lower bound optimization problems using CVXPY [17] with a MOSEK solver [1].

1

2

N

N

2

r'1

T2

N

r1

N

3

r2

T2

1

3

2

3

N

r3

1

r4

r3

4

4

Results. The results for the regret (in log scale) are presented in Fig. 4. The performance of ESM is compared to that of MAUCE [3], HEIST [35], and to a random strategy selecting actions uniformly at random. The computational complexity results, reported in Fig. 5, measure the running time (in sec.) to solve an instance of the approximate lower bound optimization problem C ⋄ θ ( m ) . We use m = ˜ A , and ⋄ = MF for the ring, or ⋄ = L for the star and line graph topologies.

0

200

400

600

800

1000

T

10

1

10

2

10

3

10

4

R π

(

T

)

Ring factor graph

π

RAND

π

MAUCE

π

HEIST

π

ESM

(

m

= ˜

A

)

Lower Bound

0

200

400

600

800

1000

T

10

1

10

2

10

3

10

4

R π

(

T

)

Line factor graph

0

200

400

600

800

1000

T

10

1

10

2

10

3

10

4

R π

(

T

)

Star factor graph

50

100

150

200

m

10

1

10

2

C MF

θ

(

m

)

Ring factor graph

50

100

150

200

m

10

2

C L

θ

(

m

)

Line factor graph

50

100

150

200

m

10

2

10

3

C L

θ

(

m

)

Star factor graph

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

7.2 Antenna tilt optimization

Next, we test our algorithm on a radio network optimization task. The goal is to control the vertical antenna tilt at different network Base Stations to optimize the network throughput. In the following, we detail the network model, our simulation setup, and present our experimental results.

Network model. We consider a sectorized radio network consisting of a set of sectors S = [ N ] . The set of sectors corresponds to the set of agents in our MAMAB framework. Since each sector is associated to a unique antenna, we will use the terms sector and antenna interchangeably. We assume that each sector i ∈ S serves (on the downlink) a fixed set of Users Equipments (UEs) U i (each UE is associated with a unique antenna, that from which it receives the strongest signal).

Factor graph. We model the observed reward in the radio network as a factor graph with N = |S| agent nodes and ρ = |S| factor nodes. Each sector is associated with a unique factor, which models the rewards observed in that sector. We build the factor graph based on the interference pattern of the antennas, i.e., antennas that can interfere with each other are connected to common factors. An example of such a graph and additional experimental details are reported in App. L.

Actions and rewards. The action a t,i represents the antenna tilt for sector i ∈ S and at time t . It is chosen from a discrete set of K tilts, i.e., a t,i ∈ { α 1 , . . . , α K } . The tilt for a group of sectors e is denoted by a e . Rewards are based on the throughput of UEs in sector i , which depends on the actions of a group of agents a e : r e ( a e ) = ∑ u ∈U i T i,u ( a e ) , where T i,u is the throughput of an UE u associated to sector i . Hence, the global reward for a tilt configuration a ∈ A is r ( a ) = ∑ i ∈ [ N ] ∑ u ∈U i T i,u ( a e ) . The throughput T i,u depends on channel conditions (or fading ) between the antenna and the user. These conditions rapidly evolve over time around their mean.

Simulator. We run our experiments in a proprietary mobile network simulator in an urban environment. The simulation parameters used in our experiments are reported in App. K. Based on the user positions and network parameters, the simulator computes the path loss in the network environment using a BEZT propagation model [32] and returns the throughput for each sector by conducting user association and resource allocation in a full-buffer traffic demand scenario.

Results. We test our algorithm for A i = { 2 ◦ , 7 ◦ , 13 ◦ } , and for |S| = 6 sectors. As the factor graph contains cycles, we use ⋄ = MF and select m = 3 . The results, presented in Fig. 6, are in line with the experimental findings of the previous section. However, the ESM running time is higher due to the higher complexity of the factor graph.

0

500

1000

1500

2000

2500

3000

T

10

2

10

3

10

4

10

5

R

π

(

T

)

π

RAND

π

ESM

(

m

= 3)

π

MAUCE

Lower Bound

(i) Regret (log-scale).

50

100

150

200

250

300

350

m

10

2

2

×

10

2

C MF

θ

(

m

)

2

3

4

5

6

7

8

9

(ii) Running time (in sec.).

8 Conclusions

In this paper, we investigated the problem of regret minimization in MAMABs: we derived a regret lower bound, proposed approximations of it, and devised ESM, an algorithm trading off statistical and computational efficiency. We then assessed the performance of ESM on both synthetic examples and the antenna tilt optimization problem. Interesting future research directions include proposing efficient distributed implementations of ESM, quantifying on its communication complexity, and investigating representation learning problems in MAMABs where the underlying factor graph defining the reward is unknown and needs to be learned.

References

MOSEK ApS. The MOSEK optimization toolbox for MATLAB manual. Version 9.0. 2019.

Eugenio Bargiacchi et al. 'Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems'. In: Proc. of ICML . 2018.

Eugenio Bargiacchi et al. 'Multi-agent RMax for Multi-Agent Multi-Armed Bandits'. In: Proc. of Adaptive and Learning Agents Worksh. 2022.

Stephen Boyd and Lieven Vandenberghe. Convex Optimization . Cambridge University Press, 2004.

Stephen Boyd et al. A tutorial on geometric programming . Optimization and Engineering, 2004.

Shouyuan Chen et al. 'Combinatorial Pure Exploration of Multi-Armed Bandits'. In: Proc. of NeurIPS . 2014.

Wei Chen, Yajun Wang, and Yang Yuan. 'Combinatorial Multi-Armed Bandit: General Framework and Applications'. In: Proc. of Mach. Learn. Res. 2013.

Richard Combes, Stefan Magureanu, and Alexandre Proutiere. 'Minimal Exploration in Structured Stochastic Bandits'. In: Proc. of NeurIPS . 2017.

Richard Combes and Alexandre Proutiere. 'Unimodal bandits: Regret lower bounds and optimal algorithms'. In: Proc. of ICML . 2014.

Richard Combes et al. 'Combinatorial Bandits Revisited'. In: Proc. of NeurIPS . 2015.

Richard Combes et al. 'Learning to rank: Regret lower bounds and efficient algorithms'. In: Proc. of SIGMETRICS . 2015.

Thibaut Cuvelier, Richard Combes, and Eric Gourdin. 'Asymptotically Optimal Strategies For Combinatorial Semi-Bandits in Polynomial Time'. In: Proc. of ALT . 2021.

Thibaut Cuvelier, Richard Combes, and Eric Gourdin. 'Statistically Efficient, PolynomialTime Algorithms for Combinatorial Semi-Bandits'. In: Proc. ACM Meas. Anal. Comput. Syst. (2021).

Rina Dechter. 'Bucket Elimination: A Unifying Framework for Reasoning'. In: Artif. Intell. (1999).

Rina Dechter, Natalia Flerova, and Radu Marinescu. 'Search Algorithms for M Best Solutions for Graphical Models'. In: AAAI Press, 2012.

Rémy Degenne and Vianney Perchet. 'Combinatorial semi-bandit with known covariance'. In: Proc. of NeurIPS . 2016.

Steven Diamond and Stephen Boyd. 'CVXPY: A Python-embedded modeling language for convex optimization'. In: JMLR (2016).

Todd L Graves and Tze Leung Lai. 'Asymptotically efficient adaptive choice of control laws incontrolled markov chains'. In: SIAM journal on control and optimization (1997).

Carlos Guestrin, Daphne Koller, and Ronald Parr. 'Max-norm Projections for Factored MDPs'. In: Proc. of IJCAI . 2001.

Carlos Guestrin et al. 'Efficient Solution Algorithms for Factored MDPs'. In: J. Artif. Int. Res. (2003).

Botao Hao, Tor Lattimore, and Csaba Szepesvári. 'Adaptive Exploration in Linear Contextual Bandit'. In: Proc. of AISTATS . 2020.

Yassir Jedra and Alexandre Proutiere. 'Optimal Best-arm Identification in Linear Bandits'. In: Proc. of NeurIPS . 2020.

Emilie Kaufmann. 'Contributions to the Optimal Solution of Several Bandit Problems'. Habilitation à diriger des recherches. Université de Lille, Nov. 2020.

Branislav Kveton et al. 'Tight Regret Bounds for Stochastic Combinatorial Semi-Bandits'. In: PLMR . Proceedings of Machine Learning Research. PMLR, 2015.

Tze Leung Lai and Herbert Robbins. 'Asymptotically efficient adaptive allocation rules'. In: Advances in applied mathematics (1985).

E.L. Lawler. 'A procedure for computing the k best solutions to discrete optimization problems and its application to the shortest path problem'. In: Management Science (1972).

Marloes Maathuis et al. Handbook of Graphical Models . CRC Press, Inc., 2018.

Stefan Magureanu, Richard Combes, and Alexandre Proutiere. 'Lipschitz Bandits: Regret Lower Bound and Optimal Algorithms'. In: Proc. of COLT . 2014.

P.J. Modi. 'Distributed constraint optimization for multiagent'. PhD thesis. University of Southern California, 2003.

D. Nilsson. 'An Efficient Algorithm for Finding the M Most Probable Configurationsin Probabilistic Expert Systems'. In: Statistics and Computing (1998).

Pierre Perrault et al. 'Statistical Efficiency of Thompson Sampling for Combinatorial SemiBandits'. In: Proc. of NeurIPS . 2020.

Theodore Rappaport. Wireless Communications: Principles and Practice . Prentice Hall, 2001.

James Renegar. 'A polynomial-time algorithm, based on Newton's method, for linear programming'. In: Mathematical Programming (1988).

David Sontag and Tommi Jaakkola. 'On Iteratively Constraining the Marginal Polytope for Approximate Inference and MAP'. In: Semantic Scholar (2007).

Ruben Stranders et al. 'DCOPs and bandits: exploration and exploitation in decentralised coordination'. In: Proc. of AAMAS . 2012.

Filippo Vannella, Alexandre Proutiere, and Jaeseong Jeong. 'Best Arm Identification in MultiAgent Multi-Armed Bandits'. In: Proc. of ICML . 2023.

Timothy Verstraeten et al. 'Multi-Agent Thompson Sampling for Bandit Applications with Sparse Neighbourhood Structures'. In: Scientific Reports (2020).

Andrew Wagenmaker, Julian Katz-Samuels, and Kevin Jamieson. Experimental Design for Regret Minimization in Linear Bandits . 2020.

Martin Wainwright and Michael Jordan. Graphical Models, Exponential Families, and Variational Inference . Now Publishers Inc., 2008.

Siwei Wang and Wei Chen. 'Thompson Sampling for Combinatorial Semi-Bandits'. In: Proc. of ICML . 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Multi-Agent Multi-Armed Bandits problem with factor graph reward structure, which is motivated by the real-world antenna tilt optimization problem. It first proposes an asymptotic lower bound that involves an optimization problem with exponential number of variables and constraints. Then, this optimization problem is relaxed by approximation techniques motivated from probabilistic graphical model. Finally, this paper proposes an learning algorithm based on the approximated lower bound and supports it with both synthetic and real-world experiments.\n\nSTRENGTHS:\n- The relaxation contains novel application of various techniques from probabilistic graphical model to this concrete problem.\n- The proposed algorithm can be computationally efficient and is flexible to lie at different points on the trade-off curve between statistical and computational complexity.\n- The proposed algorithm enjoys good asymptotic regret and the theoretical guarantee is corroborated with real-world experiment.\n\nWEAKNESSES:\nAlthough I don't view this as a serious issue, it seems this paper focuses a lot in how to solve the problem in a computationally efficient way while the analysis of the algorithm's performance is a little bit limited. For example, we do not know how far the approximation is away from the exact lower bound and how the algorithm with locally tree-like reduction will perform in general.\n\n### Suggestions on Writing\n- It may be better to briefly mention the rough magnitude of $C^{\\diamond}_{\\theta}(m)$ again after stating Theorem 6.1.\n\nQUESTIONS:\n- What is the intuition behind using $\\min_{a_i\\in\\mathcal{A}\\_i}\\frac{N_{t, i, a_i}}{w_{t, i, a_i}}$ as the action selection for exploration? In particular, how does this selection rule approximately satisfies the lower bound interpretation that action $a$ should expectedly be selected about $v^\\star_a\\log(T)$ times?\n- It seems the varialbes $w=(w_i)\\_{i\\in[N]}$ is undefined for $C^{\\diamond}_{\\hat{\\theta}_t}(m)$ when $\\diamond=\\mathrm{L}$. How will the ESM algorithm proceed when $\\diamond=\\mathrm{L}$?\n- Is that possible to analyze how the algorithm will perform when $\\diamond=\\mathrm{L}$ and the factor graph is not an acyclic factor graph? (It certainly cannot be better than the lower bound.) Also, can we see how it will perform empirically?\n- Is that possible to quantify the difference between $C^\\star_{\\theta}$ and $C^{\\diamond}_{\\theta}(m)$? That is, how far is the approximation away from the exact lower bound?\n- In the antenna tilt experiment, does the choice of action set $\\mathcal{A}_i=\\lbrace 2^\\circ, 7^\\circ, 13^\\circ\\rbrace$ contain any special considerations (why these three degrees)?\n\nLIMITATIONS:\nThe limitations are addressed well in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the ESM algorithm for the regret minimization problem in Multi-Agent Multi-Armed Bandits, where the rewards are defined through a factor graph. The ESM algorithm attains a dedicate trade-off between statistical efficiency and computational efficiency in n Multi-Agent Multi-Armed Bandits.  The ESM algorithm is inspired by simple upper bounds of the regret lower bound, which characterizes the minimal expected number of times each global action should be explored and is computationally intractable to be exploited in the design of efficient algorithms. By tuning this upper bound, the ESM explores the trade-off between the achievable regret and the complexity of computing the corresponding exploration process. The ESM has a regret that asymptotically matches this upper bound. The regret and computational complexity of ESM are assessed numerically, using both synthetic and real-world experiments in radio communications networks.\n\nSTRENGTHS:\nThis paper establishes a lower bound for the Multi-Agent Multi-Armed Bandits, where the rewards are defined through a factor graph. \n\nSimple upper bound of the regret lower bound is established, which enables the design ESM. \n\nAsymptotic regret upper bound of ESM is derived.\n\nWEAKNESSES:\nThe regret lower bound looks not new and the proofs does not contribute new techniques. They look like straightforward applications of techniques from structured bandits.  \n\nThough upper bounds of the regret lower bound are established, the tightness of the upper bound is not proved. It is unknown how tight the upper bound is. Thus, it is unknown how much statistical efficiency is lost.  \n\nThe regret upper bound of ESM is asymptotic, which looks much weak than the finite round regret upper bound. \n\nThe gap between the regret upper bound of ESM and the regret lower bound is not quantified. It is unknown how tight the regret upper bound is.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nSee comments above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper focuses on an interesting problem, which is a version of multi-armed bandits in which there are multiple agents and an action needs to be selected for each one of them. This is a problem that has been studied before in the literature and the authors here provide improved bounds on the regret by approximately a factor of k^d, where k is the number of agents and the degree of a graph that defines the reward function. Moreover, the authors' algorithm gives a knob that controls the tradeoff between the regret and the time spent solving the multi-agent optimization problem of each round.

The authors finally provide an experimental evaluation of their algorithm on a synthetic and a real dataset. The plots show that the regret is low and improves over a simple baseline and over previous work.

STRENGTHS:
- The paper improves over the state of the art in theoretical guarantees
- The paper is well written, easy to follow
- The experimental evaluation shows good results

WEAKNESSES:
- The theoretical results are shown to be an improvement against [2] and [36] but the experimental part does not compare against [2] and [36].
- The related work pointers seem incomplete. The vast literature of combinatorial bandits seems to be very related (again combinatorial problems need to be solved in each round) but is not mentioned.

QUESTIONS:
- Why do you not compare against [2] and [36] in the experiments and rather choose weaker baselines?
- How does your problem compare against combinatorial bandits and why can't techniques from there be used in your setting?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper consider the regret minimization problem in MAMABs, which typically has an exponentially large action space in relation to the number of agents. The authors first establish a lower bound result for general MAMAB by generalizing techniques from single agent bandit literature. More precisely, the lower bound is obtained by solving a convex optimization problem with exponentially many variables and constraints. Next, the authors present an equivalent characterization of the program for reward structures with acyclic factor graphs and provide mean-field approximation programs for general graph models. Finally, the authors design computationally tractable algorithms based on the proposed equivalent(approximation) program for acyclic factor graphs(general graphs).\n\nSTRENGTHS:\n1. Most writing is clear and easy to follow, even for those without extensive knowledge in MAMABs. Most claims and definitions are well-supported by existing literature.\n\n2. Both the proposed lower bound and the proposed approximation scheme are interesting.\n\n3. Authors also provide simulations and real-world applications to further enhances the relevance and practicality of the proposed methods.\n\nWEAKNESSES:\nWhile the paper overall is well-written and informative, I found some parts of the presentation to be unclear. Specifically, there were several points where I had questions and would have appreciated more explanation or clarification.\n\nQUESTIONS:\n1. What is the trade-off when letting $\\varepsilon, \\gamma\\to 0$ in Algorithm 1? It appears that Theorem 6.1 always encourages selecting small values for $\\varepsilon,\\gamma$\n\n2. In the acyclic factor graphs setting, while the proposed characterization is statistically tight, what is the computational complexity of the proposed algorithm to achieve optimal regret(i.e. select $m = K^N - 1$)?  Will it achieve the same polynomial-time complexity as in [11] in this setting?\n\n3. In line 155, authors said \"Unfortunately, for general graphs, we have that $C^L_\\theta < C_\\theta^\\star$ (a direct consequence\n156 of [36, Prop. 4.1]), and hence it is impossible to devise algorithms based on this approximation.\" \nI am wondering why it is impossible to propose an approximation algorithm based on the  $C^L_\\theta$ value (since the proposed approximation program in the mean-field approximation is also not guaranteed to have a value equal to $C_\\theta^\\star$)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}