SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {e \in [ \rho ]: a _ {e} \neq a _ {e} ^ {\star}} \tilde {v} _ {e, a _ {e}} ^ {- 1} \leq \Delta_ {m} ^ {2}, \quad \forall a \in \mathcal {A} \backslash \cup_ {j \in [ m ]} \left\{a ^ {(j + 1)} \right\}. \tag {8}
$$
Then, for any  $\diamondsuit \in \{\mathrm{L},\mathrm{MF}\}$ ,  $m \in [K^N - 2]$ , we have  $C_{\theta}^{\diamond}(m + 1) \leq C_{\theta}^{\diamond}(m)$ ,  $C_{\theta}^{\star} \leq C_{\theta}^{\diamond}(m)$ , and by definition  $C_{\theta}^{\diamond}(K^{N} - 1) = C_{\theta}^{\diamond}$ .
The proof is reported in App. F. Clearly,  $C_{\theta}^{\diamond}(m)$  has still  $|\mathcal{A}|$  constraints (7)-(8). However, as the gap  $\Delta_m$  used in (8) is constant, these constraints are now a linear sum of terms depending on group actions  $a_e$ . For constraints with this type of structure, there exists an efficient and provably equivalent representation. The procedure yielding this representation, which we refer to as FCR, is based on a generalization of the popular Factored LP algorithm described in [19, 20] for Factored Markov Decision Processes (FMDPs).
For the sake of brevity, we briefly describe the procedure below and postpone its detailed exposition to App. G. FCR is inspired by the Variable Elimination (VE) procedure in graphical models [14]. It iteratively eliminates constraints from (8), according to an elimination order  $\mathcal{O}$ . The elimination procedure induces an elimination graph, which encodes dependencies between constraints as we perform elimination. As shown in the following lemma, the number of constraints is exponential in the degree  $A_{\mathcal{O}}$  of the elimination graph induced by the order of elimination  $\mathcal{O}$ .
Lemma 5.5. There exists a procedure which, given the constraints in (8) returns a provably equivalent constraint set of size  $O(NK^{A\circ +1})$ .
Although for general graphs finding an ordering  $\mathcal{O}$  minimizing  $A_{\mathcal{O}}$  is an  $\mathcal{NP}$ -hard problem [14], for specific graphs there are orderings yielding  $A_{\mathcal{O}} \ll N$ . For example, these orderings yield  $A_{\mathcal{O}} = 2$  for line or star factor graphs, and  $A_{\mathcal{O}} = 3$  for ring factor graphs, independently of the number of agents  $N$  (see Fig. 3 and refer to App. J for details). Solving  $C_{\theta}^{\diamond}(m)$ , requires computing the first  $m + 1$  best global actions and the  $m$  minimal gaps. To solve this task, the elim- $m$ -opt algorithm [15] has complexity  $O((m + 1)NK^{A_{\mathcal{O}} + 1})$  (see App. G). Fig. 2 shows an illustration of the trade-off between statistical and computational complexity. Note that ESM is meant to be applied when  $m$  does not grow exponentially in  $N$ . In practice, we observed that selecting  $m = \tilde{A}$  yields a good trade-off between statistical complexity and computational complexity.
6 The ESM algorithm
In this section, we present ESM, an algorithm whose regret matches (asymptotically) our approximated lower bounds. The algorithm is inspired by OSSB [8]. It ensures that sub-optimal actions are sampled as prescribed by the solution of the approximated optimization problems  $C_{\theta}^{\mathrm{MF}}(m)$  or  $C_{\theta}^{\mathrm{L}}(m)$ : each group  $e$  must explore each group action  $a_{e}$  for  $\tilde{v}_{e,a_e}\log (T)$  times, and selecting the action yielding the largest estimated reward for the remaining rounds. As the lower bounds depend on the unknown parameter  $\theta$ , it has to be estimated.
Figure 2: Left: idealized curve illustrating possible ranges of the trade-off between statistical and computational complexity for  $C^{\mathrm{MF}}(m)$ , when varying  $m$ . Right: an instance of this trade-off for a line factor graph (darker colors for the points represent higher running times). Selecting  $m = \tilde{A}$  yields a good trade-off between computational and statistical complexity for this instance.
Generally, the estimation of  $\theta$  would require evaluating an exponentially large number of components, i.e.,  $\theta(a), \forall a \in \mathcal{A}$ . Instead, by leveraging the factored reward structure, we can simply focus on estimating group parameters  $(\theta_e)_{e \in [\rho]}$ . We define the estimate at time  $t$ , group  $e$ , and action  $a_e$  as  $\hat{\theta}_{t,e,a_e} = \frac{1}{N_{t,e,a_e}} \sum_{s \in [t]: a_s, e = a_e} r_{s,e}$  where  $N_{t,e,a_e} = \sum_{s \in [t]} \mathbb{1}_{\{a_s, e = a_e\}}$  is the number of times action  $a_e$  is selected for group  $e$ . We also define  $\hat{\theta}_t = (\hat{\theta}_{t,e,a_e})_{e \in [\rho], a_e \in \mathcal{A}_e}$ , and  $N_t = (N_{t,e,a_e})_{e \in [\rho], a_e \in \mathcal{A}_e}$ .
Algorithm 1 ESM(A0,ε,γ,◇,m)
Sample each group actions in  $\mathcal{A}_0$  once and update  $(N_{T_0},\hat{\theta}_{T_0})$ $s_{T_0} = 0$  Initialization for  $t = T_0,\dots ,T$  do  $(\tilde{v}_{t,e})_{e\in [\rho ]}\gets S o l v e C_{\hat{\theta}_t}^\diamond (m)$  if  $N_{t,e,a_e}\geq (1 + \gamma)\tilde{v}_{t,e,a_e}\log (t),\forall e\in [\rho ],a_e\in \mathcal{A}_e$  then Exploitation else  $s_t = s_{t - 1} + 1$  if  $\min_{e\in [\rho ],a_e\in \mathcal{A}_e}N_{t,e,a_e}\leq \varepsilon s_t$  then Estimation  $a_{t}\in \mathcal{A}_{0}:a_{t,e^{\prime}} = b_{e^{\prime}}$  with  $(e^{\prime},b_{e^{\prime}})\in \arg \min_{e,a_e}N_{t,e,a_e}$  else Exploration  $a_{t}\in \mathcal{A}:a_{t,e^{\prime}} = b_{e^{\prime}}$  with  $(e^{\prime},b_{e^{\prime}})\in \arg \min_{e,a_e}\frac{N_{t,e,a_e}}{\tilde{v}_{t,e,a_e}}$  Update  $(N_{t,e,a_t,e},\hat{\theta}_{t,e,a_t,e})_{e\in [\rho ]}$
The pseudocode of ESM is presented in Alg. 1. It takes as inputs two exploration parameters  $\varepsilon, \gamma > 0$ , an exploration set  $\mathcal{A}_0 \subset \mathcal{A}$ , the approximation parameter  $m \in [K^N - 1]$ , and  $\diamond \in \{\mathrm{MF}, \mathrm{L}\}$  depending on the targeted regret lower bound approximation. The parameters  $\varepsilon, \gamma > 0$  impact the amount of exploration performed by ESM. When decreasing both these parameters the exploration of ESM also decreases. After an initialization phase, the algorithm alternates between three additional phases as described below.
Initialization. In the initialization phase, we select actions from  $\mathcal{A}_0$  to ensure that each group action is sampled at least once. The set  $\mathcal{A}_0 \subseteq \mathcal{A}$  is chosen in such a way that it covers all possible group actions, i.e.,  $\mathcal{A}_0$  is such that  $\forall e \in [\rho], \forall a_e \in \mathcal{A}_e, \exists b \in \mathcal{A}_0: b_e = a_e$ . In App. I, we present an efficient routine to select  $\mathcal{A}_0$ . Let  $T_0 = \inf \left\{t \geq 0 : N_{t,e,a_e} > 0, \forall e \in [\rho], a_e \in \mathcal{A}_e\right\}$  be the length of the initialization phase. For  $t \leq T_0$  we select  $a_t \in \mathcal{A}_0 : a_{t,e'} = b_{e'}$  with  $(e', b_{e'}) \in \arg \min_{e,a_e} N_{t,e,a_e}$  (with ties breaking arbitrarily), i.e., we select a global action containing the most under-explored group action. This choice ensures that  $T_0 \leq \tilde{A}$ . For  $t > T_0$ , the algorithm solves the approximated lower bound optimization problem  $C_{\hat{\theta}_t}^\diamond(m)$  and alternates between exploitation, exploration, and estimation.
Exploitation. If  $N_{t,e,a_e} \geq (1 + \gamma) \tilde{v}_{t,e,a_e} \log(t)$ , ESM enters the exploitation phase: it selects the best empirical action  $a_{\hat{\theta}_t}^\star = \arg \max_{a \in \mathcal{A}} \sum_{e \in [\rho]} \hat{\theta}_{t,e,a_e}$ . Generally, computing  $a_{\hat{\theta}_t}^\star$  requires a max operation over an exponential number of actions  $a \in \mathcal{A}$ . Fortunately, due to the factored structure, we can implement the max operation efficiently through a VE procedure [39] (see App. G).
Estimation. If not enough information has been gathered, ESM enters an estimation phase, where it selects the least explored group action similarly to the initialization phase. This ensures that the certainty equivalence holds, i.e., that  $\hat{\theta}_t$  is estimated accurately.
Exploration. Otherwise, the algorithm enters the exploration phase and selects actions as suggested by the solution of  $C_{\hat{\theta}_t}^\diamond (m)$ . More precisely, we select a global action  $a_{t}\in \mathcal{A}$  which contains a group action  $a_{t,e'} = b_{e'}$  that minimizes the following ratio where  $e^{\prime}$  and  $b_{e^{\prime}}$  are the group index and group action which minimize  $\frac{N_{t,e,a_e}}{\tilde{v}_{t,e,a_e}}$ .
Upper bound. We establish that the ESM algorithm achieves a regret, matching the approximate lower bound  $C_{\theta}^{\diamond}(m) \log (T)$ , asymptotically as  $T \to \infty$ . The proof is given in App. D.
Theorem 6.1. Let  $\varepsilon < 1 / |\mathcal{A}_0|$ . For any  $m\in [K^N -1]$ , we have that
1.  $\lim \sup_{T\to \infty}\frac{R^{\pi}(T)}{\log(T)}\leq C_{\theta}^{\mathrm{MF}}(m)\xi (\varepsilon ,\gamma),$  for  $\pi = \operatorname {ESM}(\mathcal{A}_0,\varepsilon ,\gamma ,\mathrm{MF},m)$  ,for any  $\theta$
2.  $\lim \sup_{T\to \infty}\frac{R^{\pi}(T)}{\log(T)}\leq C_{\theta}^{\mathrm{L}}(m)\xi (\varepsilon ,\gamma),$  for  $\pi = \operatorname {ESM}(\mathcal{A}_0,\varepsilon ,\gamma ,\mathrm{L},m)$  , for any  $\theta$  described by acyclic factor graphs,
where  $\xi$  is a function such that  $\lim_{(\varepsilon, \gamma) \to (0, 0)} \xi(\varepsilon, \gamma) = 1$ .
In this section, we present numerical experiments to assess the performance of our algorithm. We propose two sets of experiments: (i) a set of synthetic MAMABs with different graph topologies, and (ii) an industrial use-case from the radio communication domain: antenna tilt optimization. The code for the synthetic experiments and the additional experiments presented in App. K is available at this link.
7.1 Synthetic Experiments
Problem instances. We consider the factor graphs depicted in Fig. 3. The expected rewards are selected uniformly at random in the interval [0, 10]. In our experiments, select  $N = 5$  and  $K = 3$ . We execute our experiments for  $N_{\mathrm{sim}} = 5$  independent runs. Following previous work [8], we select  $\gamma = 0$ , and  $\varepsilon = 0.01$ . The elimination order is chosen as  $\mathcal{O} = [N]$ . We implement the solver for the lower bound optimization problems using CVXPY [17] with a MOSEK solver [1].
Figure 3: Factor graphs used in the synthetic experiments: ring (left), line (center), star (right).
Results. The results for the regret (in log scale) are presented in Fig. 4. The performance of ESM is compared to that of MAUCE [3], HEIST [35], and to a random strategy selecting actions uniformly at random. The computational complexity results, reported in Fig. 5, measure the running time (in sec.) to solve an instance of the approximate lower bound optimization problem  $C_{\theta}^{\diamond}(m)$ . We use  $m = \tilde{A}$ , and  $\diamond = \mathrm{MF}$  for the ring, or  $\diamond = \mathrm{L}$  for the star and line graph topologies.
Figure 4: Regret results for the synthetic instances.
Figure 5: Running time (in sec.) to solve an instance of  $C_{\diamond}^{\theta}(m)$ , when varying  $m$ .
7.2 Antenna tilt optimization
Next, we test our algorithm on a radio network optimization task. The goal is to control the vertical antenna tilt at different network Base Stations to optimize the network throughput. In the following, we detail the network model, our simulation setup, and present our experimental results.
Network model. We consider a sectorized radio network consisting of a set of sectors  $S = [N]$ . The set of sectors corresponds to the set of agents in our MAMAB framework. Since each sector is associated to a unique antenna, we will use the terms sector and antenna interchangeably. We assume that each sector  $i \in S$  serves (on the downlink) a fixed set of Users Equipments (UEs)  $\mathcal{U}_i$  (each UE is associated with a unique antenna, that from which it receives the strongest signal).
Factor graph. We model the observed reward in the radio network as a factor graph with  $N = |S|$  agent nodes and  $\rho = |\mathcal{S}|$  factor nodes. Each sector is associated with a unique factor, which models the rewards observed in that sector. We build the factor graph based on the interference pattern of the antennas, i.e., antennas that can interfere with each other are connected to common factors. An example of such a graph and additional experimental details are reported in App. L.
Actions and rewards. The action  $a_{t,i}$  represents the antenna tilt for sector  $i \in S$  and at time  $t$ . It is chosen from a discrete set of  $K$  tilts, i.e.,  $a_{t,i} \in \{\alpha_1, \ldots, \alpha_K\}$ . The tilt for a group of sectors  $e$  is denoted by  $a_e$ . Rewards are based on the throughput of UEs in sector  $i$ , which depends on the actions of a group of agents  $a_e$ :  $r_e(a_e) = \sum_{u \in \mathcal{U}_i} T_{i,u}(a_e)$ , where  $T_{i,u}$  is the throughput of an UE  $u$  associated to sector  $i$ . Hence, the global reward for a tilt configuration  $a \in \mathcal{A}$  is  $r(a) = \sum_{i \in [N]} \sum_{u \in \mathcal{U}_i} T_{i,u}(a_e)$ . The throughput  $T_{i,u}$  depends on channel conditions (or fading) between the antenna and the user. These conditions rapidly evolve over time around their mean.
**Simulator.** We run our experiments in a proprietary mobile network simulator in an urban environment. The simulation parameters used in our experiments are reported in App. K. Based on the user positions and network parameters, the simulator computes the path loss in the network environment using a BEZT propagation model [32] and returns the throughput for each sector by conducting user association and resource allocation in a full-buffer traffic demand scenario.
Results. We test our algorithm for  $\mathcal{A}_i = \{2^\circ, 7^\circ, 13^\circ\}$ , and for  $|S| = 6$  sectors. As the factor graph contains cycles, we use  $\diamondsuit = \mathrm{MF}$  and select  $m = 3$ . The results, presented in Fig. 6, are in line with the experimental findings of the previous section. However, the ESM running time is higher due to the higher complexity of the factor graph.
(i) Regret (log-scale).
Figure 6: Results for the antenna tilt optimization experiments.
(ii) Running time (in sec.).
In this paper, we investigated the problem of regret minimization in MAMABs: we derived a regret lower bound, proposed approximations of it, and devised ESM, an algorithm trading off statistical and computational efficiency. We then assessed the performance of ESM on both synthetic examples and the antenna tilt optimization problem. Interesting future research directions include proposing efficient distributed implementations of ESM, quantifying on its communication complexity, and investigating representation learning problems in MAMABs where the underlying factor graph defining the reward is unknown and needs to be learned.
[1] MOSEK ApS. The MOSEK optimization toolbox for MATLAB manual. Version 9.0. 2019.
[2] Eugenio Bargiacchi et al. "Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems". In: Proc. of ICML. 2018.
[3] Eugenio Bargiacchi et al. "Multi-agent RMax for Multi-Agent Multi-Armed Bandits". In: Proc. of Adaptive and Learning Agents Worksh. 2022.
[4] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[5] Stephen Boyd et al. A tutorial on geometric programming. Optimization and Engineering, 2004.
[6] Shouyuan Chen et al. "Combinatorial Pure Exploration of Multi-Armed Bandits". In: Proc. of NeurIPS. 2014.
[7] Wei Chen, Yajun Wang, and Yang Yuan. "Combinatorial Multi-Armed Bandit: General Framework and Applications". In: Proc. of Mach. Learn. Res. 2013.
[8] Richard Combes, Stefan Magureanu, and Alexandre Proutiere. "Minimal Exploration in Structured Stochastic Bandits". In: Proc. of NeurIPS. 2017.
[9] Richard Combes and Alexandre Proutiere. "Unimodal bandits: Regret lower bounds and optimal algorithms". In: Proc. of ICML. 2014.
[10] Richard Combes et al. "Combinatorial Bandits Revisited". In: Proc. of NeurIPS. 2015.
[11] Richard Combes et al. "Learning to rank: Regret lower bounds and efficient algorithms". In: Proc. of SIGMETRICS. 2015.
[12] Thibaut Cuvelier, Richard Combes, and Eric Gourdin. "Asymptotically Optimal Strategies For Combinatorial Semi-Bandits in Polynomial Time". In: Proc. of ALT. 2021.