SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similarly to the strongly observable graph case, we also derive an adaptive tuning strategy for  $\gamma$  to achieve a more refined regret bound  $\widetilde{\mathcal{O}}\left(\sqrt{\sum_{t=1}^{T}\widetilde{\alpha}_{t}\mathbf{Reg}_{\mathrm{Sq}}} + \left(\sum_{t=1}^{T}\sqrt{d_{t}}\right)^{\frac{2}{3}}\mathbf{Reg}_{\mathrm{Sq}}^{\frac{1}{3}}\right)$  where  $\widetilde{\alpha}_{t}$  is the independence number of the subgraph induced by nodes with self-loops in  $G_{t}$  and  $d_{t}$  is the weakly domination number of  $G_{t}$ . This is again achieved without explicitly computing  $\widetilde{\alpha}_{t}$  and  $d_{t}$ ; see Appendix D for details.
In this section, we show that solving  $\mathrm{argmin}_{p\in \Delta (K)}\overline{\mathrm{dec}}_{\gamma}(p;\widehat{f},x,G)$  in Algorithm 1 is equivalent to solving a convex program, which can be easily and efficiently implemented in practice.
Theorem 3.6. Solving  $\mathrm{argmin}_{p\in \Delta (K)}\overline{\mathrm{dec}}_{\gamma}(p;\widehat{f},x,G)$  is equivalent to solving the following convex optimization problem.
$$
\min  _ {p \in \Delta (K), z} p ^ {\top} \widehat {f} + z \tag {5}
$$
$$
\text {s u b j e c t} \quad \forall a \in [ K ]: \frac {1}{\gamma} \| p - e _ {a} \| _ {\operatorname {d i a g} (G ^ {\top} p) ^ {- 1}} ^ {2} \leq \widehat {f} (x, a) + z,
$$
$$
G ^ {\top} p \succ 0,
$$
where  $\widehat{f}$  in the objective is a shorthand for  $\widehat{f}(x, \cdot) \in \mathbb{R}^K$ ,  $e_a$  is the  $a$ -th standard basis vector, and  $\succ$  means element-wise greater.
The proof is deferred to Appendix A.4. Note that this implementation is not restricted to the deterministic feedback graphs but applies to the general stochastic feedback graph case. In Appendix A.3, we provide the 20 lines of Python code that solves Eq. (5).
4 Examples with Closed-Form Solutions
In this section, we present examples and corresponding closed-form solutions of  $p$  that make the value  $\overline{\operatorname{dec}}_{\gamma}(p; \widehat{f}, x, G)$  upper bounded by at most a constant factor of  $\min_p \overline{\operatorname{dec}}_{\gamma}(p; \widehat{f}, x, G)$ . This offers an alternative to solving the convex program defined in Theorem 3.6 for special (and practically relevant) cases, thereby enhancing the efficiency of our algorithm. All the proofs are deferred to Appendix B.
Cops-and-Robbers Graph. The "cops-and-robbers" feedback graph  $G_{\mathrm{CR}} = 11^{\top} - I$ , also known as the loopless clique, is the full feedback graph removing self-loops. Therefore,  $G_{\mathrm{CR}}$  is strongly observable with independence number  $\alpha = 1$ . Let  $a_1$  be the node with the smallest value of  $\widehat{f}$  and  $a_2$  be the node with the second smallest value of  $\widehat{f}$ . Our proposed closed-form distribution  $p$  is only supported on  $\{a_1, a_2\}$  and defined as follows:
$$
p _ {a _ {1}} = 1 - \frac {1}{2 + \gamma \left(\widehat {f} _ {a _ {2}} - \widehat {f} _ {a _ {1}}\right)}, \quad p _ {a _ {2}} = \frac {1}{2 + \gamma \left(\widehat {f} _ {a _ {2}} - \widehat {f} _ {a _ {1}}\right)}. \tag {6}
$$
In the following proposition, we show that with the construction of  $p$  in Eq. (6),  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G_{\mathrm{CR}})$  is upper bounded by  $\mathcal{O}(1 / \gamma)$ , which matches the order of  $\min_p \overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G)$  based on Theorem 3.2 since  $\alpha = 1$ .
Proposition 1. When  $G = G_{\mathrm{CR}}$ , given any  $\widehat{f}$ , context  $x$ , the closed-form distribution  $p$  in Eq. (6) guarantees that  $\overline{\operatorname{dec}}_{\gamma}(p; \widehat{f}, x, G_{\mathrm{CR}}) \leq \mathcal{O}\left(\frac{1}{\gamma}\right)$ .
Apple Tasting Graph. The apple tasting feedback graph  $G_{\mathrm{AT}} = \begin{bmatrix} 1 & 1 \\ 0 & 0 \end{bmatrix}$  consists of two nodes, where the first node reveals all and the second node reveals nothing. This scenario was originally proposed by Helmbold et al. [2000] and recently denoted the spam filtering graph [van der Hoeven et al., 2021]. The independence number of  $G_{\mathrm{AT}}$  is 1. Let  $\widehat{f}_1$  be the oracle prediction for the first node and let  $\widehat{f}_2$  be the prediction for the second node. We present a closed-form solution  $p$  for Eq. (4) as follows:
$$
p _ {1} = \left\{ \begin{array}{l l} 1 & \widehat {f} _ {1} \leq \widehat {f} _ {2} \\ \frac {2}{4 + \gamma \left(\widehat {f} _ {1} - \widehat {f} _ {2}\right)} & \widehat {f} _ {1} > \widehat {f} _ {2} \end{array} , \quad p _ {2} = 1 - p _ {1}. \right. \tag {7}
$$
We show that this distribution  $p$  satisfies that  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G_{\mathrm{AT}})$  is upper bounded by  $\mathcal{O}(1 / \gamma)$  in the following proposition. We remark that directly applying results from [Foster et al., 2021] cannot lead to a valid upper bound since the second node does not have a self-loop.
Proposition 2. When  $G = G_{\mathrm{AT}}$ , given any  $\widehat{f}$ , context  $x$ , the closed-form distribution  $p$  in Eq. (7) guarantees that  $\overline{\operatorname{dec}}_{\gamma}(p; \widehat{f}, x, G_{\mathrm{AT}}) \leq \mathcal{O}\left(\frac{1}{\gamma}\right)$ .
Inventory Graph. In this application, the algorithm needs to decide the inventory level in order to fulfill the realized demand arriving at each round. Specifically, there are  $K$  possible chosen inventory levels  $a_1 < a_2 < \ldots < a_K$  and the feedback graph  $G_{\mathrm{inv}}$  has entries  $G(i,j) = 1$  for all  $1 \leq j \leq i \leq K$  and  $G(i,j) = 0$  otherwise, meaning that picking the inventory level  $a_i$  informs about all actions  $a_{j \leq i}$ . This is because items are consumed until either the demand or the inventory is exhausted. The independence number of  $G_{\mathrm{inv}}$  is 1. Therefore, (very) large  $K$  is statistically tractable, but naively solving the convex program Eq. (5) requires superlinear in  $K$  computational cost. We show in the following proposition that there exists an analytic form of  $p$ , which guarantees that  $\overline{\operatorname{dec}}_{\gamma}(p; \widehat{f}, x, G_{\mathrm{inv}})$  can be bounded by  $\mathcal{O}(1 / \gamma)$ .
Proposition 3. When  $G = G_{\mathrm{inv}}$ , given any  $\widehat{f}$ , context  $x$ , there exists a closed-form distribution  $p \in \Delta(K)$  guaranteeing that  $\overline{\operatorname{dec}}_{\gamma}(p; \widehat{f}, x, G_{\mathrm{inv}}) \leq \mathcal{O}\left(\frac{1}{\gamma}\right)$ , where  $p$  is defined as follows:  $p_j = \max \left\{\frac{1}{1 + \gamma(\widehat{f}_j - \min_i \widehat{f}_i)} - \sum_{j' > j} p_{j'}\right\}$  for all  $j \in [K]$ .
Undirected Self-Aware Graph. For the undirected and self-aware feedback graph  $G$ , which means that  $G$  is symmetric and has diagonal entries all 1, we also show that a certain closed-form solution of  $p$  satisfies that  $\overline{\operatorname{dec}}_{\gamma}(p; \widehat{f}, x, G)$  is bounded by  $\mathcal{O}(\frac{\alpha}{\gamma})$ .
Proposition 4. When  $G$  is an undirected self-aware graph, given any  $\widehat{f}$ , context  $x$ , there exists a closed-form distribution  $p \in \Delta(K)$  guaranteeing that  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G) \leq \mathcal{O}\left(\frac{\alpha}{\gamma}\right)$ .
In this section, we use empirical results to demonstrate the significant benefits of SquareCB.G in leveraging the graph feedback structure and its superior effectiveness compared to SquareCB. Following Foster and Krishnamurthy [2021], we use progressive validation (PV) loss as the evaluation metric, defined as  $L_{\mathrm{pv}}(T) = \frac{1}{T}\sum_{t=1}^{T}\ell_{t,a_t}$ . All the feedback graphs used in the experiments are deterministic. We run experiments on CPU Intel Xeon Gold 6240R 2.4G and the convex program solver is implemented via Vowpal Wabbit [Langford et al., 2007].
5.1 SquareCB.G under Different Feedback Graphs
In this subsection, we show that our SquareCB.G benefits from considering the graph structure by evaluating the performance of SquareCB.G under three different feedback graphs. We conduct experiments on RCV1 dataset and leave the implementation details in Appendix C.1.
The performances of SquareCB.G under bandit graph, full information graph and cops-and-robbers graph are shown in the left part of Figure 1. We observe that SquareCB.G performs the best under full information graph and performs worst under bandit graph. Under the cops-and-robbers graph, much of the gap between bandit and full information is eliminated. This improvement demonstrates the benefit of utilizing graph feedback for accelerating learning.
5.2 Comparison between SquareCB.G and SquareCB
In this subsection, we compare the effectiveness of SquareCB.G with the SquareCB algorithm. To ensure a fair comparison, both algorithms update the regressor using the same feedbacks based on the graph. The only distinction lies in how they calculate the action probability distribution. We summarize the main results here and leave the implementation details in Appendix C.2.
5.2.1 Results on Random Directed Self-aware Graphs
We conduct experiments on RCV1 dataset using random directed self-aware feedback graphs. Specifically, at round  $t$ , the deterministic feedback graph  $G_{t}$  is generated as follows. The diagonal elements of  $G_{t}$  are all 1, and each off-diagonal entry is drawn from a Bernoulli $^{(3/4)}$  distribution. The results are presented in the right part of Figure 1. Our SquareCB.G consistently outperforms SquareCB and demonstrates lower variance, particularly when the number of iterations was small. This is because when there are fewer samples available to train the regressor, it is more crucial to design an effective algorithm that can leverage the graph feedback information.
5.2.2 Results on Synthetic Inventory Dataset
In the inventory graph experiments, we create a synthetic inventory dataset and design a loss function for each inventory level  $a_{t} \in [0,1]$  with demand  $d_{t} \in [0,1]$ . Since the action set  $[0,1]$  is continuous, we discretize the action set in two different ways to apply the algorithms.
Fixed discretized action set. In this setting, we discretize the action set using fixed grid size  $\varepsilon \in \{\frac{1}{100},\frac{1}{300},\frac{1}{500}\}$ , which leads to a finite action set  $\mathcal{A}$  of size  $\frac{1}{\varepsilon} + 1$ . Note that according to Theorem 3.2, our regret does not scale with the size of the action set (to within polylog factors), as the independence number is always 1. The results are shown in the left part of Figure 2.
We remark several observations from the results. First, our algorithm SquareCB.G outperforms SquareCB for all choices  $K \in \{101, 301, 501\}$ . This indicates that SquareCB.G utilizes a better exploration scheme and effectively leverages the structure of  $G_{\mathrm{inv}}$ . Second, we observe that SquareCB.G indeed does not scale with the size of the discretized action set  $\mathcal{A}$ , since under different discretization scales, SquareCB.G has similar performances and the slight differences are from the improved approximation error with finer discretization. This matches the theoretical guarantee that we prove in Theorem 3.2. On the other hand, SquareCB does perform worse when the size of the action set increases, matching its theoretical guarantee which scales with the square root of the size of the action set.
Adaptively changing action set. In this setting, we adaptively discretize the action set  $[0,1]$  according to the index of the current round. Specifically, for SquareCB.G, we uniformly discretize the action set  $[0,1]$  with size  $\lceil \sqrt{t}\rceil$ , whose total discretization error is  $\mathcal{O}(\sqrt{T})$  due to the Lipschitzness of the loss function. For SquareCB, to optimally balance the dependency on the size of the action set and the discretization error, we uniformly discretize the action set  $[0,1]$  into  $\lceil t^{\frac{1}{3}}\rceil$  actions. The results are illustrated in the right part of Figure 2. We can observe that SquareCB.G consistently outperforms SquareCB by a clear margin.
Multi-armed bandits with feedback graphs have been extensively studied. An early example is the apple tasting problem of Helmbold et al. [2000]. The general formulation was introduced by Mannor and Shamir [2011]. Alon et al. [2015] characterized the minimax rates in terms of graph-theoretic quantities. Follow-on work includes relaxing the assumption that the graph is observed prior to decision [Cohen et al., 2016]; analyzing the distinction between the stochastic and adversarial settings [Alon et al., 2017]; considering stochastic feedback graphs [Li et al., 2020, Esposito et al., 2022]; instance-adaptivity [Ito et al., 2022]; data-dependent regret bound [Lykouris et al., 2018, Lee et al., 2020]; and high-probability regret under adaptive adversary [Neu, 2015, Luo et al., 2023].
The contextual bandit problem with feedback graphs has received relatively less attention. Wang et al. [2021] provide algorithms for adversarial linear bandits with uninformed graphs and stochastic contexts. However, this work assumes several unrealistic assumptions on both the policy class and the context space and is not comparable to our setting, since we consider the informed graph setting with adversarial context. Singh et al. [2020] study a stochastic linear bandits with informed feedback graphs and are able to improve over the instance-optimal regret bound for bandits derived in [Lattimore and Szepesvari, 2017] by utilizing the additional graph-based feedbacks.
Our work is also closely related to the recent progress in designing efficient algorithms for classic contextual bandits. Starting from [Langford and Zhang, 2007], numerous works have been done to the development of practically efficient algorithms, which are based on reduction to either cost-sensitive classification oracles [Dudik et al., 2011, Agarwal et al., 2014] or online regression oracles [Foster and Rakhlin, 2020, Foster et al., 2020, 2021, Zhu and Mineiro, 2022]. Following the latter trend, our work assumes access to an online regression oracle and extends the classic bandit problems to the bandits with general feedback graphs.
In this paper, we consider the design of practical contextual bandits algorithm with provable guarantees. Specifically, we propose the first efficient algorithm that achieves near-optimal regret bound for contextual bandits with general directed feedback graphs with an online regression oracle.
While we study the informed graph feedback setting, where the entire feedback graph is exposed to the algorithm prior to each decision, many practical problems of interest are possibly uninformed graph feedback problems, where the graph is unknown at the decision time. It is unclear how to formulate an analogous minimax problem to Eq. (1) under the uninformed setting. One idea is to consume the additional feedback in the online regressor and adjust the prediction loss to reflect this additional structure, e.g., using the more general version of the E2D framework which incorporates arbitrary side observations [Foster et al., 2021]. Cohen et al. [2016] consider this uninformed setting in the non-contextual case and prove a sharp distinction between the adversarial and stochastic settings: even if the graphs are all strongly observable with bounded independence number, in the adversarial setting the minimax regret is  $\Theta(T)$  whereas in the stochastic setting the minimax regret is  $\Theta(\sqrt{\alpha T})$ . Intriguingly, our setting is semi-adversarial due to realizability of the mean loss, and therefore it is apriori unclear whether the negative adversarial result applies.
In addition, bandits with graph feedback problems often present with associated policy constraints, e.g., for the apple tasting problem, it is natural to rate limit the informative action. Therefore, another interesting direction is to combine our algorithm with the recent progress in contextual bandits with knapsack [Slivkins and Foster, 2022], leading to more practical algorithms.
Acknowledgments HL and MZ are supported by NSF Awards IIS-1943607.
Alekh Agarwal, Miroslav Dudík, Satyen Kale, John Langford, and Robert Schapire. Contextual bandit learning with predictable rewards. In Artificial Intelligence and Statistics, pages 19-26. PMLR, 2012.
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning, pages 1638-1646. PMLR, 2014.
Noga Alon, Nicolo Cesa-Bianchi, Claudio Gentile, and Yishay Mansour. From bandits to experts: A tale of domination and independence. Advances in Neural Information Processing Systems, 26, 2013.
Noga Alon, Nicolo Cesa-Bianchi, Ofer Dekel, and Tomer Koren. Online learning with feedback graphs: Beyond bandits. In Conference on Learning Theory, pages 23-35. PMLR, 2015.
Noga Alon, Nicolo Cesa-Bianchi, Claudio Gentile, Shie Mannor, Yishay Mansour, and Ohad Shamir. Nonstochastic multi-armed bandits with graph-structured feedback. SIAM Journal on Computing, 46(6):1785-1826, 2017.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.
Djallel Bouneffouf, Irina Rish, and Charu Aggarwal. Survey on applications of multi-armed and contextual bandits. In 2020 IEEE Congress on Evolutionary Computation (CEC), pages 1-8. IEEE, 2020.
Gergely Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. Advances in Neural Information Processing Systems, 28, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Mark Rucker, Joran T Ash, John Langford, Paul Mineiro, and Ida Momennejad. Eigen memory tree. arXiv preprint arXiv:2210.14077, 2022.
David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research, 2021.
Rahul Singh, Fang Liu, Xin Liu, and Ness Shroff. Contextual bandits with side-observations. arXiv preprint arXiv:2006.03951, 2020.
Aleksandrs Slivkins and Dylan Foster. Efficient contextual bandits with knapsacks via regression. arXiv preprint arXiv:2211.07484, 2022.
Dirk van der Hoeven, Federico Fusco, and Nicolò Cesa-Bianchi. Beyond bandit feedback in online multiclass classification. Advances in Neural Information Processing Systems, 34:13280-13291, 2021.
Vladimir G Vovk. A game of prediction with expert advice. In Proceedings of the eighth annual conference on Computational learning theory, pages 51-60, 1995.
Lingda Wang, Bingcong Li, Huozhi Zhou, Georgios B Giannakis, Lav R Varshney, and Zhizhen Zhao. Adversarial linear contextual bandits with graph-structured side observations. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10156–10164, 2021.
Yinglun Zhu and Paul Mineiro. Contextual bandits with smooth regret: Efficient learning in continuous action spaces. In International Conference on Machine Learning, pages 27574-27590. PMLR, 2022.
A Omitted Details in Section 3
Theorem 3.1. Suppose  $\overline{\mathrm{dec}}_{\gamma}(p_t;\widehat{f}_t,x_t,G_t)\leq C\gamma^{-\beta}$  for all  $t\in [T]$  and some  $\beta >0$  Algorithm 1 with  $\gamma = \max \{4,(CT)^{\frac{1}{\beta + 1}}\mathbf{Reg}_{\mathsf{Sq}}^{-\frac{1}{\beta + 1}}\}$  guarantees that  $\mathbb{E}[\mathbf{Reg}_{\mathsf{CB}}]\leq \mathcal{O}\Big(C^{\frac{1}{\beta + 1}}T^{\frac{1}{\beta + 1}}\mathbf{Reg}_{\mathsf{Sq}}^{\frac{\beta}{\beta + 1}}\Big)$ .
Proof. Following [Foster et al., 2020], we decompose  $\mathbf{Reg}_{\mathrm{CB}}$  as follows:
$$
\begin{array}{l} \mathbb {E} [ \mathrm {R e g} _ {\mathrm {C B}} ] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} f ^ {\star} \left(x _ {t}, a _ {t}\right) - \sum_ {t = 1} ^ {T} f ^ {\star} \left(x _ {t}, \pi^ {\star} \left(x _ {t}\right)\right) \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(f ^ {\star} \left(x _ {t}, a _ {t}\right) - f ^ {\star} \left(x _ {t}, \pi^ {\star} \left(x _ {t}\right)\right) - \frac {\gamma}{4} \mathbb {E} _ {A \sim G _ {t} (\cdot | a _ {t})} \left[ \sum_ {a \in A} \left(\widehat {f} _ {t} \left(x _ {t}, a\right) - f ^ {\star} \left(x _ {t}, a\right)\right) ^ {2} \right]\right) \right] \\ + \frac {\gamma}{4} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {A \sim G _ {t} (\cdot | a _ {t})} \left[ \sum_ {a \in A} \left(\widehat {f _ {t}} (x _ {t}, a) - f ^ {\star} (x _ {t}, a)\right) ^ {2} \right] \right] \\ \leq \mathbb{E}\left[\sum_{t = 1}^{T}\max_{\substack{a^{\star}\in [K]\\ f\in (\mathcal{X}\times [K]\mapsto \mathbb{R})}}\mathbb{E}_{a_{t}\sim p_{t}}\left[f(x_{t},a_{t}) - f(x_{t},a^{\star}) - \frac{\gamma}{4}\mathbb{E}_{A\sim G_{t}(\cdot |a_{t})}\left[\sum_{a\in A}\Bigl(\widehat{f}_{t}(x_{t},a) - f(x_{t},a)\Bigr)^{2}\right]\right]\right] \\ + \frac {\gamma}{4} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {A \sim G _ {t} (\cdot | a _ {t})} \left[ \sum_ {a \in A} \left(\widehat {f _ {t}} (x _ {t}, a) - f ^ {\star} (x _ {t}, a)\right) ^ {2} \right] \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \overline {{\operatorname {d e c}}} _ {\gamma} \left(p _ {t}; \widehat {f} _ {t}, x _ {t}, G _ {t}\right) \right] + \frac {\gamma}{4} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {A \sim G _ {t} (\cdot | a _ {t})} \left[ \sum_ {a \in A} \left(\widehat {f} _ {t} \left(x _ {t}, a\right) - f ^ {\star} \left(x _ {t}, a\right)\right) ^ {2} \right] \right] \tag {8} \\ \leq C T \gamma^ {- \beta} + \frac {\gamma}{4} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {A \sim G _ {t} (\cdot | a _ {t})} \left[ \sum_ {a \in A} \left(\widehat {f _ {t}} (x _ {t}, a) - f ^ {\star} (x _ {t}, a)\right) ^ {2} \right] \right]. \\ \end{array}
$$
Next, since  $\mathbb{E}[\ell_{t,a}\mid x_t] = f^{\star}(x_t,a)$  for all  $t\in [T]$  and  $a\in \mathcal{A}$ , we know that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {A \sim G _ {t} (\cdot | a _ {t})} \left[ \sum_ {a \in A} \left(\widehat {f} _ {t} (x _ {t}, a) - f ^ {\star} (x _ {t}, a)\right) ^ {2} \right] \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {A \sim G _ {t} (\cdot | a _ {t})} \left[ \sum_ {a \in A} \left(\widehat {f} _ {t} \left(x _ {t}, a\right) - \ell_ {t, a}\right) ^ {2} - \sum_ {a \in A} \left(f ^ {\star} \left(x _ {t}, a\right) - \ell_ {t, a}\right) ^ {2} \right] \right] \leq \operatorname {R e g} _ {\mathrm {S q}}, \tag {9} \\ \end{array}
$$
where the final inequality is due to Assumption 2.
$$
\mathbb {E} [ \mathbf {R e g} _ {\mathrm {C B}} ] \leq C T \gamma^ {- \beta} + \frac {\gamma}{4} \mathbf {R e g} _ {\mathrm {S q}}.
$$
Picking  $\gamma = \max \left\{4, \left(\frac{CT}{\mathrm{Reg}_{\mathrm{Sq}}}\right)^{\frac{1}{\beta + 1}}\right\}$ , we obtain that
$$
\mathbb {E} \left[ \mathbf {R e g} _ {\mathsf {C B}} \right] \leq \mathcal {O} \left(C ^ {\frac {1}{\beta + 1}} T ^ {\frac {1}{\beta + 1}} \mathbf {R e g} _ {\mathsf {S q}} ^ {\frac {\beta}{\beta + 1}}\right).
$$
A.1 Proof of Theorem 3.2
Before proving Theorem 3.2, we first show the following key lemma, which is useful in proving that  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G)$  is convex for both strongly and weakly observable feedback graphs  $G$ . We
highlight that the convexity of  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G)$  is crucial for both proving the upper bound of  $\min_{p \in \Delta(K)} \overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G)$  and showing the efficiency of Algorithm 1.
Lemma A.1. Suppose  $u, v, x \in \mathbb{R}^d$  with  $\langle v, x \rangle > 0$ . Then both  $g(x) = \frac{\langle u, x \rangle^2}{\langle v, x \rangle}$  and  $h(x) = \frac{(1 - \langle u, x \rangle)^2}{\langle v, x \rangle}$  are convex in  $x$ .
Proof. The function  $f(x,y) = x^{2} / y$  is convex for  $y > 0$  due to
$$
\nabla^ {2} f (x, y) = \frac {2}{y ^ {3}} \left[ \begin{array}{c} y \\ - x \end{array} \right] \left[ \begin{array}{c} y \\ - x \end{array} \right] ^ {\top} \succeq 0.
$$