SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mengxiao Zhang*
mengxiao.zhang@usc.edu
Yuheng Zhang *
yuhengz2@illinois.edu
Olga Vrousgou
Olga.Vrousgou@microsoft.com
Haipeng Luo
haipengl@usc.edu
Paul Mineiro
pmineiro@microsoft.com
While contextual bandit has a mature theory, effectively leveraging different feedback patterns to enhance the pace of learning remains unclear. Bandits with feedback graphs, which interpolates between the full information and bandit regimes, provides a promising framework to mitigate the statistical complexity of learning. In this paper, we propose and analyze an approach to contextual bandits with feedback graphs based upon reduction to regression. The resulting algorithms are computationally practical and achieve established minimax rates, thereby reducing the statistical complexity in real-world applications.
This paper is primarily concerned with increasing the pace of learning for contextual bandits [Auer et al., 2002, Langford and Zhang, 2007]. While contextual bandits have enjoyed broad applicability [Bouneffouf et al., 2020], the statistical complexity of learning with bandit feedback imposes a data lower bound for application scenarios [Agarwal et al., 2012]. This has inspired various mitigation strategies, including exploiting function class structure for improved experimental design [Zhu and Mineiro, 2022], and composing with memory for learning with fewer samples [Rucker et al., 2022]. In this paper we exploit alternative graph feedback patterns to accelerate learning: intuitively, there is no need to explore a potentially suboptimal action if a presumed better action, when exploited, yields the necessary information.
The framework of bandits with feedback graphs is mature and provides a solid theoretical foundation for incorporating additional feedback into an exploration strategy [Mannor and Shamir, 2011, Alon et al., 2015, 2017]. Succinctly, in this framework, the observation of the learner is decided by a directed feedback graph  $G$ : when an action is played, the learner observes the loss of every action to which the chosen action is connected. When the graph only contains self-loops, this problem reduces to the classic bandit case. For non-contextual bandits with feedback graphs, [Alon et al., 2015] provides a full characterization on the minimax regret bound with respect to different graph theoretic quantities associated with  $G$  according to the type of the feedback graph.
However, contextual bandits with feedback graphs have received less attention [Singh et al., 2020, Wang et al., 2021]. Specifically, there is no prior work offering a solution for general feedback graphs
and function classes. In this work, we take an important step in this direction by adopting recently developed minimax algorithm design principles in contextual bandits, which leverage realizability and reduction to regression to construct practical algorithms with strong statistical guarantees [Foster et al., 2018, Foster and Rakhlin, 2020, Foster et al., 2020, Foster and Krishnamurthy, 2021, Foster et al., 2021, Zhu and Mineiro, 2022]. Using this strategy, we construct a practical algorithm for contextual bandits with feedback graphs that achieves the optimal regret bound. Moreover, although our primary concern is accelerating learning when the available feedback is more informative than bandit feedback, our techniques also succeed when the available feedback is less informative than bandit feedback, e.g., in spam filtering where some actions generate no feedback. More specifically, our contributions are as follows.
Contributions. In this paper, we extend the minimax framework proposed in [Foster et al., 2021] to contextual bandits with general feedback graphs, aiming to promote the utilization of different feedback patterns in practical applications. Following [Foster and Rakhlin, 2020, Foster et al., 2021, Zhu and Mineiro, 2022], we assume that there is an online regression oracle for supervised learning on the loss. Based on this oracle, we propose SquareCB.G, the first algorithm for contextual bandits with feedback graphs that operates via reduction to regression (Algorithm 1). Eliding regression regret factors, our algorithm achieves the matching optimal regret bounds for deterministic feedback graphs, with  $\widetilde{\mathcal{O}} (\sqrt{\alpha T})$  regret for strongly observable graphs and  $\widetilde{\mathcal{O}} (d^{\frac{1}{3}}T^{\frac{2}{3}})$  regret for weakly observable graphs, where  $\alpha$  and  $d$  are respectively the independence number and weakly domination number of the feedback graph (see Section 3.2 for definitions). Notably, SquareCB.G is computationally tractable, requiring the solution to a convex program (Theorem 3.6), which can be readily solved with off-the-shelf convex solvers (Appendix A.3). In addition, we provide closed-form solutions for specific cases of interest (Section 4), leading to a more efficient implementation of our algorithm. Empirical results further showcase the effectiveness of our approach (Section 5).
2 Problem Setting and Preliminary
Throughout this paper, we let  $[n]$  denote the set  $\{1,2,\ldots ,n\}$  for any positive integer  $n$ . We consider the following contextual bandits problem with informed feedback graphs. The learning process goes in  $T$  rounds. At each round  $t\in [T]$ , an environment selects a context  $x_{t}\in \mathcal{X}$ , a (stochastic) directed feedback graph  $G_{t}\in [0,1]^{\mathcal{A}\times \mathcal{A}}$ , and a loss distribution  $\mathbb{P}_t:\mathcal{X}\to \Delta ([-1,1]^{\mathcal{A}})$ ; where  $\mathcal{A}$  is the action set with finite cardinality  $K$ . For convenience, we use  $\mathcal{A}$  and  $[K]$  interchangeably for denoting the action set. Both  $G_{t}$  and  $x_{t}$  are revealed to the learner at the beginning of each round  $t$ . Then the learner selects one of the actions  $a_{t}\in \mathcal{A}$ , while at the same time, the environment samples a loss vector  $\ell_t\in [-1,1]^\mathcal{A}$  from  $\mathbb{P}_t(\cdot |x_t)$ . The learner then observes some information about  $\ell_t$  according to the feedback graph  $G_{t}$ . Specifically, for each action  $j$ , she observes the loss of action  $j$  with probability  $G_{t}(a_{t},j)$ , resulting in a realization  $A_{t}$ , which is the set of actions whose loss is observed. With a slight abuse of notation, denote  $G_{t}(\cdot |a)$  as the distribution of  $A_{t}$  when action  $a$  is picked. We allow the context  $x_{t}$ , the (stochastic) feedback graphs  $G_{t}$  and the loss distribution  $\mathbb{P}_t(\cdot |x_t)$  to be selected by an adaptive adversary. When convenient, we will consider  $G$  to be a  $K$ -by- $K$  matrix and utilize matrix notation.
Other Notations. Let  $\Delta(K)$  denote the set of all Radon probability measures over a set  $[K]$ .  $\mathrm{conv}(S)$  represents the convex hull of a set  $S$ . Denote  $I$  as the identity matrix with an appropriate dimension. For a  $K$ -dimensional vector  $v$ ,  $\mathrm{diag}(v)$  denotes the  $K$ -by- $K$  matrix with the  $i$ -th diagonal entry  $v_i$  and other entries 0. We use  $\mathbb{R}_{\geq 0}^{K}$  to denote the set of  $K$ -dimensional vectors with non-negative entries. For a positive definite matrix  $M \in \mathbb{R}^{K \times K}$ , we define norm  $\|z\|_M = \sqrt{z^\top M z}$  for any vector  $z \in \mathbb{R}^K$ . We use the  $\widetilde{\mathcal{O}}(\cdot)$  notation to hide factors that are polylogarithmic in  $K$  and  $T$ .
Realizability. We assume that the learner has access to a known function class  $\mathcal{F} \subset (\mathcal{X} \times \mathcal{A} \mapsto [-1, 1])$  which characterizes the mean of the loss for a given context-action pair, and we make the following standard realizability assumption studied in the contextual bandit literature [Agarwal et al., 2012, Foster et al., 2018, Foster and Rakhlin, 2020, Simchi-Levi and Xu, 2021].
Assumption 1 (Realizability). There exists a regression function  $f^{\star} \in \mathcal{F}$  such that  $\mathbb{E}[\ell_{t,a} \mid x_t] = f^{\star}(x_t, a)$  for any  $a \in \mathcal{A}$  and across all  $t \in [T]$ .
Two comments are in order. First, we remark that, similar to [Foster et al., 2020], misspecification can be incorporated while maintaining computational efficiency, but we do not complicate the exposition here. Second, Assumption 1 induces a "semi-adversarial" setting, wherein nature is completely free to determine the context and graph sequences; and has considerable latitude in determining the loss distribution subject to a mean constraint.
Regret. For each regression function  $f \in \mathcal{F}$ , let  $\pi_f(x_t) \coloneqq \operatorname*{argmin}_{a \in \mathcal{A}} f(x_t, a)$  denote the induced policy, which chooses the action with the least loss with respect to  $f$ . Define  $\pi^{\star} \coloneqq \pi_{f^{\star}}$  as the optimal policy. We measure the performance of the learner via regret to  $\pi^{\star}$ :  $\mathrm{Reg}_{\mathrm{CB}} \coloneqq \sum_{t=1}^{T} \ell_{t,a_t} - \sum_{t=1}^{T} \ell_{t,\pi^{\star}(x_t)}$ , which is the difference between the loss suffered by the learner and the one if the learner applies policy  $\pi^{\star}$ .
Regression Oracle We assume access to an online regression oracle  $\mathrm{Alg}_{\mathrm{Sq}}$  for function class  $\mathcal{F}$ , which is an algorithm for online learning with squared loss. We consider the following protocol. At each round  $t \in [T]$ , the algorithm produces an estimator  $\widehat{f}_t \in \mathrm{conv}(\mathcal{F})$ , then receives a set of context-action-loss tuples  $\{(x_t, a, \ell_{t,a})\}_{a \in A_t}$  where  $A_t \subseteq \mathcal{A}$ . The goal of the oracle is to accurately predict the loss as a function of the context and action, and we evaluate its performance via the square loss  $\sum_{a \in A_t} (\widehat{f}_t(x_t, a) - \ell_{t,a})^2$ . We measure the oracle's cumulative performance via the following square-loss regret to the best function in  $\mathcal{F}$ .
Assumption 2 (Bounded square-loss regret). The regression oracle  $\mathbf{Alg}_{\mathbb{S}^q}$  guarantees that for any (potentially adaptively chosen) sequence  $\{(x_t, a, \ell_{t,a})\}_{a \in A_t, t \in [T]}$  in which  $A_t \subseteq \mathcal{A}$ ,
$$
\sum_ {t = 1} ^ {T} \sum_ {a \in A _ {t}} \left(\widehat {f} _ {t} \left(x _ {t}, a\right) - \ell_ {t, a}\right) ^ {2} - \inf  _ {f \in \mathcal {F}} \sum_ {t = 1} ^ {T} \sum_ {a \in A _ {t}} \left(f \left(x _ {t}, a\right) - \ell_ {t, a}\right) ^ {2} \leq \mathbf {R e g} _ {\mathbf {S q}}.
$$
For finite  $\mathcal{F}$ , Vovk's aggregation algorithm yields  $\mathrm{Reg}_{\mathrm{Sq}} = \mathcal{O}(\log |\mathcal{F}|)$  [Vovk, 1995]. This regret is dependent upon the scale of the loss function, but this need not be linear with the size of  $A_{t}$ , e.g., the loss scale can be bounded by 2 in classification problems. See Foster and Krishnamurthy [2021] for additional examples of online regression algorithms.
3 Algorithms and Regret Bounds
In this section, we provide our main algorithms and results.
3.1 Algorithms via Minimax Reduction Design
Our approach is to adapt the minimax formulation of [Foster et al., 2021] to contextual bandits with feedback graphs. In the standard contextual bandits setting (that is,  $G_{t} = I$  for all  $t$ ), Foster et al. [2021] define the Decision-Estimation Coefficient (DEC) for a parameter  $\gamma > 0$  as  $\mathrm{dec}_{\gamma}(\mathcal{F}) \coloneqq \sup_{\widehat{f} \in \mathrm{conv}(\mathcal{F}), x \in \mathcal{X}} \mathrm{dec}_{\gamma}(\mathcal{F}; \widehat{f}, x)$ , where
$$
\begin{array}{l} \operatorname {d e c} _ {\gamma} (\mathcal {F}; \widehat {f}, x) := \inf  _ {p \in \Delta (K)} \operatorname {d e c} _ {\gamma} (p, \mathcal {F}; \widehat {f}, x) \\ := \inf  _ {p \in \Delta (K)} \sup  _ { \begin{array}{c} a ^ {\star} \in [ K ] \\ f ^ {\star} \in \mathcal {F} \end{array} } \mathbb {E} _ {a \sim p} \left[ f ^ {\star} (x, a) - f ^ {\star} (x, a ^ {\star}) - \frac {\gamma}{4} \cdot \left(\widehat {f} (x, a) - f ^ {\star} (x, a)\right) ^ {2} \right]. \tag {1} \\ \end{array}
$$
Their proposed algorithm is as follows. At each round  $t$ , after receiving the context  $x_{t}$ , the algorithm first computes  $\widehat{f}_{t}$  by calling the regression oracle. Then, it solves the solution  $p_t$  of the minimax problem defined in Eq. (1) with  $\widehat{f}$  and  $x$  replaced by  $\widehat{f}_{t}$  and  $x_{t}$ . Finally, the algorithm samples an action  $a_{t}$  from the distribution  $p_t$  and feeds the observation  $(x_{t},a_{t},\ell_{t,a_{t}})$  to the oracle. Foster et al. [2021] show that for any value  $\gamma$ , the algorithm above guarantees that
$$
\mathbb {E} \left[ \operatorname {R e g} _ {\mathrm {C B}} \right] \leq T \cdot \det  _ {\gamma} (\mathcal {F}) + \frac {\gamma}{4} \cdot \operatorname {R e g} _ {\mathrm {S q}}. \tag {2}
$$
However, the minimax problem Eq. (1) may not be solved efficiently in many cases. Therefore, instead of obtaining the distribution  $p_t$  which has the exact minimax value of Eq. (1), Foster et al.
Algorithm 1 SquareCB.G. Note Theorem 3.6 provides an efficient implementation of Eq. (4).
Input: parameter  $\gamma \geq 4$ , a regression oracle  $\mathbf{Alg}_{\mathrm{Sq}}$
Receive context  $x_{t}$  and directed feedback graph  $G_{t}$ .
Obtain an estimator  $\widehat{f_t}$  from the oracle  $\mathbf{A}\mathrm{lg}_{\mathbf{Sq}}$
Compute the distribution  $p_t \in \Delta(K)$  such that  $p_t = \operatorname*{argmin}_{p \in \Delta(K)} \overline{\operatorname{dec}}_\gamma(p; \widehat{f}_t, x_t, G_t)$ , where
$$
\begin{array}{l} \overline {{\operatorname {d e c} _ {\gamma}}} (p; \widehat {f} _ {t}, x _ {t}, G _ {t}) \\ := \sup  _ {\substack {a ^ {\star} \in [ K ] \\ f ^ {\star} \in \Phi}} \mathbb {E} _ {a \sim p} \left[ f ^ {\star} \left(x _ {t}, a\right) - f ^ {\star} \left(x _ {t}, a ^ {\star}\right) - \frac {\gamma}{4} \mathbb {E} _ {A \sim G _ {t} (\cdot | a)} \left[ \sum_ {a ^ {\prime} \in A} \left(\widehat {f} _ {t} \left(x _ {t}, a ^ {\prime}\right) - f ^ {\star} \left(x _ {t}, a ^ {\prime}\right)\right) ^ {2} \right] \right], \tag{4} \\ \end{array}
$$
and  $\Phi \coloneqq \mathcal{X}\times [K]\mapsto \mathbb{R}$
Sample  $a_{t}$  from  $p_t$  and observe  $\{\ell_{t,j}\}_{j\in A_t}$  where  $A_{t}\sim G_{t}(\cdot |a_{t})$
Feed the tuples  $\{(x_{t},j,\ell_{t,j})\}_{j\in A_{t}}$  to the oracle  $\mathbf{Alg}_{\mathbf{Sq}}$
[2021] show that any distribution that gives an upper bound  $C_{\gamma}$  on  $\mathrm{dec}_{\gamma}(p, \mathcal{F}; \widehat{f}, x)$  also works and enjoys a regret bound with  $\mathrm{dec}_{\gamma}(\mathcal{F})$  replaced by  $C_{\gamma}$  in Eq. (2).
To extend this framework to the setting with feedback graph  $G$ , we define  $\mathrm{dec}_{\gamma}(\mathcal{F};\widehat{f},x,G)$  as follows
$$
\begin{array}{l} \operatorname {d e c} _ {- \gamma} (\mathcal {F}; \widehat {f}, x, G) \\ := \inf  _ {p \in \Delta (K)} \operatorname {d e c} _ {\gamma} (p, \mathcal {F}; \widehat {f}, x, G) \\ := \inf  _ {p \in \Delta (K)} \sup  _ {\substack {a ^ {\star} \in [ K ] \\ f ^ {\star} \in \mathcal {F}}} \mathbb {E} _ {a \sim p} \left[ f ^ {\star} (x, a) - f ^ {\star} (x, a ^ {\star}) - \frac {\gamma}{4} \mathbb {E} _ {A \sim G (\cdot | a)} \left[ \sum_ {a ^ {\prime} \in A} \left(\widehat {f} _ {t} (x, a ^ {\prime}) - f ^ {\star} (x, a ^ {\prime})\right) ^ {2} \right] \right]. \tag{3} \\ \end{array}
$$
Compared with Eq. (1), the difference is that we replace the squared estimation error on action  $a$  by the expected one on the observed set  $A \sim G(\cdot | a)$ , which intuitively utilizes more feedbacks from the graph structure. When the feedback graph is the identity matrix, we recover Eq. (1). Based on  $\mathrm{dec}_{\gamma}(\mathcal{F};\widehat{f},x,G)$ , our algorithm SquareCB.G is shown in Algorithm 1. As what is done in [Foster et al., 2021], in order to derive an efficient algorithm, instead of solving the distribution  $p_t$  with respect to the supremum over  $f^{\star} \in \mathcal{F}$ , we solve  $p_t$  that minimize  $\overline{\mathrm{dec}}_{\gamma}(p;\widehat{f},x_t,G_t)$  (Eq. (4)), which takes supremum over  $f^{\star} \in (\mathcal{X} \times [K] \mapsto \mathbb{R})$ , leading to an upper bound on  $\mathrm{dec}_{\gamma}(\mathcal{F};\widehat{f},x_t,G_t)$ . Then, we receive the loss  $\{\ell_{t,j}\}_{j \in A_t}$  and feed the tuples  $\{(x_t,j,\ell_{t,j})\}_{j \in A_t}$  to the regression oracle  $\mathrm{Alg}_{\mathrm{Sq}}$ . Following a similar analysis to [Foster et al., 2021], we show that to bound the regret  $\mathrm{Reg}_{\mathrm{CB}}$ , we only need to bound  $\overline{\mathrm{dec}}_{\gamma}(p_t;\widehat{f}_t,x_t,G_t)$ .
Theorem 3.1. Suppose  $\overline{\mathrm{dec}}_{\gamma}(p_t; \widehat{f}_t, x_t, G_t) \leq C\gamma^{-\beta}$  for all  $t \in [T]$  and some  $\beta > 0$ . Algorithm 1 with  $\gamma = \max \{4, (CT)^{\frac{1}{\beta + 1}} \mathbf{Reg}_{\mathsf{Sq}}^{-\frac{1}{\beta + 1}}\}$  guarantees that  $\mathbb{E}[\mathbf{Reg}_{\mathsf{CB}}] \leq \mathcal{O}\left(C^{\frac{1}{\beta + 1}} T^{\frac{1}{\beta + 1}} \mathbf{Reg}_{\mathsf{Sq}}^{\frac{\beta}{\beta + 1}}\right)$ .
The proof is deferred to Appendix A. In Section 3.3, we give an efficient implementation for solving Eq. (4) via reduction to convex programming.
3.2 Regret Bounds
In this section, we derive regret bounds for Algorithm 1 when  $G_{t}$ 's are specialized to deterministic graphs, i.e.,  $G_{t} \in \{0,1\}^{\mathcal{A} \times \mathcal{A}}$ . We utilize discrete graph notation  $G = ([K], E)$ , where  $E \subseteq [K] \times [K]$ ; and define  $N^{\mathrm{in}}(G,j) \triangleq \{i \in \mathcal{A} : (i,j) \in E\}$  as the set of nodes that can observe node  $j$ . In this case, at each round  $t$ , the observed node set  $A_{t}$  is a deterministic set which contains any node  $i$  satisfying  $a_{t} \in N^{\mathrm{in}}(G_{t}, i)$ . In the following, we introduce several graph-theoretic concepts for deterministic feedback graphs [Alon et al., 2015].
Strongly/Weakly Observable Graphs. For a directed graph  $G = ([K], E)$ , a node  $i$  is observable if  $N^{\mathrm{in}}(G, i) \neq \emptyset$ . An observable node is strongly observable if either  $i \in N^{\mathrm{in}}(G, i)$  or  $N^{\mathrm{in}}(G, i) = [K] \backslash \{i\}$ , and weakly observable otherwise. Similarly, a graph is observable if all its nodes are observable. An observable graph is strongly observable if all nodes are strongly observable, and weakly observable otherwise. Self-aware graphs are a special type of strongly observable graphs where  $i \in N^{\mathrm{in}}(G, i)$  for all  $i \in [K]$ .
Independent Set and Weakly Dominating Set. An independence set of a directed graph is a subset of nodes in which no two distinct nodes are connected. The size of the largest independence set of a graph is called its independence number. For a weakly observable graph  $G = ([K], E)$ , a weakly dominating set is a subset of nodes  $D \subseteq [K]$  such that for any node  $j$  in  $G$  without a self-loop, there exists  $i \in D$  such that directed edge  $(i, j) \in E$ . The size of the smallest weakly dominating set of a graph is called its weak domination number. Alon et al. [2015] show that in non-contextual bandits with a fixed feedback graph  $G$ , the minimax regret bound is  $\widetilde{\Theta}(\sqrt{\alpha T})$  when  $G$  is strongly observable and  $\widetilde{\Theta}(d^{\frac{1}{3}}T^{\frac{2}{3}})$  when  $G$  is weakly observable, where  $\alpha$  and  $d$  are the independence number and the weak domination number of  $G$ , respectively.
3.2.1 Strongly Observable Graphs
In the following theorem, we show the regret bound of Algorithm 1 for strongly observable graphs. Theorem 3.2 (Strongly observable graphs). Suppose that the feedback graph  $G_{t}$  is deterministic and strongly observable with independence number no more than  $\alpha$ . Then Algorithm 1 guarantees that
$$
\overline {{\mathsf {d e c}}} _ {\gamma} (p _ {t}; \widehat {f} _ {t}, x _ {t}, G _ {t}) \leq \mathcal {O} \left(\frac {\alpha \log (K \gamma)}{\gamma}\right).
$$
In contrast to existing works that derive a closed-form solution of  $p_t$  in order to show how large the DEC can be [Foster and Rakhlin, 2020, Foster and Krishnamurthy, 2021], in our case we prove the upper bound of  $\overline{\mathrm{dec}}_{\gamma}(p_t; \hat{f}_t, x_t, G_t)$  by using the Sion's minimax theorem and the graph-theoretic lemma proven in [Alon et al., 2015]. The proof is deferred to Appendix A.1. Combining Theorem 3.2 and Theorem 3.1, we directly have the following corollary:
Corollary 3.3. Suppose that  $G_{t}$  is deterministic, strongly observable, and has independence number no more than  $\alpha$  for all  $t \in [T]$ . Algorithm 1 with choice  $\gamma = \max \left\{4, \sqrt{\alpha T / \mathbf{Reg}_{\mathbf{Sq}}}\right\}$  guarantees that
$$
\mathbb {E} [ \mathbf {R e g} _ {\mathsf {C B}} ] \leq \widetilde {\mathcal {O}} \Big (\sqrt {\alpha T \mathbf {R e g} _ {\mathsf {S q}}} \Big).
$$
For conciseness, we show in Corollary 3.3 that the regret guarantee for Algorithm 1 depends on the largest independence number of  $G_{t}$  over  $t \in [T]$ . However, we in fact are able to achieve a move adaptive regret bound of order  $\widetilde{\mathcal{O}}\left(\sqrt{\sum_{t=1}^{T} \alpha_{t} \mathbf{Reg}_{\mathbf{Sq}}}\right)$  where  $\alpha_{t}$  is the independence number of  $G_{t}$ . It is straightforward to achieve this by applying a standard doubling trick on the quantity  $\sum_{t=1}^{T} \alpha_{t}$ , assuming we can compute  $\alpha_{t}$  given  $G_{t}$ , but we take one step further and show that it is in fact unnecessary to compute  $\alpha_{t}$  (which, after all, is NP-hard [Karp, 1972]): we provide an adaptive tuning strategy for  $\gamma$  by keeping track the cumulative value of the quantity  $\min_{p \in \Delta(K)} \overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}_{t}, x_{t}, G_{t})$  and show that this efficient method also achieves the adaptive  $\widetilde{\mathcal{O}}\left(\sqrt{\sum_{t=1}^{T} \alpha_{t} \mathbf{Reg}_{\mathbf{Sq}}}\right)$  regret guarantee; see Appendix D for details.
3.2.2 Weakly Observable Graphs
For the weakly observable graph, we have the following theorem.
Theorem 3.4 (Weakly observable graphs). Suppose that the feedback graph  $G_{t}$  is deterministic and weakly observable with weak domination number no more than  $d$ . Then Algorithm 1 with  $\gamma \geq 16d$  guarantees that
$$
\overline {{\operatorname {d e c}}} _ {\gamma} \left(p _ {t}; \widehat {f} _ {t}, x _ {t}, G _ {t}\right) \leq \mathcal {O} \left(\sqrt {\frac {d}{\gamma}} + \frac {\widetilde {\alpha} \log (K \gamma)}{\gamma}\right),
$$
where  $\widetilde{\alpha}$  is the independence number of the subgraph induced by nodes with self-loops in  $G_{t}$ .
The proof is deferred to Appendix A.2. Similar to Theorem 3.2, we do not derive a closed-form solution to the strategy  $p_t$  but prove this upper bound using the minimax theorem. Combining Theorem 3.4 and Theorem 3.1, we are able to obtain the following regret bound for weakly observable graphs, whose proof is deferred to Appendix A.2.
Corollary 3.5. Suppose that  $G_{t}$  is deterministic, weakly observable, and has weak domination number no more than  $d$  for all  $t \in [T]$ . In addition, suppose that the independence number of the subgraph induced by nodes with self-loops in  $G_{t}$  is no more than  $\widetilde{\alpha}$  for all  $t \in [T]$ . Then, Algorithm 1 with  $\gamma = \max \{16d, \sqrt{\widetilde{\alpha} T / \mathbf{Reg}_{\mathsf{Sq}}}, d^{\frac{1}{3}}T^{\frac{2}{3}}\mathbf{Reg}_{\mathsf{Sq}}^{-\frac{2}{3}}\}$  guarantees that
$$
\mathbb {E} [ \mathbf {R e g} _ {\mathrm {C B}} ] \leq \widetilde {\mathcal {O}} \left(d ^ {\frac {1}{3}} T ^ {\frac {2}{3}} \mathbf {R e g} _ {\mathrm {S q}} ^ {\frac {1}{3}} + \sqrt {\widetilde {\alpha} T \mathbf {R e g} _ {\mathrm {S q}}}\right).
$$