SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Khaled Eldowa*
Universita degli Studi di Milano, Milan, Italy  
khaled.eldowa@unimi.it
Emmanuel Esposito*
Università degli Studi di Milano, Milan, Italy & Istituto Italiano di Tecnologia, Genoa, Italy emmanuel@emmanuelesposito.it
Tommaso Cesari
University of Ottawa, Ottawa, Canada tcesari@uottawa.ca
Università degli Studi di Milano, Milan, Italy & Politecnico di Milano, Milan, Italy nicolo.cesa-bianchi@unimi.it
In this work, we improve on the upper and lower bounds for the regret of online learning with strongly observable undirected feedback graphs. The best known upper bound for this problem is  $\mathcal{O}\big(\sqrt{\alpha T\ln K}\big)$ , where  $K$  is the number of actions,  $\alpha$  is the independence number of the graph, and  $T$  is the time horizon. The  $\sqrt{\ln K}$  factor is known to be necessary when  $\alpha = 1$  (the experts case). On the other hand, when  $\alpha = K$  (the bandits case), the minimax rate is known to be  $\Theta\big(\sqrt{KT}\big)$  and a lower bound  $\Omega\big(\sqrt{\alpha T}\big)$  is known to hold for any  $\alpha$ . Our improved upper bound  $\mathcal{O}\big(\sqrt{\alpha T(1 + \ln(K / \alpha))}\big)$  holds for any  $\alpha$  and matches the lower bounds for bandits and experts, while interpolating intermediate cases. To prove this result, we use FTRL with  $q$ -Tsallis entropy for a carefully chosen value of  $q \in [1/2, 1)$  that varies with  $\alpha$ . The analysis of this algorithm requires a new bound on the variance term in the regret. We also show how to extend our techniques to time-varying graphs, without requiring prior knowledge of their independence numbers. Our upper bound is complemented by an improved  $\Omega\big(\sqrt{\alpha T(\ln K) / (\ln \alpha)}\big)$  lower bound for all  $\alpha > 1$ , whose analysis relies on a novel reduction to multitask learning. This shows that a logarithmic factor is necessary as soon as  $\alpha < K$ .
Feedback graphs [29] provide an elegant interpolation between two popular online learning models: multiarmed bandits and prediction with expert advice. When learning with an undirected feedback graph  $G$  over  $K$  actions, the online algorithm observes not only the loss of the action chosen in each round, but also the loss of the actions that are adjacent to it in the graph. Two important special cases of this setting are: prediction with expert advice (when  $G$  is a clique) and  $K$ -armed bandits (when  $G$  has no edges). When losses are generated adversarially, the regret in the feedback graph setting with strong observability has been shown to scale with the independence number  $\alpha$  of  $G$ . Intuitively, denser graphs, which correspond to smaller independence numbers, provide more feedback to the learner, thus enabling a better control on regret. More specifically, the best known upper and lower bounds on the regret after  $T$  rounds are  $\mathcal{O}\big(\sqrt{\alpha T\log K}\big)$  and  $\Omega \big(\sqrt{\alpha T}\big)$  [3, 4]. It has been known for three decades that this upper bound is tight for  $\alpha = 1$  (the experts case, [9, 10]). When  $\alpha = K$
(the bandits case), the lower bound  $\Omega\big(\sqrt{KT}\big)$  —which has also been known for nearly three decades [7, 8]—was matched by a corresponding upper bound  $\mathcal{O}\big(\sqrt{KT}\big)$  only in 2009 [5]. These results show that in feedback graphs, the logarithmic factor  $\sqrt{\log K}$  is necessary (at least) for the  $\alpha = 1$  case, while it must vanish from the minimax regret as  $\alpha$  grows from 1 to  $K$ , but the current bounds fail to capture this fact. In this work, we prove new upper and lower regret bounds that for the first time account for this vanishing logarithmic factor.
To prove our new upper bound, we use the standard FTRL algorithm run with the  $q$ -Tsallis entropy regularizer ( $q$ -FTRL for short). It is well-known [1] that for  $q = \frac{1}{2}$  this algorithm (run with appropriate loss estimates) achieves regret  $\mathcal{O}\big(\sqrt{KT}\big)$  when  $\alpha = K$  (bandits case), while for  $q \to 1^{-}$  the same algorithm (without loss estimates) recovers the bound  $\mathcal{O}\big(\sqrt{T\log K}\big)$  when  $\alpha = 1$  (experts case). When  $G$  contains all self-loops, we show in Theorem 1 that, if  $q$  is chosen as a certain function  $q(\alpha, K)$ , then  $q(\alpha, K)$ -FTRL, run with standard importance-weighted loss estimates, achieves regret  $\mathcal{O}\big(\sqrt{\alpha T(1 + \log(K / \alpha))}\big)$ . This is a strict improvement over the previous bound, and matches the lower bounds for bandits and experts while interpolating the intermediate cases. This interpolation is reflected by our choice of  $q$ , which goes from  $\frac{1}{2}$  to 1 as  $\alpha$  ranges from 1 to  $K$ . The main technical hurdle in proving this result is an extension to arbitrary values of  $q \in \left[\frac{1}{2}, 1\right)$  of a standard result—see, e.g., [29, Lemma 3]—that bounds in terms of  $\alpha$  the variance term in the regret of  $q$ -FTRL. In Theorem 2, using a modified loss estimate, this result is extended to any strongly observable undirected graph [2], a class of feedback graphs in which some of the actions do not reveal their loss when played. In Theorem 3, we show via a doubling trick that our new upper bound can also be obtained (up to constant factors) without the need of knowing (or computing)  $\alpha$ . As the resulting algorithm is oblivious to  $\alpha$ , our analysis also applies to arbitrary sequences of graphs  $G_{t}$ , where  $K$  is constant but the independence number  $\alpha_{t}$  of  $G_{t}$  can change over time, and the algorithm observes  $G_{t}$  only after choosing an action (the so-called uninformed case). In this setting, the analysis of the doubling trick is complicated by the non-trivial dependence of the regret on the sequence of  $\alpha_{t}$ .
We also improve on the  $\Omega\left(\sqrt{\alpha T}\right)$  lower bound by proving a new  $\Omega\left(\sqrt{\alpha T \log_{\alpha} K}\right)$  lower bound for all  $\alpha > 1$ . This is the first result showing the necessity—outside the experts' case—of a logarithmic factor in the minimax regret for all  $\alpha < K$ . Our proof uses a stochastic adversary generating both losses and feedback graphs via i.i.d. draws from a joint distribution. This sequence of losses and feedback graphs can be used to define a hard instance of the multi-task bandits problem, a variant of the combinatorial bandits framework [12]. We then prove our result by adapting known lower bounding techniques for multi-task bandits [6]. Note that for values of  $\alpha$  bounded away from 2 and  $K$ , the logarithmic factor  $\log_{\alpha} K$  in the lower bound is smaller than the corresponding factor  $1 + \log(K / \alpha)$  in the upper bound. Closing this gap remains an open problem.
1.1 Additional related work
Several previous works have used the  $q$ -Tsallis regularizer with  $q$  tuned to specific values other than  $\frac{1}{2}$  and 1. For example, in [35, Section 4],  $q$  is chosen as a function of  $K$  to prove a regret bound of  $\mathcal{O}\left(\sqrt{\alpha T(\log K)^3}\right)$  for any strongly observable directed feedback graph, which shaves off a  $\log T$  factor compared to previous works. This bound is worse than the corresponding bounds for undirected graphs because the directed setting is harder. Specific choices of  $q$  have been considered to improve the regret in settings of online learning with standard bandit feedback. For example, the choice  $q = \frac{2}{3}$  was used in [31] to improve the analysis of regret in bandits with decoupled exploration and exploitation. Regret bounds for arbitrary choices of  $q$  are derived in [36, 23] for a best-of-both-worlds analysis of bandits, though  $q = \frac{1}{2}$  remains the optimal choice. The  $\frac{1}{2}$ -Tsallis entropy and the Shannon entropy ( $q = 1$ ) regularizers have been combined before in different ways to obtain best-of-both-worlds guarantees for the graph feedback problem [18, 22]. The idea of using values of  $q \in (\frac{1}{2}, 1)$  for feedback graphs is quite natural and has been brought up before, e.g., in [32], but achieving an improved dependence on the graph structure by picking a suitable value of  $q$  has not been, to the best of our knowledge, successfully pursued before. On the other hand, an approach based on a similar use of the  $q$ -Tsallis regularizer has been employed by [26] for the problem of multiarmed bandits with sparse losses to achieve a  $\mathcal{O}\left(\sqrt{sT\ln(K / s)}\right)$  regret bound, where  $s$  is the maximum number of nonzero losses at any round.
Our lower bound is reminiscent of the  $\Omega\left(\sqrt{KT\log_KN}\right)$  lower bound proved in [33] for the problem of bandits with expert advice (with  $N\geq K$  being the number of experts); see also [17] and [34]. In
that problem, at each time step, experts suggest distributions over actions to the learner, whose regret is computed against the best expert in hindsight. Although the two settings are different, the variant of the multitask bandit problem that our lower bound construction simulates is the same as the one used in the proof of [17, Theorem 7].
For any integer  $n \geq 1$ , let  $[n] = \{1, \ldots, n\}$ . We consider the following game played over  $T$  rounds between a learner with action set  $V = [K]$  and the environment. At the beginning of the game, the environment secretly selects a sequence of losses  $(\ell_t)_{t \in [T]}$ , where  $\ell_t \colon V \to [0,1]$ ,<sup>2</sup> and a sequence of undirected graphs  $(G_t)_{t \in [T]}$  over the set of actions  $V$ , that is,  $G_t = (V, E_t)$ . At any time  $t$ , the learner selects an arm  $I_t$  (possibly at random), then pays loss  $\ell_t(I_t)$  and observes the feedback graph  $G_t$  and all losses  $\ell_t(i)$  of neighbouring actions  $i \in N_{G_t}(I_t)$ , where  $N_{G_t}(i) = \{j \in V : (i,j) \in E_t\}$  (see Online Protocol 1). In this work, we only focus on strongly observable graphs [2]. An undirected graph  $G$  is strongly observable if for every  $i \in V$ , at least one of the following holds:  $i \in N_G(i)$  or  $i \in N_G(j)$  for all  $j \neq i$ .
The performance of the learner is measured by the regret
$$
R _ {T} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell_ {t} \left(I _ {t}\right) \right] - \min  _ {i \in [ K ]} \sum_ {t = 1} ^ {T} \ell_ {t} (i).
$$
where the expectation is over the learner's internal randomization.
Online Protocol 1 Online learning with feedback graphs
environment: (hidden) losses  $\ell_t\colon V\to [0,1]$  and graphs  $G_{t} = (V,E_{t})$  , for all  $t = 1,\dots ,T$    
for  $t = 1,\ldots ,T$  do The learner picks an action.  $I_{t}\in V$  (possibly at random) The learner incurs loss  $\ell_t(I_t)$  The learner observes losses  $\{(i,\ell_t(i)):i\in N_{G_t}(I_t)\}$  and graph  $G_{t}$    
end for
We denote by  $\Delta_K$  the simplex  $\{p\in [0,1]^K:\| p\| _1 = 1\}$ . For any graph  $G$ , we define its independence number as the cardinality of the largest set of nodes such that no two nodes are neighbors, and denote it by  $\alpha (G)$ . For simplicity, we use  $N_{t}$  to denote the neighbourhood  $N_{G_t}$  in the graph  $G_{t}$  and we use  $\alpha_{t}$  to denote the independence number  $\alpha (G_t)$  of  $G_{t}$  at time  $t$ .
3 FTRL with Tsallis Entropy for Undirected Feedback Graphs
As a building block, in this section, we focus on the case when all the feedback graphs  $G_{1}, \ldots, G_{T}$  have the same independence number  $\alpha_{1} = \dots = \alpha_{T} = \alpha$ , whereas the general case is treated in the next section. For simplicity, we start with the assumption that all nodes have self-loops:  $(i, i) \in E_{t}$  for all  $i \in V$  and all  $t$ . We later lift this requirement and show that the regret guarantees that we provide can be extended to general strongly observable undirected feedback graphs, only at the cost of a constant multiplicative factor.
The algorithm we analyze is  $q$ -FTRL (described in Algorithm 1), which is an instance of the following regularized leader (FTRL) framework—see, e.g., [30, Chapter 7]—with the (negative)  $q$ -Tsallis entropy
$$
\psi_ {q} (x) = \frac {1}{1 - q} \left(1 - \sum_ {i \in V} x (i) ^ {q}\right) \quad \forall x \in \Delta_ {K},
$$
as the regularizer, whose parameter  $q \in (0,1)$  can be tuned according to our needs. Since we do not observe all the losses in a given round, the algorithm makes use of unbiased estimates for the losses. When all self-loops are present, we define the estimated losses in the following standard manner. Let
For notational convenience, we will sometimes treat the loss functions  $\ell_t\colon V\to [0,1]$  as vectors with components  $\ell_t(1),\ldots ,\ell_t(K)$ .
$I_{t}$  be the action picked at round  $t$ , which is drawn from the distribution  $p_t \in \Delta_K$  maintained by the algorithm, the loss estimate for an action  $i \in V$  at round  $t$  is given by
$$
\widehat {\ell} _ {t} (i) = \frac {\ell_ {t} (i)}{P _ {t} (i)} \mathbb {I} \left\{I _ {t} \in N _ {t} (i) \right\}, \tag {1}
$$
where  $P_{t}(i) = \mathbb{P}\big(I_{t} \in N_{t}(i)\big) = \sum_{j \in N_{t}(i)} p_{t}(j)$ . This estimate is unbiased in the sense that  $\mathbb{E}_{t}\left[\widehat{\ell}_{t}(i)\right] = \ell_{t}(i)$  for all  $t \in [T]$  and all  $i \in V$ , where we denote  $\mathbb{E}_{t}[\cdot] = \mathbb{E}[\cdot | I_{1}, \ldots, I_{t-1}]$ .
Algorithm 1  $q$ -FTRL for undirected feedback graphs
input:  $q\in (0,1)$ $\eta >0$    
initialization:  $p_1(i)\gets 1 / K$  for all  $i = 1,\dots ,K$    
for  $t = 1,\ldots ,T$  do Select action  $I_{t}\sim p_{t}$  and incur loss  $\ell_t(I_t)$  Observe losses  $\{(i,\ell_t(i)):i\in N_t(I_t)\}$  and graph  $G_{t}$  Construct a loss estimate  $\widehat{\ell}_t$  for  $\ell_t$  e.g., (1) or (6) Let  $p_{t + 1}\leftarrow \arg \min_{p\in \Delta_K}\eta \langle \sum_{s = 1}^t\widehat{\ell}_s,p\rangle +\psi_q(p)$    
end for
A key part of the standard regret analysis of  $q$ -FTRL (see, e.g., the proof of Lemma 3 in Appendix A) is handling the variance term, which, with the choice of estimator given in (1), takes the following form
$$
B _ {t} (q) = \sum_ {i \in V} \frac {p _ {t} (i) ^ {2 - q}}{P _ {t} (i)}. \tag {2}
$$
By Hölder's inequality, this term can be immediately upper bounded by
$$
B _ {t} (q) \leq \sum_ {i \in V} p _ {t} (i) ^ {1 - q} \leq \left(\sum_ {i \in V} p _ {t} (i)\right) ^ {1 - q} \left(\sum_ {i \in V} 1 ^ {1 / q}\right) ^ {q} = K ^ {q},
$$
while previous results on the regret analysis of multiarmed bandits with graph feedback [29, 3] would give
$$
B _ {t} (q) \leq \sum_ {i \in V} \frac {p _ {t} (i)}{P _ {t} (i)} \leq \alpha .
$$
However, the former result would only recover a  $\mathcal{O}(\sqrt{KT})$  regret bound (regardless of  $\alpha$ ) with the best choice of  $q = 1/2$ , which could be trivially achieved by ignoring side-observations of the losses, whereas the latter bound would only manage to achieve a  $\mathcal{O}(\sqrt{\alpha T \ln K})$  regret bound, incurring the extra  $\sqrt{\ln K}$  factor for all values of  $\alpha$ . Other results in the literature (e.g., see [2, 4, 16, 19, 22, 25, 32, 35]) do not bring an improvement in this setting when bounding the  $B_{t}(q)$  term and, hence, do not suffice for achieving the desired regret bound. The following lemma provides a novel and improved bound on quantities of the same form as  $B_{t}(q)$  in terms of the independence number  $\alpha_{t} = \alpha$  of the undirected graph  $G_{t}$ .
Lemma 1. Let  $G = (V, E)$  be any undirected graph with  $|V| = K$  vertices and independence number  $\alpha(G) = \alpha$ . Let  $b \in [0,1]$ ,  $p \in \Delta_K$  and consider any nonempty subset  $U \subseteq \{v \in V : v \in N_G(v)\}$ . Then,
$$
\sum_ {v \in U} \frac {p (v) ^ {1 + b}}{\sum_ {u \in N _ {G} (v)} p (u)} \leq \alpha^ {1 - b}.
$$
Proof. First of all, observe that we can restrict ourselves to the subgraph  $G[U]$  induced by  $U$ , i.e., the graph  $G[U] = (U, E \cap (U \times U))$ . This is because the neighbourhoods in this graph are such that  $N_{G[U]}(v) \subseteq N_G(v)$  for all  $v \in U$ , and its independence number is  $\alpha(G[U]) \leq \alpha(G)$ . Hence, it suffices to prove the claimed inequality for any undirected graph  $G = (V, E)$  with all self-loops, any  $p \in [0, 1]^K$  such that  $\| p \|_1 \leq 1$ , and the choice  $U = V$ . We assume this in what follows without loss of generality.
For any subgraph  $H \subseteq G$  with vertices  $V(H) \subseteq V$ , denote the quantity we want to upper bound by
$$
Q (H) = \sum_ {v \in V (H)} \frac {p (v) ^ {1 + b}}{\sum_ {u \in N _ {G} (v)} p (u)}.
$$
Our aim is thus to provide an upper bound to  $Q(G)$ .
Consider a greedy algorithm that incrementally constructs a subset of vertices in the following way: at each step, it selects a vertex  $v$  that maximizes  $p(v)^b / \left( \sum_{u \in N_G(v)} p(u) \right)$ , it adds  $v$  to the solution, and it removes  $v$  from  $G$  together with its neighbourhood  $N_G(v)$ . This step is iterated on the remaining graph until no vertex is left.
Let  $S = \{v_{1},\ldots ,v_{s}\} \subseteq V$  be the solution returned by the above greedy algorithm on  $G$ . Also let  $G_{1},\dots,G_{s + 1}$  be the sequence of graphs induced by the operations of the algorithm, where  $G_{1} = G$  and  $G_{s + 1}$  is the empty graph, and let  $N_{r}(v) = N_{G_{r}}(v)$  for  $v\in V(G_r)$ . At every step  $r\in [s]$  of the greedy algorithm, the contribution to  $Q(G)$  of the removed vertices  $N_{r}(v_{r})$  amounts to
$$
\begin{array}{l} Q \left(G _ {r}\right) - Q \left(G _ {r + 1}\right) = \sum_ {v \in N _ {r} \left(v _ {r}\right)} \frac {p (v) ^ {1 + b}}{\sum_ {u \in N _ {1} (v)} p (u)} \leq \sum_ {v \in N _ {r} \left(v _ {r}\right)} p (v) \frac {p \left(v _ {r}\right) ^ {b}}{\sum_ {u \in N _ {1} \left(v _ {r}\right)} p (u)} \\ \leq \frac {\sum_ {v \in N _ {1} (v _ {r})} p (v)}{\sum_ {u \in N _ {1} (v _ {r})} p (u)} p (v _ {r}) ^ {b} = p (v _ {r}) ^ {b}, \\ \end{array}
$$
where the last inequality is due to the fact that  $N_{i}(v) \subseteq N_{j}(v)$  for all  $i \geq j$  and  $v \in V_{i}$ . Therefore, we can observe that
$$
Q (G) = \sum_ {r = 1} ^ {s} \left(Q \left(G _ {r}\right) - Q \left(G _ {r + 1}\right)\right) \leq \sum_ {v \in S} p (v) ^ {b}.
$$
The solution  $S$  is an independent set of  $G$  by construction. Consider now any independent set  $A \subseteq V$  of  $G$ . We have that
$$
\begin{array}{l} \sum_ {v \in A} p (v) ^ {b} \leq \max  _ {x \in \Delta_ {K}} \sum_ {v \in A} x (v) ^ {b} = | A | \max  _ {x \in \Delta_ {K}} \sum_ {v \in A} \frac {x (v) ^ {b}}{| A |} \\ \leq | A | \max  _ {x \in \Delta_ {K}} \left(\frac {1}{| A |} \sum_ {v \in A} x (v)\right) ^ {b} \leq | A | ^ {1 - b} \leq \alpha^ {1 - b}, \tag {3} \\ \end{array}
$$
where the second inequality follows by Jensen's inequality and the fact that  $b \in [0,1]$ .
Observe that this upper bound is tight for general probability distributions  $p \in \Delta_K$  over the vertices  $V$  of any strongly observable undirected graph  $G$  (containing at least one self-loop), as it is exactly achieved by the distribution  $p^{\star} \in \Delta_K$  defined as  $p^{\star}(i) = \frac{1}{|S|}\mathbb{I}\{i \in S\}$  for some maximum independent set  $S \subseteq V$  of  $G$ . Using this lemma, the following theorem provides our improved upper bound under the simplifying assumptions we made thus far.
Theorem 1. Let  $G_{1}, \ldots, G_{T}$  be a sequence of undirected feedback graphs, where each  $G_{t}$  contains all self-loops and has independence number  $\alpha_{t} = \alpha$  for some common value  $\alpha \in [K]$ . If Algorithm 1 is run with input
$$
q = \frac {1}{2} \left(1 + \frac {\ln (K / \alpha)}{\sqrt {\ln (K / \alpha) ^ {2} + 4} + 2}\right) \in [ 1 / 2, 1) \quad a n d \quad \eta = \sqrt {\frac {2 q K ^ {1 - q}}{T (1 - q) \alpha^ {q}}} ,
$$
and loss estimates (1), then its regret satisfies  $R_{T} \leq 2\sqrt{e\alpha T(2 + \ln(K / \alpha))}$ .
Proof. One can verify that for any  $i \in V$ , the loss estimate  $\widehat{\ell_t}(i)$  defined in (1) satisfies  $\mathbb{E}_t[\widehat{\ell_t}(i)^2] \leq 1 / P_t(i)$ . Hence, using also that  $\mathbb{E}_t[\widehat{\ell_t}(i)] = \ell_t(i)$ , Lemma 2 in Appendix A implies that
$$
\begin{array}{l} R _ {T} \leq \frac {K ^ {1 - q}}{\eta (1 - q)} + \frac {\eta}{2 q} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \sum_ {i \in V} \frac {p _ {t} (i) ^ {2 - q}}{P _ {t} (i)} \right] (4) \\ \leq \frac {K ^ {1 - q}}{\eta (1 - q)} + \frac {\eta}{2 q} \alpha^ {q} T, (5) \\ \end{array}
$$
where the second inequality follows by Lemma 1 with  $b = 1 - q$  since all actions  $i \in V$  are such that  $i \in N_G(i)$ . Our choices for  $q$  and  $\eta$  allow us to further upper bound the right-hand side of (5) by
$$
\begin{array}{l} \sqrt {\frac {2 K ^ {1 - q} \alpha^ {q}}{q (1 - q)} T} = \sqrt {2 T \exp \left(1 + \frac {1}{2} \ln (\alpha K) - \frac {1}{2} \sqrt {\ln (K / \alpha) ^ {2} + 4}\right) \left(2 + \sqrt {\ln (K / \alpha) ^ {2} + 4}\right)} \\ \leq \sqrt {2 e \alpha T \left(2 + \sqrt {\ln (K / \alpha) ^ {2} + 4}\right)} \leq 2 \sqrt {e \alpha T \sqrt {\ln (K / \alpha) ^ {2} + 4}} \\ \leq 2 \sqrt {e \alpha T (2 + \ln (K / \alpha))}. \\ \end{array}
$$
The regret bound achieved in the above theorem achieves the optimal regret bound for the experts setting (i.e.,  $\alpha = 1$ ) and the bandits setting (i.e.,  $\alpha = K$ ) simultaneously. Moreover, it interpolates the intermediate cases for  $\alpha$  ranging between 1 and  $K$ , introducing the multiplicative logarithmic factor only for graphs with independence number strictly smaller than  $K$ . We remark that the chosen values of  $q$  and  $\eta$  do in fact minimize the right-hand side of (5). Note that we relied on the knowledge of  $\alpha$  to tune the parameter  $q$ . This is undesirable in general. We will show how to lift this requirement in Section 4. The same comment applies to Theorem 2, below.
We now show how to achieve the improved regret bound of Theorem 1 in the case of strongly observable undirected feedback graphs where some self-loops may be missing; i.e., there may be actions  $i \in V$  such that  $i \notin N_G(i)$ . Using the loss estimator defined in (1) may lead to a large variance term due to the presence of actions without self-loops. One approach to deal with this—see, e.g., [35] or [28—is to suitably alter the loss estimates of these actions.
Define  $S_{t} = \{i\in V:i\notin N_{t}(i)\}$  as the subset of actions without self-loops in the feedback graph  $G_{t}$  at each time step  $t\in [T]$ . The idea is that we need to carefully handle some action  $i\in S_{t}$  only in the case when the probability  $p_t(i)$  of choosing  $i$  at round  $t$  is sufficiently large, say, larger than  $1 / 2$ . Define the set of such actions as  $J_{t} = \{i\in S_{t}:p_{t}(i) > 1 / 2\}$  and observe that  $|J_{t}|\leq 1$ . Similarly to [35], define new loss estimates
$$
\widehat {\ell} _ {t} (i) = \left\{ \begin{array}{l l} \frac {\ell_ {t} (i)}{P _ {t} (i)} \mathbb {I} \left\{I _ {t} \in N _ {t} (i) \right\} & \text {i f} i \in V \backslash J _ {t} \\ \frac {\ell_ {t} (i) - 1}{P _ {t} (i)} \mathbb {I} \left\{I _ {t} \in N _ {t} (i) \right\} + 1 & \text {i f} i \in J _ {t} \end{array} \right. \tag {6}
$$
for which it still holds that  $\mathbb{E}_t[\widehat{\ell}_t] = \ell_t$  and that  $\mathbb{E}_t[\widehat{\ell}_t(i)^2 ]\leq 1 / P_t(i)$  for all  $i\notin J_{t}$ . This change, along with the use of Lemma 1 for the actions in  $V\setminus S_{t}$ , suffices in order to prove the following regret bound (see Appendix B for the proof) when the feedback graphs do not necessarily contain self-loops for all actions.
Theorem 2. Let  $G_{1}, \ldots, G_{T}$  be a sequence of strongly observable undirected feedback graphs, where each  $G_{t}$  has independence number  $\alpha_{t} = \alpha$  for some common value  $\alpha \in [K]$ . If Algorithm 1 is run with input
$$
q = \frac {1}{2} \bigg (1 + \frac {\ln (K / \alpha)}{\sqrt {\ln (K / \alpha) ^ {2} + 4} + 2} \bigg) \in [ 1 / 2, 1) \qquad a n d \qquad \eta = \frac {1}{3} \sqrt {\frac {2 q K ^ {1 - q}}{T (1 - q) \alpha^ {q}}} ,
$$
and loss estimates (6), then its regret satisfies  $R_{T} \leq 6\sqrt{e\alpha T(2 + \ln(K / \alpha))}$ .
4 Adapting to Arbitrary Sequences of Graphs