SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the previous section, we assumed for simplicity that all the graphs have the same independence number. This independence number was then used to tune  $q$ , the parameter of the Tsallis entropy regularizer used by the algorithm. In this section, we show how to extend our approach to the case when the independence numbers of the graphs are neither the same nor known a-priori by the learner. Had these independence numbers been known a-priori, one approach is to set  $q$  as in Theorem 2 but using the average independence number
$$
\bar {\alpha} _ {T} = \frac {1}{T} \sum_ {t = 1} ^ {T} \alpha_ {t}.
$$
Doing so would allow us to achieve a  $\mathcal{O}\Big(\sqrt{\sum_{t = 1}^{T}\alpha_{t}(1 + \ln(K / \bar{\alpha}_{T}))}\Big)$  regret bound. We now show that we can still recover a bound of the same order without prior knowledge of  $\bar{\alpha}_T$ . For round  $t$  and any fixed  $q\in [0,1]$ , define
$$
H _ {t} (q) = \sum_ {i \in V \backslash S _ {t}} \frac {p _ {t} (i) ^ {2 - q}}{P _ {t} (i)} .
$$
We know from Lemma 1 that  $H_{t}(q) \leq \alpha_{t}^{q}$ . Thus, we can leverage these observations and use a doubling trick (similar in principle to [3]) to guess the value of  $\bar{\alpha}_{T}$ . This approach is outlined in Algorithm 2. Starting with  $r = 0$  and  $T_{r} = 1$ , the idea is to instantiate Algorithm 1 at time-step  $T_{r}$  with  $q$  and  $\eta$  set as in Theorem 2 but with  $2^{r}$  replacing the independence number. Then, at  $t \geq T_{r}$ , we increment  $r$  and restart Algorithm 1 only if
$$
\frac {1}{T} \sum_ {s = T _ {r}} ^ {t} H _ {s} \left(q _ {r}\right) ^ {1 / q _ {r}} > 2 ^ {r + 1},
$$
since (again thanks to Lemma 1) the left-hand side of the above inequality is always upper bounded by  $\bar{\alpha}_T$ . The following theorem shows that this approach essentially enjoys the same regret bound of Theorem 2 up to an additive  $\log_2\bar{\alpha}_T$  term.
Algorithm 2  $q$ -FTRL for an arbitrary sequence of strongly observable undirected graphs
input: Time horizon  $T$
define: For each  $r\in \{0,\dots ,\lfloor \log_2K\rfloor \}$
$$
q _ {r} = \frac {1}{2} \left(1 + \frac {\ln (K / 2 ^ {r})}{\sqrt {\ln (K / 2 ^ {r}) ^ {2} + 4} + 2}\right)
$$
$$
\eta_ {r} = \sqrt {\frac {2 q _ {r} K ^ {1 - q _ {r}}}{1 1 T (1 - q _ {r}) (2 ^ {r}) ^ {q _ {r}}}}
$$
initialization:  $T_0 \gets 1$ ,  $r \gets 0$ , instantiate Algorithm 1 with  $q = q_0$ ,  $\eta = \eta_0$ , and loss estimates (6) for  $t = 1, \ldots, T$  do
Perform one step of the current instance of Algorithm 1
if  $\frac{1}{T}\sum_{s = T_r}^{t}H_s(q_r)^{1 / q_r} > 2^{r + 1}$  then
$$
r \leftarrow r + 1
$$
$$
T _ {r} \leftarrow t + 1
$$
Restart Algorithm 1 with  $q = q_r$ ,  $\eta = \eta_r$ , and loss estimates (6)
Theorem 3. Let  $C = 4\sqrt{6e}\frac{\sqrt{\pi} + \sqrt{4 - 2\ln 2}}{\ln 2}$ . Then, the regret of Algorithm 2 satisfies
$$
R _ {T} \leq C \sqrt {\sum_ {t = 1} ^ {T} \alpha_ {t} \left(2 + \ln \left(\frac {K}{\bar {\alpha} _ {T}}\right)\right)} + \log_ {2} \bar {\alpha} _ {T}.
$$
Proof sketch. For simplicity, we sketch here the proof for the case when in every round  $t$ , all the nodes have self-loops; hence,  $H_{t}(q) = B_{t}(q)$ . See the full proof in Appendix C, which treats the general case in a similar manner. Let  $n = \lceil \log_2\bar{\alpha}_T\rceil$  and assume without loss of generality that  $\bar{\alpha}_{T} > 1$ . Since Lemma 1 implies that for any  $r$  and  $t$ ,  $B_{t}(q_{r})\leq \alpha_{t}^{q_{r}}$ , we have as a consequence that for any  $t\geq T_r$
$$
\frac {1}{T} \sum_ {s = T _ {r}} ^ {t} B _ {s} \left(q _ {r}\right) ^ {1 / q _ {r}} \leq \frac {1}{T} \sum_ {s = T _ {r}} ^ {t} \alpha_ {s} \leq \bar {\alpha} _ {T} \leq 2 ^ {n}.
$$
Hence, the maximum value of  $r$  that the algorithm can reach is  $n - 1$ . In doing so, we will execute  $n$  instances of Algorithm 1, each corresponding to a value of  $r \in \{0, \dots, n - 1\}$ . For every such  $r$ , we upper bound the instantaneous regret at step  $T_{r + 1} - 1$  (the step when the restarting condition is satisfied) by 1, hence the added  $\log_2 \bar{\alpha}_T$  term in the regret bound. For the rest of the interval; namely, for  $t \in [T_r, T_{r + 1} - 2]$ , we have via (4) that the regret of Algorithm 1 is bounded by
$$
\frac {K ^ {1 - q _ {r}}}{\eta_ {r} \left(1 - q _ {r}\right)} + \frac {\eta_ {r}}{2 q _ {r}} \mathbb {E} \sum_ {t = T _ {r}} ^ {T _ {r + 1} - 2} B _ {t} \left(q _ {r}\right). \tag {7}
$$
Define  $T_{r:r+1} = T_{r+1} - T_r - 1$ , and notice that
$$
\begin{array}{l} \sum_ {t = T _ {r}} ^ {T _ {r + 1} - 2} B _ {t} \left(q _ {r}\right) \leq T _ {r: r + 1} \left(\frac {1}{T _ {r : r + 1}} \sum_ {t = T _ {r}} ^ {T _ {r + 1} - 2} B _ {t} \left(q _ {r}\right) ^ {1 / q _ {r}}\right) ^ {q _ {r}} \\ \leq T _ {r: r + 1} \left(\frac {T}{T _ {r : r + 1}} 2 ^ {r + 1}\right) ^ {q _ {r}} \leq 2 T \left(2 ^ {r}\right) ^ {q _ {r}}, \\ \end{array}
$$
where the first inequality follows due to Jensen's inequality since  $q_{r} \in (0,1)$ , and the second follows from the restarting condition of Algorithm 2. After, plugging this back into (7), we can simply use the definitions of  $\eta_r$  and  $q_{r}$  and bound the resulting expression in a similar manner to the proof of Theorem 1. Overall, we get that
$$
R _ {T} \leq 4 \sqrt {3 e T} \sum_ {r = 0} ^ {n - 1} \sqrt {2 ^ {r} \ln \left(e ^ {2} K 2 ^ {- r}\right)} + \log_ {2} \bar {\alpha} _ {T},
$$
from which the theorem follows by using Lemma 4 in Appendix A, which shows, roughly speaking, that the sum on the right-hand side is of the same order as its last term.
Although Algorithm 2 requires knowledge of the time horizon, this can be dealt with by applying a standard doubling trick on  $T$  at the cost of a larger constant factor. It is also noteworthy that the bound we obtained is of the form  $\sqrt{T\bar{\alpha}_T(1 + \ln(K / \bar{\alpha}_T))}$  and not  $\sqrt{\sum_t\alpha_t(1 + \ln(K / \alpha_t))}$ . Although both coincide with the bound of Theorem 2 when  $\alpha_{t}$  is the same for all time steps, the latter is smaller via the concavity of  $x(1 + \ln (K / x))$  in  $x$ . It is not clear, however, whether there is a tuning of  $q\in (0,1)$  that can achieve the second bound (even with prior knowledge of the entire sequence  $\alpha_{1},\ldots ,\alpha_{T}$  of independence numbers).
5 An Improved Lower Bound via Multitask Learning
In this section we provide a new lower bound on the minimax regret showing that, apart from the bandits case, a logarithmic factor is indeed necessary in general. When the graph is fixed over time, it is known that a lower bound of order  $\sqrt{\alpha T}$  holds for any value of  $\alpha$  [3, 29]. Whereas for the experts case  $(\alpha = 1)$ , the minimax regret is of order  $\sqrt{T \ln K}$  [9]. The following theorem provides, for the first time, a lower bound that interpolates between the two aforementioned bounds for the intermediate values of  $\alpha$ .
Theorem 4. Pick any  $K \geq 2$  and any  $\alpha$  such that  $2 \leq \alpha \leq K$ . Then, for any algorithm and for all  $T \geq \frac{\alpha \log_{\alpha} K}{4 \log(4/3)}$ , there exists a sequence of losses and feedback graphs  $G_1, \ldots, G_T$  such that  $\alpha(G_t) = \alpha$  for all  $t = 1, \ldots, T$  and
$$
R _ {T} \geq \frac {1}{1 8 \sqrt {2}} \sqrt {\alpha T \log_ {\alpha} K}.
$$
In essence, the proof of this theorem (see Appendix D) constructs a sequence of feedback graphs and losses that is equivalent to a hard instance of the multitask bandit problem (MTB) [12], an important special case of combinatorial bandits with a convenient structure for proving lower bounds [6, 15, 21]. We consider a variant of MTB in which, at the beginning of each round, the decision-maker selects an arm to play in each one of  $M$  stochastic bandit games. Subsequently, the decision-maker only observes (and suffers) the loss of the arm played in a single randomly selected game. For proving the lower bound, we use a class of stationary stochastic adversaries (i.e., environments), each generating graphs and losses in a manner that simulates an MTB instance.
Fix  $2 \leq \alpha \leq K = |V|$  and assume for simplicity that  $M = \log_{\alpha} K$  is an integer. We now construct an instance of online learning with time-varying feedback graphs  $G_t = (V, E_t)$  with  $\alpha(G_t) = \alpha$  that is equivalent to an MTB instance with  $M$  bandit games each containing  $\alpha$  "base actions". Since  $K = \alpha^M$ , we can uniquely identify each action in  $V$  with a vector  $a = (a(1), \ldots, a(M))$  in
<sup>3</sup>As a lower bound, this is known to hold asymptotically as  $K$  and  $T$  grow. However, it can also be shown to hold non-asymptotically (though with worse leading constants); see [20, Theorem 3.22] or [11, Theorem 3.6].
Figure 1: This figure shows an example of the multi-task bandit construction used to prove the lower bound. Here,  $K = 8$  and  $\alpha = 2$ ; thus, the number of games is  $M = 3$ . Each action is identified by a tuple of three numbers, each corresponding to a choice of one out of a pair of "base actions" in each game. Each of the three graphs in the figure corresponds to a game, such that two actions share an edge if and only if they choose the same base action in the corresponding game. At every round, a graph is randomly drawn, and all actions belonging to the same clique suffer the same loss.
$[\alpha]^M$ . The action  $a_t \in V$  chosen by the learner at round  $t$  is equivalent to a choice of base actions  $a_t(1), \ldots, a_t(M)$  in the  $M$  games. The feedback graph at every round is sampled uniformly at random from a set of  $M$  undirected graphs  $\{G^i\}_{i=1}^M$ , where  $G^i = (V, E^i)$  is such that  $(a, a') \in E^i$  if and only if  $a(i) = a'(i)$ . This means (see Figure 1) that each graph  $G^i$  consists of  $\alpha$  isolated cliques  $\{C_{i,j}\}_{j=1}^{\alpha}$  such that an action  $a$  belongs to clique  $C_{i,j}$  if and only if  $a(i) = j$ . Clearly, the independence number of any such graph is  $\alpha$ . Drawing feedback graph  $G_t = G^i$  corresponds to the activation of game  $i$  in the MTB instance. Hence, choosing  $a_t \in V$  with feedback graph  $G_t = G^i$  is equivalent to playing base action  $a_t(i)$  in game  $i$  in the MTB. As for the losses, we enforce that, given a feedback graph  $G_t$ , all actions that belong to the same clique of the feedback graph are assigned the same loss. Namely, if  $G_t = G^i$  and  $a(i) = a'(i) = j$ , then  $\ell_t(a) = \ell_t(a')$ , which can be seen as the loss  $\ell_t(j)$  assigned to base action  $j$  in game  $G^i$ . To choose the distribution of the losses for the base actions, we apply the classic needle-in-a-haystack approach of [7] over the  $M$  games. More precisely, we construct a different environment for each action  $a \in V$  in such a way that the distribution of the losses in each MTB game slightly favors (with a difference of a small  $\varepsilon > 0$ ) the base action corresponding to  $a$  in that game. The proof then proceeds similarly to, for example, the proof of Theorem 5 in [6] or Theorem 7 in [17].
While both our upper and lower bounds achieve the desired goal of interpolating between the minimax rates of experts and bandits, the logarithmic factors in the two bounds are not exactly matching. In particular, if we compare  $1 + \log_2(K / \alpha)$  and  $\log_{\alpha}K$ , we can see that although they coincide at  $\alpha = 2$  and  $\alpha = K$ , the former is larger for intermediate values. It is reasonable to believe that the upper bound is of the correct order, seeing as it arose naturally as a result of choosing the best parameter for the Tsallis entropy regularizer, whereas achieving the extra logarithmic term in the lower bound required a somewhat contrived construction.
In this work, we have shown that a proper tuning of the  $q$ -FTRL algorithm allows one to achieve a  $\mathcal{O}\big(\sqrt{\alpha T(1 + \ln(K / \alpha))}\big)$  regret for the problem of online learning with undirected strongly observable feedback graphs. Our bound interpolates between the minimax regret rates of the bandits and the experts problems, the two extremes of the strongly observable graph feedback spectrum.
Furthermore, we have shown that an analogous bound can be achieved when the graphs vary over time, and without requiring any prior knowledge on the graphs. These results are complemented by our new lower bound of  $\Omega\left(\sqrt{\alpha T(\ln K) / (\ln\alpha)}\right)$ , which holds for  $\alpha \geq 2$  and shows the necessity of a logarithmic factor in the minimax regret except for the bandits case. While our results provide the tightest characterization to date of the minimax rate for this setting, closing the small remaining gap (likely on the lower bound side) is an interesting problem. After the submission of this manuscript, a subsequent work [14] showed a lower bound for fixed feedback graphs composed of disjoint cliques that would imply worst-case optimality (up to constant factors) of our proposed algorithm for each pair of  $K$  and  $\alpha$ —see Appendix E for a more detailed comparison with results therein. Extending our results to the case of directed strongly observable feedback graphs is a considerably harder task—see Appendix F for a preliminary discussion. Better understanding this more general setting is an interesting future direction.
KE, EE, and NCB gratefully acknowledge the financial support from the MUR PRIN grant 2022EKNE5K (Learning in Markets and Society), funded by the NextGenerationEU program within the PNRR scheme (M4C2, investment 1.1), the FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRR-PE-AI scheme (M4C2, investment 1.3, line on Artificial Intelligence), and the EU Horizon CL4-2022-HUMAN-02 research and innovation action under grant agreement 101120237, project ELIAS (European Lighthouse of AI for Sustainability). TC gratefully acknowledges the support of the University of Ottawa through grant GR002837 (Start-Up Funds) and that of the Natural Sciences and Engineering Research Council of Canada (NSERC) through grants RGPIN-2023-03688 (Discovery Grants Program) and DGECR-2023-00208 (Discovery Grants Program, DGECR - Discovery Launch Supplement).
[1] J. D. Abernethy, C. Lee, and A. Tewari. Fighting bandits with a new kind of smoothness. Advances in Neural Information Processing Systems, 28, 2015.
[2] N. Alon, N. Cesa-Bianchi, O. Dekel, and T. Koren. Online learning with feedback graphs: Beyond bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of the 28th Conference on Learning Theory, volume 40 of JMLR Workshop and Conference Proceedings, pages 23-35. JMLR.org, 2015.
[3] N. Alon, N. Cesa-Bianchi, C. Gentile, S. Mannor, Y. Mansour, and O. Shamir. Nonstochastic multi-armed bandits with graph-structured feedback. SIAM Journal on Computing, 46(6):1785–1826, 2017.
[4] N. Alon, N. Cesa-Bianchi, C. Gentile, and Y. Mansour. From bandits to experts: A tale of domination and independence. Advances in Neural Information Processing Systems, 26, 2013.
[5] J.-Y. Audibert and S. Bubeck. Minimax policies for adversarial and stochastic bandits. In  $COLT$ , volume 7, pages 1-122, 2009.
[6] J.-Y. Audibert, S. Bubeck, and G. Lugosi. Regret in online combinatorial optimization. Mathematics of Operations Research, 39(1):31-45, 2014.
[7] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of IEEE 36th annual foundations of computer science, pages 322–331. IEEE, 1995.
[8] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.
[9] N. Cesa-Bianchi, Y. Freund, D. Haussler, D. P. Helmbold, R. E. Schapire, and M. K. Warmuth. How to use expert advice. Journal of the ACM (JACM), 44(3):427-485, 1997.
[10] N. Cesa-Bianchi, Y. Freund, D. P. Helmbold, D. Haussler, R. E. Schapire, and M. K. Warmuth. How to use expert advice. In Proceedings of the 25th Annual ACM Symposium on Theory of Computing, pages 382-391, 1993.
[11] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
[34] N. M. Vural, H. Gokcesu, K. Gokcesu, and S. S. Kozat. Minimax optimal algorithms for adversarial bandit problem with multiple plays. IEEE Transactions on Signal Processing, 67(16):4383-4398, 2019.
[35] J. Zimmert and T. Lattimore. Connections between mirror descent, Thompson sampling and the information ratio. Advances in Neural Information Processing Systems, 32, 2019.
[36] J. Zimmert and Y. Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. The Journal of Machine Learning Research, 22(1):1310-1358, 2021.