Title: value: On the Minimax Regret for Online Learning with Feedback Graphs
Authors: value: Khaled Eldowa
Emmanuel Esposito
Tommaso Cesari
Nicolò Cesa-Bianchi
Venue: NeurIPS 2023
Keywords: value: Online learning
Feedback graphs
Multiarmed bandits

Abstract: value: In this work, we improve on the upper and lower bounds for the regret of online learning with strongly observable undirected feedback graphs. The best known upper bound for this problem is $\mathcal{O}\bigl(\sqrt{\alpha T\ln K}\bigr)$, where $K$ is the number of actions, $\alpha$ is the independence number of the graph, and $T$ is the time horizon. The $\sqrt{\ln K}$ factor is known to be necessary when $\alpha = 1$ (the experts case). On the other hand, when $\alpha = K$ (the bandits case), the minimax rate is known to be $\Theta\bigl(\sqrt{KT}\bigr)$, and a lower bound $\Omega\bigl(\sqrt{\alpha T}\bigr)$ is known to hold for any $\alpha$. Our improved upper bound $\mathcal{O}\bigl(\sqrt{\alpha T(1+\ln(K/\alpha))}\bigr)$ holds for any $\alpha$ and matches the lower bounds for bandits and experts, while interpolating intermediate cases. To prove this result, we use FTRL with $q$-Tsallis entropy for a carefully chosen value of $q \in [1/2, 1)$ that varies with $\alpha$. The analysis of this algorithm requires a new bound on the variance term in the regret. We also show how to extend our techniques to time-varying graphs, without requiring prior knowledge of their independence numbers. Our upper bound is complemented by an improved $\Omega\bigl(\sqrt{\alpha T(\ln K)/(\ln\alpha)}\bigr)$ lower bound for all $\alpha > 1$, whose analysis relies on a novel reduction to multitask learning. This shows that a logarithmic factor is necessary as soon as $\alpha < K$.


Review discussion:
- The paper is well-written and the proposed algorithm is easy to follow.
- The designed algorithm improves upon the best known $T$-dependent regret bound for undirected strongly observable graphs.
Specifically, the important technical contribution is Lemma 1, which shows that the stability term of the FTRL can be bounded by $\alpha^{1+b}$ where $b$ is decided by the parameter choice of Tsallis-entropy, compared to the one with log factors obtained in [Alon et al., 2015] and [Zimmert and Lattimore, 2019].
- The lower bound also tries to bridge the previous gap between the full-information case and the feedback graph case, although there is still a gap between the upper and the lower bound and the technical used in the lower bound is very similar to the one used for proving the lower bound for contextual bandit in [Seldin and Lugosi, 2016].
I do not find major weaknesses of this paper, except that the current upper bound is only obtained for the undirected strongly observable graph, which is also discussed by the authors in the appendix. Generalizing this technique to the general strongly observable graphs would be interesting.
- The current algorithm is an FTRL-based algorithm. I wonder whether a certain type of online mirror descent based algorithm with a time-varying learning rate can also achieve similar results?
See Weakness and Questions.
No ethics review needed.
We thank the reviewer for the feedback. As mentioned in our answer to Reviewer TWzC, we hope the case of directed feedback graphs could be addressed by extending the proposed techniques.
Regarding your question, OMD could be used in place of FTRL with the same techniques presented in this work. With regards to the adoption of a time-varying learning rate, we remark that adapting to a sequence of graphs with arbitrary independence numbers would also require the Tsallis entropy parameter to adapt to such a sequence, which poses significant challenges for the analysis. This is why we adopted an approach based on the doubling trick.
Thanks for the authors' response. The response addresses my questions and I keep the original score.
I think the paper is very interesting both from the perspective of results as well as from the perspective of techniques. I find very interesting the fact that the provided regret bound matches the respective regret bounds for the bandit ($\alpha = K$) and the full-information case ($\alpha=1$).
Also the authors provide a nearly matching lower bound that is based on a novel reduction to multitask learning. Both the techniques for the upper and lower bound require no trivial ideas that are nicely illustrated in the current write up. I also liked the fact that the provided algorithm can be modified to handle the case of time-varying independence numbers.
I overall believe that the paper provides very solid results and that will be of great interest of the online learning audience of NeurIPS.
The only limitation of the paper that I can find is the assumption on undirected graphs. The latter is also signified by the authors and considering the complexity of the current I think it is more than fair to consider the directed case for future work.
N/A
No ethics review needed.
We thank the reviewer for the feedback. As mentioned in our answer to Reviewer J2zm, we agree that the directed feedback graphs case is an interesting future direction, which we hope could be addressed by extending the proposed techniques.
Thank you for your response. After reading the other reviews, I am confident that this is a very good paper and I have decided to keep my current score.
The paper is very well-written and pleasant to read. The question is well posed, and motivated by the fact that improving the $\sqrt{\log K}$ factor for learning with experts, from $O(\sqrt{KT\log K})$ to $O(\sqrt{KT})$ remained open for a significant amount of time in the literature.
Hence, interpolating the regret bounds in terms of the remaining factor $\sqrt{\log K}$ between the known bounds $O(\sqrt{\alpha T\log k})$ and $\Omega(\sqrt{\alpha T})$ seems an important question.
The upper bounds use the same algorithm (FTRL with $q$-Tsallis entropy) with adapted parameters $q$, which also interpolates between known approaches for the extreme cases $\alpha=1$ or $\alpha=K$, giving new insights on the impact of this parameter on the learning behavior.
Although the paper presents new contributions to interpolate between experts and bandits, the proofs follow classical arguments in the literature and seem to bring limited original ideas.
In particular, it was known that the $q$-Tsallis entropy specifically allows achieving the minimax entropy in both end cases. The main difficulty in the generalization to feedback graphs is in a novel bound for the variance term in the FTRL analysis (Lemma 3), which itself is a generalization of known results in the literature for $q=1$. The rest of the proof follows standard analysis.
The lower bound heavily relies on the fact that graphs are allowed to vary over time. It seems that the approach cannot be extended to the important case of interest of a fixed graph (which seemed to be the main case studied in the previous literature on feedback graph bandits). The paper would also be significantly strengthened if the lower bound could be improved to match the upper bound when e.g. the independence number $\alpha$ grows as a polynomial in $K$---in that case, the present paper does not provide improvements in terms of rates compared to the existing literature.
As such, the gap left by this paper seems to remain quite important.
- Does the analysis carry over for adaptive adversaries, or is the oblivious assumption important?
- l.97 the independence number is the cardinality of such a set, right?
- What happens if we assume that the graph is constant over time? Would you expect similar lower bounds to hold?
The limitations were very well addressed by the authors, who clearly identify the new bound on the variance (Lemma 3) as the key added technical contribution for the upper bounds.
No ethics review needed.
We thank the reviewer for their feedback. We remark that obtaining the key result of Lemma 1 was an obstacle for prior work in the way of obtaining improved guarantees using $q$-Tsallis entropy with feedback graphs.
Although we focused on the case of time-varying graphs in our lower bound construction, we conjecture that the upper bound is tight for all values of $K$ and $\\alpha$, even in the case of some fixed graph.
In any case, please note that the lower bound we provided is the first such result that hints at the necessity of a logarithmic factor for the minimax regret of this problem (beyond the experts case). To address the question about adaptive adversaries, the analysis does carry over straightforwardly if the adversary adapts to the previous choices of the learner with our current notion of regret, but it would be interesting to investigate the more challenging notions of dynamic or adaptive regret.
Finally, we thank the reviewer for pointing out the typo.
Thank you for your answers and comments. After re-examing the paper, I agree that the contributions are significant and the new bounds in the lemmas are significant. I have updated my score as a result.
As stated in the summary, the main technical novelty is the analysis of the variance of the IPW estimator in the setting of feedback graphs and FTRL. This is itself a rather interesting contribution and I expect this to be useful in various other contexts. The main result presented in the paper resolves an interesting and natural question in the literature.
One main drawback of the approach of $\alpha$ dependent choice of the $q$ is that it makes the algorithm extremely inefficient (in theory at least).
In particular, the independence number of a graph is known to (extremely) hard to approximate;
in particular, unless $P=NP$ there is no "non-trivial" approximation algorithm even. This is to be contrasted with the previous work in the feedback graph setting which are efficient ( polynomial in $N$ the size of the graph/number of bandits). It remains an excellent open question to see if there is an efficient algorithm achieving this bound.
- Is it immediately clear how the regret behaves when one only has an estimate of the independence number instead of the exact quantity?
Yes
No ethics review needed.
We thank the reviewer for their feedback. It is indeed true that computing, or even approximating, $\\alpha$ is computationally intractable in general. Please note that we do address this issue in Section 4, where we not only generalize our approach to time-varying feedback graphs with possibly different independence numbers, but also lift the requirement of knowing these independence numbers. If we specialize this setting to that of a fixed graph, we can avoid requiring knowledge of $\\alpha$ altogether. We thank the reviewer for pointing out this possible misunderstanding. We presented the first result assuming the knowledge of $\\alpha$ to simplify the presentation, but we will make sure to clarify this important feature of our theory in the revised version.
We thank the authors for pointing this out and apologize for the oversight. Increasing the score accordingly.

Meta-review:
role: PC
data: comment: value: There is unanimous support for this paper. An important innovation is a technically novel analysis of a key step from the now-standard Tsallis entropy-based regret analysis (specifically, bounding a certain variance-like quantity in such a way as to get dependence on the independence number rather than the number of actions). There initially were concerns about the level of technical innovation, but the from the discussion period with the authors, the concerned reviewers agreed about the technical novelty. While ultimately, this work "only" amounts to refining bounds at the level of log factors, such refinements often are not easy, which makes this contribution impressive. This work will be a welcome contribution to this year's conference.

Decision: Accept (spotlight)