Understanding Bandits with Graph Feedback

Abstract: Houshuang Chen, zengfeng Huang, Shuai Li, Chihao Zhang

Reviews and author responses:

 Dear reviewer, we apologize for skipping over your main concern and we will respond below. 

**the contribution is quite limited since it only focuses on logarithmic gains**

Although our improvement from $\delta$ to $\delta^*$ seems to be incremental and only brings logarithmic gains, we believe that the observation here is important and non-trivial for the following two reasons.

First, as mentioned in the introduction, the ultimate goal of this line of research is to identify the correct graph parameter to characterize the mini-max regret. We believe and give evidence to show that $\delta^*$ is a reasonable candidate as matching lower bound has been proved for some instances. Therefore, the step from $\delta$ to $\delta^*$, or the logarithmic gain in some cases, is crucial and might be already tight.

Secondly, as $\delta^*$ is the optimal solution of a certain LP, this allows us to prove lower bounds in a more natural way: applying the strong duality theorem and using a new rounding method to construct hard instances from fractional solutions of the dual LP. This observation explains why a lower bound in terms of $\delta^*$ should exist and it is also the main technical contribution of this work. Some improved and optimal lower bounds are obtained in this way and we think this method towards lower bound is new to the community.

**It is not tight for many graphs and even worse than previous work for some graphs...In particular, for general graphs, it seems that the gap between the upper and lower bound may be of order $\delta^\*/n$**

Our gap between lower and upper bound is $(\alpha\log n)^{1/3}$. For all known instances, it is no worse than the previous one. Of course, as we explained in the last response, we are not able to prove this for all instances at the current time.

On the other hand, unlike upper bounds, the usefulness of lower bounds is theoretical and is mainly used to justify how good the algorithms can reach. Therefore, it is not that important for us to obtain all best lower bounds in a *uniform* way. For a specific instance, we can safely believe in the best known lower bounds (either proved in this paper or before).

 I thank the authors for their response to my review. While I am currently travelling (for the next 1.5 week) and in a very tight time schedule, I only give a high level response. I also read other reviews.

In general, I am not fully satisfied with the answer from the authors, which did not addressed my main concern: the contribution is quite limited since it only focuses on logarithmic gains, it is not tight for many graphs and even worse than previous work for some graphs.
I thus decide to keep my score of 6.

 Dear reviewer, thanks for your comments and we will proofread the paper for a better presentation and correct these minor errors in the revised version.

 Dear reviewer, thanks for reading and the comments. We will reply to your concerns below.

First of all, we apologize for not including too much elaboration on the model, and the missing of figures to explain terminologies. This is mainly due to the page limitation, and we will try to improve the presentation in the full version.

Although this is a theoretically oriented paper and mainly aims at narrowing the gap between upper and lower bounds, we will add some numerical experiments with our algorithm in the revised version. Thanks for your suggestion.

We will also include some discussions about the practical motivation and justify our structural assumption in the revised version. In fact, bandits with graph feedback has some notable applications including web advertising, video recommendation, etc. 

In web advertising (see [24, Mannor and Shamir, NeurIPS 2011] for details), the vertex set of the feedback graph is all ads and the edges characterize the correlation between ads. Suppose ad $i$ stands for running shoes, ad $j$ stands for a wheelchair. Then a user who clicks ad $i$ is unlikely interested in $j$ and vice versa. So in the feedback graph, we have an undirected edge between $i$ and $j$. 

Another potential application is video recommendation in some popular short-video apps (for example, TikTok). Every time the app starts, it plays a video that hopes to capture your attention. One way to improve the quality of the start-up recommendation is to show some similar videos alongside and ask the user to pick some to watch next. The recommendation algorithm can then learn whether those similar videos are liked by the user. Therefore, we let the vertex set of the feedback graph be all videos and add an edge between two videos if they are similar. This is a weakly observable graph.

In both applications described above, the number of edges incident to a specific vertex (either correlated ads or similar videos) is small. Therefore our assumption in Corollary 4, the graph is of bounded in-degree, is reasonable.

Minor concerns:

**In Lemma 7, do the authors mean "Any graph  contains ..." ?**  
Yes, and no assumptions are required.

**There are some "??" references.**  
We will correct this in the revised version. Thanks.


 Dear reviewer, thanks for the comment. We will reply to your concerns below.

**Cost of pre-processing**

Yes, the linear program to determine the exploration probability in our algorithm can be efficiently computed both in theory and in practice using standard LP solver while it is NP-hard to determine the exploration probability used in previous work [Alon et al. 2015]. And this pre-processing step is also necessary in the algorithm of [Alon et al. 2015].



**Does it("we can not find an instance") mean that you could not find one but there might be some? Did you try simulations with random graphs?**  

Well, we conjecture no such graph exists but no proof yet. Simulation with random graphs is a good idea but a few technical difficulties exist. The main difficulty is that the quantities $\delta$ and $\alpha$ are in general NP-hard to compute. Therefore, it is more realistic to attack the problem from a theoretical point of view, either finding a proof or constructing a counter-example. This is an interesting question for future work. 

**Experiments \& numerical results**
We will add some experimental results in the revised version. Thanks for suggestion.


**Lem. 7: "The graph". Do you mean "For any graph"?**  
Yes.

**Thm. 8: it only outperforms in the regimes $\log^2 n>\delta>\log n$.**  
Thanks for pointing this out and after careful check we find that it outperforms in the regimes $o(\log^2 n\cdot \log\log n)>\delta>\omega(\log n)$

 Dear reviewer, thanks for the comments. We would like to justify that some special cases that we studied in the paper are not that special, especially in practice.

For example, we narrow the gap for graphs of bounded in-degree graphs in Corollary 4. In fact, this is the case for some notable applications of bandits with graph feedback including web advertising, video recommendation, etc. 

In web advertising (see [24, Mannor and Shamir, NeurIPS 2011] for details), the vertex set of the feedback graph is all ads and the edges characterize the correlation between ads. Suppose ad $i$ stands for running shoes, ad $j$ stands for a wheelchair. Then a user who clicks ad $i$ is unlikely interested in $j$ and vice versa. So in the feedback graph, we have an undirected edge between $i$ and $j$. 

Another potential application is video recommendation in some popular short-video apps (for example, TikTok). Every time the app starts, it plays a video that hopes to capture your attention. One way to improve the quality of the start-up recommendation is to show some similar videos alongside and ask the user to pick some to watch next. The recommendation algorithm can then learn whether those similar videos are liked by the user. Therefore, we let the vertex set of the feedback graph be all videos and add an edge between two videos if they are similar. This is a weakly observable graph.

In both applications described above, the number of edges incident to a specific vertex (either correlated ads or similar videos) is small. Therefore our assumption in Corollary 4, the graph is of bounded in-degree, is reasonable.

This paper tries to improve our understanding of the hardness parameters in adversarial bandits with weakly observable graphs.
They propose a slightly modified algorithm with improved regret guarantee in certain instances and novel lower bounds.  This paper tries to narrow the gap between upper and lower bounds for adversarial bandits with graph feedback in the `weakly observable' $T^{2/3}$ regime.
They do that on two frontiers, first of all, they show that the weak domination number used in previous upper bounds is not the exact quantity that appears in the analysis of OSMD with Shannon entropy. Instead, the fractional weak domination number appears, which in special cases can be up to a factor $\ln(n)$ smaller. The algorithm used for the upper bound is a standard mirror descent with shannon entropy.
Aside from special constructions, the ratio between fractional and vanilla weak domination number is often O(1) and hence the upper bound does improve on special instances only.
They present three types of lower bounds. First is a general bound based on the k-independent packing number. This result subsumes the SOTA lower bound as a special case.
Second is a reduction to the special case of 1-independent packing. One can relate the upper and lower bound in this case based on the integrality gap, which is incomparable to the previously known lower bound and it is unclear if it is always tighter.
Finally they also show that a lower bound based on the logarithm of the Independence number (over the subgraph without self-loops) is possible.
This allows them to close the gap between upper and lower bound completely for instances where the fractional weak domination number is O(1) and the independence number is Omega(|V|).

The paper is written clearly and the proofs seem to be correct.

The main question is the significance of the results. Previous algorithms are provably optimal up to a $\ln(n)$ gap. Even with the new theorems, the worst-case gap is at least $\ln(n)/\ln\ln(n)$ in general graphs. 
On the positive side, this problem is very difficult, even in the T^1/2 regime. They have at least closed the gap between upper and lower bounds on some special graphs for the first time.
 yes

This paper considers the multi-armed bandit problem with graph feedbacks.  The authors focus on the class of weakly observable graphs for which the minimax regret is known to be of order $\Theta(T^{2/3})$. In previous work, the existing upper and lower bounds were expressed in terms of the weak domination number of the graph and showed a gap. To reduce it, the authors propose two new notions to measure the complexity of a graph: the k-packing independence and the fractional weak domination number. 

They design an algorithm based on Online Stochastic Mirror Descent (OSMD) to which they add an exploration that depends on the graph. Their algorithm satisfies an upper bound on regret in terms of the graph's fractional weak domination number, instead of the (larger) weak domination number that was used for the existing upper bounds. 

The authors also prove a lower bound in terms of the k-packing independence number, from which they derive various lower bounds. For some graphs (such as complete bipartite graphs), their upper and lower bounds are of the same order (no gap) and for many others (trees, bounded degree,...) the gap between the two is smaller than in previous work.  This article is relatively well written and the bandit problem with graph feedback is surely relevant to the community with many applications. The main contributions of the paper are in my opinion the following:
- the paper introduces two new notions for understanding the complexity of graph feedback that seem to provide tighter results than existing ones. 
- The paper designs a new variant of OSMD for bandits with graph feedback with a tight upper bound on its regret. For some graphs, it exactly matches their lower bound and it is thus optimal.
- The authors were able to relate their two notions and express their lower and upper bounds in terms of fractional low dominance numbers. The approach used for the lower bounds seems to use new technical tools that could be useful to the community.  

On the other hand, the improvements over previous work mainly consist of gaining logarithmic terms in the rates. At this point, we would like to get the tightest possible results. It is therefore unfortunate that the bounds still suffer from a logarithmic gap in many cases and even worse for some cases. In particular, for general graphs, it seems that the gap between the upper and lower bound may be of order $\delta^*/n$, which makes the gap even larger than in previous work.   


Comments:  
1. What is the cost of solving the linear program in the pre-processing part of the algorithm? Is it simpler to solve the fractional program than to solve the integral one? Do the other algorithms for graph feedbacks also need this pre-processing step?

2. Some simulations to illustrate for some graphs the difference between the bounds could be interesting. 

Minor comments:
- Table 1 should be re-worked. It is too large, with a lot of information which may yield some confusion. 
- It is stated in line 97 that "we can not find an instance", does it mean that you could not find one but their might be some? Did you try simulations with random graphs?
- Lem. 7: "The graph". Which one? Do you mean "For any graph"?
- Thm. 8: it only outperforms in the regimes $\log^2 n > \delta > \log n$ Yes

The authors study the problem of adversarial $K$ armed bandits with graph feedback, where selecting an arm $i$ provides an observation of the reward of all arms $j$ that are neighbors of $i$ in a given graph $G = (V,E)$, with $V$ the set of arms. The authors prove new regret lower bounds and provide new regret upper bounds for online mirror descent applied to this problem. The main idea is to consider the "fractional weak domination number" which, to the best of my understanding, quantifies the number of rounds necessary in order to obtain observations from all of the arms. While the results seem interesting, the paper is quite technical and difficult to read.   1) The paper is quite dry and hard to read, especially for a new reader:
- the results are presented before stating the model. This makes section 1 almost impossible to understand without reading Sections 2 and 3.
- The results are stated as a succession of theorems and lemmas, with little intuition nor graphical illustrations about the notions introduced (dominating sets, vertex packing sets, 1-degenerate graphs etc.)


2) No numerical experiments are presented, although simulating the proposed algorithm on some topologies and computing the corresponding regret lower bounds does not seem very hard to do. 

3) There is no discussion about the practical applications, in particular to justify the structural assumptions on the graphs at hand. The authors demonstrate that, in some graph structures, their results outperform the literature. However since one does not know what kind of structures are practically relevant, this seems relatively arbitrary.

4) Minor remarks:
- In Lemma 7, what does "the Graph G contains..." mean? Do the authors mean "Any graph $G$ contains ..." ? Are there any assumptions on $G$ required ?
- "Theorem 8 outperforms ..." is a part of the sentence missing ?  
- There are some "??" references. N/A

This paper studies adversarial multi-armed bandits with graph feedback and provides new upper and lower bounds for the regret, which sharpen previous bounds in some regimes and provide tight bounds for some graphs.

One new idea is to define a linear program for fractional weak domination in a graph and its dual, fractional vertex packing. Lower and upper bounds are given in terms of the optimal value of this LP and its integrality gap (Theorems 1 and 3). 

Another lower bound is proved by using the notion of the maximum k-packing independent set of a graph (Theorem 2).

A summary of new bounds is given in Table 1. The min-max regret rate is determined for complete bipartite graphs up to the constant and for trees and bounded in-degree graphs up to a $(\log n)^{1/3}$ factor.  I like this paper. The ideas are new and interesting from a mathematical perspective, and the results are strong. This is a natural, important problem in online learning and even slightest improvements are of significant interest.

The paper is generally well-written. It needs just a round of proofreading to improve its flow. Some minor comments below.

I haven't verified the proofs, which appear in the supplementary material.

Comments:
- Line 63: section number is missing.
- Lemma 7: "The graph" should be "Any graph."
- Corollary 10: it's weird that the bound hasn't been simplified.
- Line 272: "If fact" should be "In fact" The authors have adequately addressed the limitations.

Meta-review: While the reviewers felt that the results are limited in their relevance and scope (they only apply to weakly observable graphs and the improvements are within log factors), they also thought---and I concur---that the paper is tackling a very basic and challenging problem, and from a mathematical point of view it contains some interesting ideas (the connection to integrality gaps of LPs is particularly compelling).  I strongly encourage the authors to take the reviewers’ comments into consideration when preparing the final version and carefully implement their suggestions.

Decision: accept