Title: Stochastic contextual bandits with graph feedback: from independence number to MAS number
Venue: NeurIPS 2024
Accepted: yes
Keywords: contextual bandits

graph feedback

minimax rate
Abstract: We consider contextual bandits with graph feedback, a class of interactive learning problems with richer structures than vanilla contextual bandits, where taking an action reveals the rewards for all neighboring actions in the feedback graph under all contexts. Unlike the multi-armed bandits setting where a growing literature has painted a near-complete understanding of graph feedback, much remains unexplored in the contextual bandits counterpart. In this paper, we make inroads into this inquiry by establishing a regret lower bound $\Omega(\sqrt{\beta_M(G) T})$, where $M$ is the number of contexts, $G$ is the feedback graph, and $\beta_M(G)$ is our proposed graph-theoretic quantity that characterizes the fundamental learning limit for this class of problems. Interestingly, $\beta_M(G)$ interpolates between $\alpha(G)$ (the independence number of the graph) and $\mathsf{m}(G)$ (the maximum acyclic subgraph (MAS) number of the graph) as the number of contexts $M$ varies. We also provide algorithms that achieve near-optimal regret for important classes of context sequences and/or feedback graphs, such as transitively closed graphs that find applications in auctions and inventory control. In particular, with many contexts, our results show that the MAS number essentially characterizes the statistical complexity for contextual bandits, as opposed to the independence number in multi-armed bandits.

Stochastic contextual bandits with graph feedback: from independence number to MAS number

Abstract

We consider contextual bandits with graph feedback, a class of interactive learning problems with richer structures than vanilla contextual bandits, where taking an action reveals the rewards for all neighboring actions in the feedback graph under all contexts. Unlike the multi-armed bandits setting where a growing literature has painted a near-complete understanding of graph feedback, much remains unexplored in the contextual bandits counterpart. In this paper, we make inroads into this inquiry by establishing a regret lower bound Ω( √ β M ( G ) T ) , where M is the number of contexts, G is the feedback graph, and β M ( G ) is our proposed graphtheoretic quantity that characterizes the fundamental learning limit for this class of problems. Interestingly, β M ( G ) interpolates between α ( G ) (the independence number of the graph) and m ( G ) (the maximum acyclic subgraph (MAS) number of the graph) as the number of contexts M varies. We also provide algorithms that achieve near-optimal regret for important classes of context sequences and/or feedback graphs, such as transitively closed graphs that find applications in auctions and inventory control. In particular, with many contexts, our results show that the MAS number essentially characterizes the statistical complexity for contextual bandits, as opposed to the independence number in multi-armed bandits.

1 Introduction

Contextual bandits encode a rich class of sequential decision making problems in reality, including clinical trials, personalized healthcare, dynamic pricing, recommendation systems (Bouneffouf et al., 2020). However, due to the exploration-exploitation trade-off and a potentially large context space, the pace of learning for contextual bandits could be slow, and the statistical complexity of learning could be costly for application scenarios with bandit feedback (Agarwal et al., 2012). There are two common approaches to alleviate the burden of sample complexity, either by exploiting the function class structure for the reward (Zhu and Mineiro, 2022), or by utilizing additional feedback available during exploration.

In this paper we focus on the second approach, and aim to exploit the feedback structure efficiently in contextual bandits. The framework of formulating the feedback structure as feedback graphs in bandits has a long history (Mannor and Shamir, 2011; Alon et al., 2015, 2017; Lykouris et al., 2020), where a direct edge between two actions indicates choosing one action provides the reward information for the other. Such settings have also been generalized to contextual cases (Balseiro et al., 2019; Dann et al., 2020; Han et al., 2024), where counterfactual rewards could be available under different contexts. Typical results in these settings are that, the statistical complexity of bandits with feedback graphs is characterized by some graph-theoretic quantities, such as the independence number or the maximum acyclic subgraph (MAS) number of the feedback graph.

general variables determining the rewards. A widely studied alternative is the structured setting that leverages certain structures in the dependence on the context. Examples of the latter include linear contextual bandits (Auer, 2002; Agrawal and Goyal, 2013), which assume a linear reward function on the contexts, and their variants (Chu et al., 2011; Li et al., 2017; Agrawal and Devanur, 2016).

Despite the existing results, especially in multi-armed bandits where a near-complete characterization of the optimal regret is available (Alon et al., 2015; Kocák and Carpentier, 2023; Eldowa et al., 2024), the statistical complexity of contextual bandits with feedback graphs is much less understood. For example, consider the case where there is a feedback graph G across the actions and a complete feedback graph across the contexts (termed as complete cross-learning in (Balseiro et al., 2019)). In this case, for a long time horizon T , the optimal regret scales as ˜ Θ( √ α ( G ) T ) when there is only one context (Alon et al., 2015), but only an upper bound ˜ O ( √ m ( G ) T ) is known regardless of the number of contexts (Dann et al., 2020). Here α ( G ) and m ( G ) denote the independence number and the MAS number of the graph G , respectively; we refer to Section 1.1 for the precise definitions. While α ( G ) = m ( G ) for all undirected graphs, for directed graphs their gap could be significant. It is open if the change from α ( G ) to m ( G ) is essential with the increasing number of contexts, not to mention the precise dependence on the number of contexts.

1.1 Notations

For n ∈ N , let [ n ] := { 1 , · · · , n } . For two probability measures P and Q over the same space, let TV ( P, Q ) = ∫ | d P -d Q | / 2 be the total variation (TV) distance, and KL ( P ∥ Q ) = ∫ d P log(d P/ d Q ) be the Kullback-Leibler (KL) divergence. We use the standard asymptotic notations O, Ω , Θ , as well as ˜ O, ˜ Ω , ˜ Θ to denote respective meanings within polylogarithmic factors.

We use the following graph-theoretic notations. For a directed graph G = ( V, E ) , let u → v denote that ( u, v ) ∈ E . For u ∈ V , let N out ( u ) = { v ∈ V : u → v } be the set of out-neighbors of u (including u itself). We will also use N out ( A ) = ∪ v ∈ A N out ( v ) to denote the set of all out-neighbors of vertices in A . The independence number α ( G ) , dominating number δ ( G ) , and maximum acyclic subgraph (MAS) number m ( G ) are defined as

̸

α ( G ) = max {| I | : I ⊆ V is an independent set, i.e. u ̸→ v, ∀ u = v ∈ I } ,

δ ( G ) = min {| J | : J ⊆ V is a dominating set, i.e. N out ( J ) = V } , m ( G ) = max {| D | : D ⊆ V induces an acyclic subgraph of G } ,

respectively. It is easy to show that max { α ( G ) , δ ( G ) } ≤ m ( G ) , with a possibly unbounded gap, and a probabilistic argument also shows that δ ( G ) = O ( α ( G ) log | V | ) (cf. Lemma A.1).

1.2 Our results

In this paper we focus on contextual bandits with both feedback graphs across actions and complete cross-learning across contexts. This setting was proposed in (Han et al., 2024), with applications to bidding in first-price auctions. As opposed to an arbitrary feedback graph across all context-action pairs in (Dann et al., 2020), we assume a complete cross-learning because of two reasons. First, in many scenarios the contexts encode different states which only play roles in the reward function; in other words, the counterfactual rewards for all contexts can be observed by plugging different contexts into the reward function. Such examples include bidding in auctions (Balseiro et al., 2019; Han et al., 2024) and sleeping bandits modeled in (Schneider and Zimmert, 2024). Second, this scenario is representative and sufficient to reflect the main ideas and findings of this paper. Discussion and results under more general settings is left to Section 4.1.

Throughout this paper we consider the following stochastic contextual bandits. At the beginning of each round t ∈ [ T ] during the time horizon T , an oblivious adversary chooses a context c t ∈ [ M ] and reveals it to the learner, and the learner chooses an action a t ∈ [ K ] . There is a strongly observable 1 directed feedback graph G = ([ K ] , E ) across the actions such that all rewards in ( r t,c,a ) c ∈ [ M ] , ( a t ,a ) ∈ E are observable, where we assume no structure in the rewards except that r t,c,a ∈ [0 , 1] . In our stochastic environment, the mean reward E [ r t,c,a ] = µ c,a is unknown but invariant with time. We are interested in the characterization of the minimax regret achieved by the

̸

learner:

where the infimum is over all admissible policies based on the available observations. In the sequel we might also constrain the class of context sequences to c T ∈ C , and we will use R ⋆ T ( G,M, C ) and R T ( π T ; G,M, C ) to denote the respective meanings by taking the supremum over c T ∈ C .

Our first result concerns a new lower bound on the minimax regret.

Theorem 1.1 (Minimax lower bound) . For T ≥ β M ( G ) 3 , it holds that R ⋆ T ( G,M ) = Ω( √ β M ( G ) T ) , where the graph-theoretic quantity β M ( G ) is given by

and I i ̸→ I j means that u ̸→ v whenever u ∈ I i and v ∈ I j .

Theorem 1.1 provides a minimax lower bound on the optimal regret, depending on both the number of contexts M and the feedback graph G . Note that the independent subsets I 1 , . . . , I M are allowed to be empty if needed. It is clear that β 1 ( G ) = α ( G ) is the independence number, and β M ( G ) = m ( G ) whenever M ≥ m ( G ) . This leads to the following corollary.

Corollary 1.2 (Tightness of MAS number) . For any graph G , if M ≥ m ( G ) and T ≥ m ( G ) 3 , one has R ⋆ T ( G,M ) = Ω( √ m ( G ) T ) .

Corollary 1.2 shows that, the regret change from ˜ Θ( √ α ( G ) T ) in multi-armed bandits to ˜ O ( √ m ( G ) T ) in contextual bandits (Dann et al., 2020) is in fact not superfluous when there are many contexts. In other words, although the independence number determines the statistical complexity of multi-armed bandits with graph feedback, the statistical complexity in contextual bandits with many contexts is completely characterized by the MAS number .

For intermediate values of M ∈ (1 , m ( G )) , the next result shows that the quantity β M ( G ) is tight for a special class C SA of context sequence called self-avoiding contexts . Acontext sequence ( c 1 , · · · , c T ) is called self-avoiding iff c s = c t for s < t implies c s = c s +1 = · · · = c t (or in other words, contexts do not jump back). For example, 113222 is self-avoiding, but 12231 is not. This assumption is reasonable when contexts model a nonstationary environment changing slowly, e.g. the environment changes from season to season.

Theorem 1.3 (Upper bound for self-avoiding contexts) . For self-avoiding contexts, there is a policy π achieving R T ( π ; G,M, C SA ) = ˜ O ( √ β M ( G ) T ) . This policy can be implemented in polynomial-time, and does not need to know the context sequence in advance.

As the minimax lower bound in Theorem 1.1 is actually shown under C SA , for large T , Theorem 1.3 establishes a tight regret bound for stochastic contextual bandits with graph feedback and selfavoiding contexts. The policy used in Theorem 1.3 is based on arm elimination, where a central step of exploration is to solve a sequential game in general graphs which has minimax value ˜ Θ( β M ( G )) and could be of independent interest.

For general context sequences, we have a different sequential game in which we do not have a tight characterization of the minimax value in general. Instead, we have the following upper bound, which exhibits a gap compared with Theorem 1.1.

Theorem 1.4 (Upper bound for general contexts) . For general contexts, there is a policy π achieving

where

Fortunately, additional assumptions on the feedback graph G can be leveraged to recover the tight regret bound:

Corollary 1.5 (Upper bound for transtively closed or undirected feedback) . For any undirected or transitively closed graph G , the policy π in Theorem 1.4 achieves a near-optimal regret R T ( π ; G,M ) = ˜ O ( √ β M ( G ) T ) .

A directed graph G is called transitively closed if u → v and v → w imply that u → w . In reality directed feedback graphs are often transitively closed, for a directed structure of the feedback typically indicates a partial order over the actions. Examples include bidding in auctions (Zhao and Chen, 2019; Han et al., 2024) and inventory control (Huh and Rusmevichientong, 2009), both of which exhibit the one-sided feedback structure i → j for i ≤ j . For general graphs, Theorem 1.4 gives another graph-theoretic quantity β M ( G ) . Note that β M ( G ) ≥ β M ( G ) as there is no acyclic requirement between I c 's in (3), which in turn is due to a technical difficulty of non-self-avoiding contexts. Further discussions on this gap are deferred to Section 4.3.

Interestingly, the upper bound quantities β M ( G ) and β M ( G ) are not explicitly linear in M and are always no larger than α ( G ) M . Hence our results partially answer an open problem in (Hao et al., 2022, Remark 5.11) that if the dependence of regret bound O ( √ α ( G ) MT ) on M can be improved.

1.3 Related work

The study of bandits with feedback graphs has a long history dating back to (Mannor and Shamir, 2011). For (both adversarial and stochastic) multi-armed bandits, a celebrated result in (Alon et al., 2015, 2017) shows that the optimal regret scales as ˜ Θ( √ α ( G ) T ) if T ≥ α ( G ) 3 ; the case of smaller T was settled in (Kocák and Carpentier, 2023), where the optimal regret is a mixture of √ T and T 2 / 3 rates. For stochastic bandits, simpler algorithms based on arm elimination or upper confidence bound (UCB) are also proposed (Lykouris et al., 2020; Han et al., 2024), while the UCB algorithm is only known to achieve an upper bound of ˜ O ( √ m ( G ) T ) . 2 In addition to strongly observable graphs we primarily focus on, weakly observable graphs have also drawn vast interest (Alon et al., 2015; Chen et al., 2021) where the optimal regret is characterized by the dominating number δ ( G ) . There exploration plays a more significant role due to weaker observability of certain nodes, leading to an optimal regret ˜ Θ( δ ( G ) 1 / 3 T 2 / 3 ) . We will briefly discuss the regret characterization of our contextual setting with weakly observable graphs in Section 4.1 and 4.2.

Recently, the graph feedback was also extended to contextual bandits under the name of 'crosslearning' (Balseiro et al., 2019; Schneider and Zimmert, 2024). The work (Balseiro et al., 2019) considered both complete and partial cross-learning, and showed that the optimal regret for stochastic bandits with complete cross learning is ˜ Θ( √ KT ) . Motivated by bidding in first-price auctions, (Han et al., 2024) generalized the setting to general graph feedback across actions and complete cross-learning across contexts, a setting used in the current paper. The finding in (Han et al., 2024) is that the effects of graph feedback and cross-learning could be 'decoupled': a regret upper bound ˜ O ( √ min { α ( G ) M,K } T ) is shown, which is tight only for a special choice of the feedback graph G . The work (Dann et al., 2020) considered a tabular reinforcement learning setting with adversarial initial states, so that their setting with episode length H = 1 coincides with our problem with a general feedback graph G across all context-action pairs. They showed that the UCB algorithm achieves a regret upper bound ˜ O ( √ m ( G ) T ) ; however, their lower bound was only Ω( √ α ( G ) T ) when T ≥ α ( G ) 3 . Therefore, tight lower bounds that work for general graphs G are still underexplored in the literature, and our regret upper bounds in Theorems 1.3 and 1.4 also improve or generalize the existing results.

The problem of bandits with feedback is also closely related to partial monitoring games (Bartók et al., 2014). Although this is a more general setting which subsumes bandits with graph feedback, the results in the literature (Bartók et al., 2014; Lattimore, 2022; Foster et al., 2023a) typically have tight dependence on T , but often not on other parameters such as the dimensionality. Similar issues also applied to the recent line of work (Foster et al., 2021, 2023b) aiming to provide a unified complexity measure based on the decision-estimation coefficient (DEC); the nature of the two-point lower bound

used there often leaves a gap. We also point to some recent work (Zhang et al., 2024) which adopted the DEC framework and established regret bounds for contextual bandits with graph feedback, but no cross-learning across contexts, based on regression oracles.

2 Hard instance and the regret lower bound

In this section we sketch the proof of the minimax lower bound R ⋆ T ( G,M, C SA ) = Ω( √ β M ( G ) T ) for T ≥ β M ( G ) 3 and general ( G,M ) , implying Theorem 1.1. We first identify a hard instance that corresponds to the graph-theoretic quantity β M ( G ) , and then present the core exploration-exploitation tradeoff in the proof to arrive at the fundamental limit of learning under this instance. This approach has been widely adopted in the bandit literature. The complete proof is deferred to Appendix B.

The proof uses the definition (2) of β M ( G ) to construct M independent sets I 1 , · · · , I M such that I i ̸→ I j for i < j ; by definition, the independent sets I 1 , · · · , I M are disjoint. We then construct a hard instance where the best action under context c ∈ [ M ] is distributed uniformly over I c ; since I c is an independent set, this ensures that the learner must essentially explore all actions in I c under context c . Moreover, the context sequence c T is set to be 11 · · · 122 · · · 2 · · · M , i.e. never goes back to previous contexts. This order ensures that the exploration in I c 1 during earlier rounds provides no information to the exploration in I c 2 during later rounds, whenever c 1 < c 2 . Naïvely, if the learner only explores in each I c under context c , then learning under each context c becomes a multi-armed

√

bandit problem (because I c is itself an independent set), and we can show lower bound T ∑ M c =1 | I c | with appropriate context sequence c T . Maximizing over all possible constructions gives the desired result.

It is possible, however, for the learner to choose actions outside I c to obtain information for the later rounds. To address this challenge, we use a delicate exploration-exploitation tradeoff argument to show that this pure exploration must incur a too large regret to be informative when T ≥ β M ( G ) 3 . Specifically, consider the regret incurred by this pure exploration:

where T c = { t ∈ [ T ] : c t = c } . Then for some absolute constants c 1 and c 2 , the tradeoff can be formulated as two lower bounds of the regret R T :

The first bound is decreasing in the amount of pure exploration, while the second one is increasing. Balancing this tradeoff gives the desired lower bound R T = Ω ( √ β M ( G ) T ) for T ≥ β M ( G ) 3 .

In summary, the key structure used in the proof is that I i ̸→ I j for i < j ; we remark that this does not preclude the possibility that I j → I i for j > i , which underlies the change from α ( G ) to m ( G ) as the number of context increases.

3 Algorithms achieving the regret upper bounds

This section provides algorithms that achieve the claimed regret upper bounds in Theorems 1.3 and 1.4. The crux of these algorithms is to exploit the structure of the feedback graph and choose a small number of actions to explore. Depending on whether the context sequence is self-avoiding or not, the above problem can be reduced to two different kinds of sequential games on the feedback graph. Given solutions to the sequential games, Sections 3.2 and 3.3 will rely on the layering technique to use these solutions on each layer, and propose the final learning algorithms via arm elimination.

3.1 Two sequential games on graphs

In this section we introduce two sequential games on graphs which are purely combinatorial and independent of the learning process. We begin with the first sequential game.

Definition 1 (Sequential game I) . Given a directed graph G = ( V, E ) and a positive integer M , the sequential game consists of M steps, where at each step c = 1 , · · · , M :

the adversary chooses a strongly observable subset A c ⊆ V disjoint from N out ( ∪ c ′ <c D c ′ ) ;

the learner chooses D c ⊆ A c such that D c dominates A c , i.e. A c ⊆ N out ( D c ) . 3

The learner's goal is to minimize the total size ∑ M c =1 | D c | of the sets D c .

The above sequential game is motivated by bandit learning under self-avoiding contexts. Consider a self-avoiding context sequence in the order of 1 , 2 , · · · , M . For c ∈ [ M ] , the set A c represents the 'active set' of actions, i.e. the set of all probably good actions, yet to be explored when context c first occurs. Thanks to the self-avoiding structure, 'yet to be explored' means that A c must be disjoint from N out ( ∪ c ′ <c D c ′ ) . The learner then plays a set of actions D c ⊆ A c to ensure that all actions in A c have been explored at least once; we note that a good choice of D c not only aims to observe all of A c , but also tries to observe as many actions as possible outside A c and make the complement of N out ( ∪ c ′ ≤ c D c ′ ) small. The final cost ∑ M c =1 | D c | characterizes the overall sample complexity to explore every active action once over all contexts.

It is clear that the minimax value of this sequential game is given by

The following lemma characterizes the quantity U ⋆ 1 ( G,M ) up to an O (log | V | ) factor.

Lemma 3.1 (Minimax value of sequential game I) . There exists an absolute constant C > 0 that

Moreover, the learner can achieve a slightly larger upper bound O ( β M ( G ) log 2 | V | ) using a polynomial-time algorithm.

The second sequential game is motivated by bandit learning with an arbitrary context sequence.

Definition 2 (Sequential game II) . Given a directed graph G = ( V, E ) and a positive integer M , the sequential game starts with an empty set D 0 = ∅ , and at time t = 1 , 2 , · · · :

the adversary chooses an integer c t ∈ [ M ] (and a set A c t ⊆ V if c t does not appear before). The adversary must ensure that A c t \ N out ( D t -1 ) is non-empty;

the learner picks a vertex v t ∈ A c t and updates D t ← D t -1 ∪ { v t } .

The game terminates at time T whenever the adversary has no further move (i.e. ∪ c A c ⊆ N out ( D T ) ), and the learner's goal is to minimize the duration T of the game.

The new sequential game reflects the case where the context sequence might not be self-avoiding, so instead of taking a set of actions at once, the learner now needs to take actions non-consecutively. Clearly the sequential game II is more difficult for the learner as it subsumes the sequential game I when the context sequence is self-avoiding: the set D c in Definition 1 is simply the collection of v t 's in Definition 2 whenever c t = c . Consequently, the minimax values satisfy U ⋆ 2 ( G,M ) ≥ U ⋆ 1 ( G,M ) . The following lemma proves an upper bound on U ⋆ 2 ( G,M ) .

Lemma 3.2 (Minimax value of sequential game II) . There exists a polynomial-time algorithm for the learner which achieves

where C > 0 is an absolute constant, β M ( G ) is given in (3) , and

3.2 Learning under self-avoiding contexts

Given a learner's algorithm for the first sequential game, we are ready to provide an algorithm for bandit learning under any self-avoiding context sequence. The algorithm relies on the well-known idea of arm elimination (Even-Dar et al., 2006): for each context c ∈ [ M ] , we maintain an active set A c consisting of all probably good actions so far under this context based on usual confidence bounds of the rewards. To embed the sequential games into the algorithm, we further make use of the layering technique in (Lykouris et al., 2020; Dann et al., 2020): for ℓ ∈ N , we construct the set A c,ℓ as the active set on layer ℓ such that all actions in A c,ℓ -1 have been taken for at least ℓ -1 times. In other words, the active set A c,ℓ is formed based on ℓ -1 reward observations of all currently active actions. As higher layer indicates higher estimation accuracy, the learner now aims to minimize the duration of each layer ℓ , which is precisely the place we will play an independent sequential game.

Algorithm 1: Arm elimination algorithm for self-avoiding contexts

Input: time horizon T , action set [ K ] , context set [ M ] , feedback graph G , a subroutine A for the sequential game I, failure probability δ ∈ (0 , 1) .

Initialize: active sets A c, 1 ← [ K ] for all contexts c ∈ [ M ] on layer 1 .

compute D c,ℓ ⊆ A c,ℓ \ N out ( ∪ c ′ <c D c ′ ,ℓ ) according to the subroutine A , based on past plays

̸

choose each action in D c,ℓ once (break the loop if c t = c or t > T during this process), and update t accordingly;

compute the empirical rewards ¯ r c,a for all actions based on all historic reward observations;

choose the following active set on the next layer:

move to the next layer ℓ ← ℓ +1 .

end end

The description of the algorithm is summarized in Algorithm 1, and we assume without loss of generality that the self-avoiding contexts comes in the order of 1 , . . . , M (the duration of some contexts might be zero). During each context, Algorithm 1 sequentially constructs a shrinking sequence of active sets A c, 1 ⊇ A c, 2 ⊇ · · · , and on each layer ℓ , the algorithm plays the sequential game I based on the current status (past plays ( A c ′ ,ℓ ) c ′ ≤ c , or equivalently ( A c ′ ,ℓ \ N out ( ∪ i<c ′ D i,ℓ )) c ′ ≤ c , of the adversary, and past plays ( D c ′ ,ℓ ) c ′ <c of the learner). 4 After the rewards of all actions of A c,ℓ have been observed once, the algorithm constructs the active set A c,ℓ +1 for the next layer based on the confidence bound (5) and sample size ℓ .

The following theorem summarizes the performance of the algorithm.

Theorem 3.3 (Regret upper bound of Algorithm 1) . Let the subroutine A for the sequential game I be the polynomial-time algorithm given by Lemma 3.1. Then with probability at least 1 -δ , the regret of Algorithm 1 is upper bounded by

On a high level, by classical confidence bound arguments, each action chosen on layer ℓ suffers from an instantaneous regret ˜ O (1 / √ ℓ ) . Moreover, Lemma 3.1 shows that the number of actions chosen on a given layer is at most ˜ O ( β M ( G )) . A combination of these two observations leads to the ˜ O ( √ β M ( G ) T ) upper bound in Theorem 3.3, and a full proof is provided in Appendix C.

3.3 Learning under general contexts

The learning algorithm under a general context sequence is described in Algorithm 2. Similar to Algorithm 1, for each context c we break the learning process into different layers, construct active sets A c,ℓ for each layer, and move to the next layer whenever all actions in A c,ℓ have been observed once on layer ℓ . The only difference lies in the choice of actions on layer ℓ , where the plays from the sequential game II are now used. The following theorem summarizes the performance of Algorithm 2, whose proof is very similar to Theorem 3.3 and deferred to Appendix C.

Algorithm 2: Arm elimination under general contexts

Input: time horizon T , action set [ K ] , context set [ M ] , feedback graph G , a subroutine A for the sequential game II, failure probability δ ∈ (0 , 1) .

Initialize: active sets A ← [ K ] for all contexts c ∈ [ M ] and layers ℓ ≥ 1 ; set of actions

D ℓ ← ∅ chosen on layer ℓ ; the current layer index ℓ ( c ) ← 1 for all c ∈ [ M ]

receive the context c t , and compute the current layer index ℓ t = ℓ ( c t ) ;

according to subroutine A , choose an action a t ∈ A c t ,ℓ t based on the active sets

(

A

c,ℓ

t

)

c

∈

[

M

]

and previously taken actions update the set of actions on layer

ℓ

t

via

compute the new layer index if

ℓ

new

(

c

)

> ℓ

(

c

)

then compute the empirical rewards ¯ r c,a for all actions based on all historic observations; choose the following active set on the new layer:

update the layer index ℓ ( c ) ← ℓ new ( c ) .

end end

end

By the second inequality in Lemma 3.2, Theorem 3.4 implies Theorem 1.4. Corollary 1.5 then follows from the following result.

Lemma 3.5. For undirected or transitively closed graph G , it holds that β dom ( G,M ) = O ( β M ( G ) log | V | ) .

4 Discussions

4.1 Weakly observable feedback graphs

Naturally, we may ask what results we would get under a weaker assumption / a more general feedback structure. If the feedback graph G is instead weakly observable 5 , then under complete cross-learning, an explore-then-commit (ETC) policy can achieve regret ˜ O ( δ ( G ) 1 / 3 T 2 / 3 ) by first exploring the minimum dominating set 6 uniformly for time δ ( G ) 1 / 3 T 2 / 3 , and then committing to the empirically best action that has suboptimality bounded by ˜ O ( δ ( G ) 1 / 3 T -1 / 3 ) with high probability. This matches the existing lower bound in (Alon et al., 2015) and is hence near-optimal.

̸

D

ℓ

D

t

ℓ

t

on the current layer;

D

ℓ

t

∪ {

a

t

}

;

{

) = min

ℓ

:

A

c,ℓ

⊈

N

out

(

D

c,ℓ .

ℓ

new

(

c

ℓ

)

}

;

4.2 Incomplete cross-learning

It is possible to further relax the assumption of complete cross-learning. Suppose the feedback across contexts is characterized by another directed graph G [ M ] (and denote G [ K ] across actions respectively), and consider a product feedback graph G [ K ] × G [ M ] over the context-action pairs such that ( a 1 , c 1 ) → ( a 2 , c 2 ) if a 1 → a 2 in G [ K ] and c 1 → c 2 in G [ M ] . Then we can get the following generalized results.

4.2.1 Weakly observable feedback graphs on actions

When the feedback graph G [ K ] is weakly observable, following the argument in Section 4.1, we can achieve regret ˜ O ( ( δ ( G [ K ] ) m ( G [ M ] ) ) 1 / 3 T 2 / 3 ) by running an ETC subroutine for each context as follows: for every context c ∈ [ M ] , we keep an 'exploration' counter n c . At each time t with context c t , if n c t ≲ δ ( G [ K ] ) 1 / 3 m ( G [ M ] ) -2 / 3 T 2 / 3 , we are in the 'exploration' stage and continue to uniformly explore the minimum dominating set of G [ K ] . Then we increase the counter for all observed contexts, i.e. n c ← n c +1 for all c ∈ N out ( c t ) in G [ M ] . Otherwise, we 'commit' to the empirically best action that has suboptimality bounded by ˜ O ( ( δ ( G [ K ] ) m ( G [ M ] ) ) 1 / 3 T -1 / 3 ) with high probability.

The key observation is that the number of times we are in the 'exploration' stage is ˜ O ( ( δ ( G [ K ] ) m ( G [ M ] ) ) 1 / 3 T 2 / 3 ) . This can be seen from a layering argument, similar to the one in Section 3.2, that the number of actually played contexts on each layer is at most m ( G [ M ] ) . Together with the bounded rewards and the bounded suboptimality in the 'commit' stage, this proves the regret upper bound.

Combining the context sequence construction in Section 2 and the lower bound argument in (Alon et al., 2015), one can also prove a matching lower bound Ω ( ( δ ( G [ K ] ) m ( G [ M ] ) ) 1 / 3 T 2 / 3 ) .

4.2.2 Strongly observable feedback graphs on actions

When G [ K ] is strongly observable, it is straightforward to generalize our upper (for self-avoiding contexts) and lower bounds in Theorem 1.1 and 1.3 with β M ( G [ K ] ) replaced by

and β dom ( G [ K ] ) in Theorem 3.4 by

where the new graph quantities are defined on the product graph G [ K ] × G [ M ] . For general contexts, this gives a tight upper bound ˜ O (√ Tβ M ( G [ K ] × G [ M ] ) ) when G [ K ] and G [ M ] are either both undirected or both transitively closed 7 . Most generally, we have a loose upper bound ˜ O (√ T min { m ( G [ K ] × G [ M ] ) , α ( G [ K ] ) M } ) .

4.3 Gap between upper and lower bounds

Although we provide tight upper and lower bounds for specific classes of context sequences (selfavoiding in Theorem 1.3) or feedback graphs (undirected or transitively closed in Corollary 1.5), in general the quantities β M ( G ) in Theorem 1.1 and min { β M ( G ) , m ( G ) } in Theorem 1.4 exhibit a gap. The following lemma gives an upper bound on this gap.

Lemma 4.1. For any graph G , it holds that

where ρ ( G ) denotes the length of the longest path in G .

Lemma 4.1 shows that if G does not contain long paths or M is large, the gap between β M ( G ) and β M ( G ) is not significant. We also comment on the challenge of closing this gap. First, we do not know a tight characterization of the minimax value of the sequential game II (cf. Definition 2), and the upper bound β dom ( G,M ) in Lemma 3.2 could be loose, as shown in the following example.

̸

Example 1. Consider an acyclic graph G = ( V, E ) with KM vertices { ( i, j ) } i ∈ [ K ] ,j ∈ [ M ] and edges ( i, j ) → ( i ′ , j ′ ) if either i < i ′ and j = j ′ , or i = i ′ and j < j ′ , and all self-loops. By choosing B c = V c = { ( i, c ) : i ∈ [ K ] } in the definition of β dom ( G,M ) in Lemma 3.2, it is clear that β dom ( G,M ) = KM . However, we show that the minimax value is U ⋆ 2 ( G,M ) = K + M -1 . The lower bound follows from U ⋆ 2 ( G,M ) ≥ U ⋆ 1 ( G,M ) ≥ β M ( G ) ≥ K + M -1 , as I 1 = { ( i, M ) : i ∈ [ K ] } and I c = { (1 , M + 1 -c ) } for 2 ≤ c ≤ M satisfy the constraints in the definition of β M ( G ) in (2) . For the upper bound, we consider the following strategy for the learner in the sequential game II: v t = ( i t , j t ) is the smallest element (under the lexicographic order over pairs) in A c t \ N out ( D t -1 ) . To show why U ⋆ 2 ( G,M ) ≤ K + M -1 , let D c be the final set of vertices chosen by the learner under context c . By the lexicographic order and the structure of G , each D c can only consist of vertices in one column. Moreover, for different c = c ′ , the row indices of D c must be entirely no smaller or entirely no larger than the row indices of D c ′ . These constraints ensure that ∑ M c =1 | D c | ≤ K + M -1 .

̸

This example shows the importance of non-greedy approaches when choosing v t . In the special case where A c = { ( i, c ) : i ∈ [ K ] } is the c -th column, within A c this is an independent set, so any greedy approach that does not look outside A c will treat the vertices in A c indifferently. In contrast, the above approach makes use of the global structure of the graph G .

The second challenge lies in the proof of the lower bound. Instead of the sequential game where the adversary and the learner take turns to play actions, the current lower bound argument assumes that the adversary tells all his plays to the learner ahead of time. We expect the sequential structure to be equally important for the lower bounds, and it is interesting to work out an argument for the minimax lower bound to arrive at a sequential quantity like U ⋆ 2 ( G,M ) .

4.4 Other open problems

Performance of the UCB algorithm. The UCB algorithm under feedback graphs has been analyzed for both multi-armed (Lykouris et al., 2020) and contextual bandits (Dann et al., 2020). However, both results only show a regret upper bound ˜ O ( √ m ( G ) T ) , even in the case of multi-armed bandits (i.e. M = 1 ). It is interesting to understand for algorithms without forced exploration (such as UCB), if the MAS number m ( G ) (rather than α ( G ) or β M ( G ) ) turns out to be fundamental.

Regret for small T . Note that our upper bounds hold for all values of T , but our lower bound requires T ≥ β M ( G ) 3 . This is not an artifact of the analysis, as the optimal regret becomes fundamentally different for smaller T . The case of multi-armed bandits has been solved completely in a recent work (Kocák and Carpentier, 2023), where the regret is a mixture of √ T and T 2 / 3 terms. We anticipate the same behavior for contextual bandits, but the exact form is unknown.

Stochastic contexts. In this paper we assume that the contexts are generated adversarially, but the case of stochastic contexts also draws some recent attention (Balseiro et al., 2019; Schneider and Zimmert, 2024), and sometimes there is a fundamental gap between the optimal performances under stochastic and adversarial contexts (Han et al., 2024). It is an interesting question whether this is the case for contextual bandits with graph feedback.

References

Alekh Agarwal, Miroslav Dudík, Satyen Kale, John Langford, and Robert Schapire. Contextual bandit learning with predictable rewards. In Artificial Intelligence and Statistics , pages 19-26. PMLR, 2012.

Shipra Agrawal and Nikhil Devanur. Linear contextual bandits with knapsacks. Advances in Neural Information Processing Systems , 29, 2016.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

Noga Alon and Joel H Spencer. The probabilistic method . John Wiley & Sons, 2016.

Noga Alon, Nicolo Cesa-Bianchi, Ofer Dekel, and Tomer Koren. Online learning with feedback graphs: Beyond bandits. In Conference on Learning Theory , pages 23-35. PMLR, 2015.

Noga Alon, Nicolo Cesa-Bianchi, Claudio Gentile, Shie Mannor, Yishay Mansour, and Ohad Shamir. Nonstochastic multi-armed bandits with graph-structured feedback. SIAM Journal on Computing , 46(6):1785-1826, 2017.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Santiago Balseiro, Negin Golrezaei, Mohammad Mahdian, Vahab Mirrokni, and Jon Schneider. Contextual bandits with cross-learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Gábor Bartók, Dean P Foster, Dávid Pál, Alexander Rakhlin, and Csaba Szepesvári. Partial monitoring-classification, regret bounds, and algorithms. Mathematics of Operations Research , 39(4): 967-997, 2014.

Djallel Bouneffouf, Irina Rish, and Charu Aggarwal. Survey on applications of multi-armed and contextual bandits. In 2020 IEEE Congress on Evolutionary Computation (CEC) , pages 1-8. IEEE, 2020.

Houshuang Chen, Shuai Li, Chihao Zhang, et al. Understanding bandits with graph feedback. Advances in Neural Information Processing Systems , 34:24659-24669, 2021.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

Vasek Chvatal. A greedy heuristic for the set-covering problem. Mathematics of operations research , 4(3):233-235, 1979.

Christoph Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, and Karthik Sridharan. Reinforcement learning with feedback graphs. Advances in Neural Information Processing Systems , 33: 16868-16878, 2020.

Khaled Eldowa, Emmanuel Esposito, Tom Cesari, and Nicolò Cesa-Bianchi. On the minimax regret for online learning with feedback graphs. Advances in Neural Information Processing Systems , 36, 2024.

Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

Uriel Feige, Shafi Goldwasser, László Lovász, Shmuel Safra, and Mario Szegedy. Approximating clique is almost np-complete. In [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science , pages 2-12. IEEE Computer Society, 1991.

Dean Foster, Dylan J Foster, Noah Golowich, and Alexander Rakhlin. On the complexity of multiagent decision making: From learning in games to partial monitoring. In The Thirty Sixth Annual Conference on Learning Theory , pages 2678-2792. PMLR, 2023a.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

Dylan J Foster, Noah Golowich, and Yanjun Han. Tight guarantees for interactive decision making with the decision-estimation coefficient. In The Thirty Sixth Annual Conference on Learning Theory , pages 3969-4043. PMLR, 2023b.

Zijun Gao, Yanjun Han, Zhimei Ren, and Zhengqing Zhou. Batched multi-armed bandits problem. Advances in Neural Information Processing Systems , 32, 2019.

Fabrizio Grandoni. A note on the complexity of minimum dominating set. Journal of Discrete Algorithms , 4(2):209-214, 2006.

Yanjun Han, Tsachy Weissman, and Zhengyuan Zhou. Optimal no-regret learning in repeated first-price auctions. Operations Research , 2024.

Botao Hao, Tor Lattimore, and Chao Qin. Contextual information-directed sampling. In International Conference on Machine Learning , pages 8446-8464. PMLR, 2022.

Woonghee Tim Huh and Paat Rusmevichientong. A nonparametric asymptotic analysis of inventory planning with censored demand. Mathematics of Operations Research , 34(1):103-123, 2009.

Richard M Karp. Reducibility among combinatorial problems . Springer, 2010.

Tomáš Kocák and Alexandra Carpentier. Online learning with feedback graphs: The true shape of regret. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 17260-17282. PMLR, 23-29 Jul 2023.

Tor Lattimore. Minimax regret for partial monitoring: Infinite outcomes and rustichini's regret. In Conference on Learning Theory , pages 1547-1575. PMLR, 2022.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017.

Thodoris Lykouris, Éva Tardos, and Drishti Wali. Feedback graph regret bounds for thompson sampling and ucb. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the 31st International Conference on Algorithmic Learning Theory , volume 117 of Proceedings of Machine Learning Research , pages 592-614. PMLR, 08 Feb-11 Feb 2020.

Shie Mannor and Ohad Shamir. From bandits to experts: On the value of side-observations. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011.

Jon Schneider and Julian Zimmert. Optimal cross-learning for contextual bandits with unknown context distributions. Advances in Neural Information Processing Systems , 36, 2024.

Mengxiao Zhang, Yuheng Zhang, Olga Vrousgou, Haipeng Luo, and Paul Mineiro. Practical contextual bandits with feedback graphs. Advances in Neural Information Processing Systems , 36, 2024.

Haoyu Zhao and Wei Chen. Stochastic one-sided full-information bandit. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 150-166. Springer, 2019.

Yinglun Zhu and Paul Mineiro. Contextual bandits with smooth regret: Efficient learning in continuous action spaces. In International Conference on Machine Learning , pages 27574-27590. PMLR, 2022.

A Auxilary Lemmas

Lemma A.1 (Lemma 8 of (Alon et al., 2015)) . For any directed graph G = ( V, E ) , one has δ ( G ) ≤ 50 α ( G ) log | V | .

For a directed graph G , there is a well-known approximate algorithm for finding the smallest dominating set: starting from D = ∅ , recursively find the vertex v with the maximum out-degree in the subgraph induced by V \ N out ( D ) , and update D ← D ∪{ v } . The following lemma summarizes the performance of this algorithm.

Lemma A.2 (Chvatal (1979)) . For any graph G = ( V, E ) , the above procedure outputs a dominating set D with

Lemma A.3 (A special case of Lemma 3 in (Gao et al., 2019)) . Let Q 1 , . . . Q n be probability measures on some common measure space (Ω , F ) , with n ≥ 2 , and Φ : Ω → [ n ] any measurable test function. Then

̸

B Deferred Proof for the Lower Bound

In this appendix, we give the complete proof of the minimax lower bound R ⋆ T ( G,M, C SA ) = Ω( √ β M ( G ) T ) for T ≥ β M ( G ) 3 and general ( G,M ) , implying Theorem 1.1.

Let I 1 , · · · , I M be the independent sets achieving the maximum in (2), by removing empty sets, combining ( I i , I i +1 ) whenever | I i | = 1 , and possibly removing the last set I M if | I M | = 1 , we arrive at disjoint subsets J 1 , · · · , J m of [ K ] such that the following conditions hold:

m ≤ M , K c ≜ | J c | ≥ 2 for all c ∈ [ m ] , and J i ̸→ J j for i < j ;

the only possible non-self-loop edges among J c = { a c, 1 , · · · , a c,K c } can only point to a c, 1 ;

∑ m c =1 K c ≥ ∑ M c =1 | I c | -1 = β M ( G ) -1 ≥ β M ( G ) / 2 whenever β M ( G ) ≥ 2 . 8

Given sets J 1 , · · · , J m , we are ready to specify the hard instance. Let u = ( u 1 , · · · , u m ) ∈ Ω := [ K 1 ] ×··· × [ K m ] be a parameter vector, the joint reward distribution P u of ( r t,c,a ) c ∈ [ M ] ,a ∈ [ K ] is a product distribution P u = ∏ c ∈ [ M ] ,a ∈ [ K ] Bern ( µ u c,a ) , where the mean parameters for the Bernoulli distributions are µ u c,a = 0 whenever c > m , and

̸

Here ∆ ∈ (0 , 1 / 4) is a gap parameter to be chosen later. We summarize some useful properties from the above construction:

under context c ∈ [ m ] , the best action under P u is a c,u c , and all other actions suffer from an instantaneous regret at least ∆ ;

under context c ∈ [ m ] , actions outside J c suffer from an instantaneous regret at least 1 / 4 ;

for u = ( u 1 , · · · , u m ) ∈ Ω and u c := ( u 1 , · · · , u c -1 , 1 , u c +1 , · · · , u m ) , the KL divergence between the observed reward distributions P u ( a ) and P u c ( a ) when choosing the action a is

Here (a) follows from our construction of P u that the only difference between P u and P u c is the reward of action a c,u c , which is observed iff a → a c,u c ; (b) is due to the property of ( J 1 , · · · , J m ) that any action in J ≤ c \{ a c,u c } does not point to a c,u c , where J ≤ c := ∪ c ′ ≤ c J c ′ .

Finally, we partition the time horizon [ T ] into consecutive blocks T 1 , · · · , T m (whose sizes will be specified later), and choose the context sequence as c t = c for all t ∈ T c . For a fixed policy, let R T be the worst-case expected regret of this policy. By the second property of the construction, it is clear that for all u ∈ Ω ,

When u is uniformly distributed over Ω , we also have

̸

̸

where (a) lower bounds the minimax regret by the Bayes regret, with the help of the first property; (b) decomposes the expectation over uniformly distributed u into u \{ u c } and u c ∈ [ K c ] ; (c) follows from Lemma A.3; (d) uses the convexity of x ↦→ e -x ; (e) results from the chain rule of KL divergence, the third property of the construction, and that K c ≥ 2 for all c ∈ [ m ] .

Next we upper bound the exponent in (7) as

where (f) crucially uses that u c = ( u 1 , · · · , u c -1 , 1 , u c +1 , · · · , u m ) does not depend on u c , so that the sum may be moved inside the expectation to get ∑ K c u c =2 ( a s = a c,u c ) ≤ 1 . Now choosing

we arrive at the final lower bound

where the last inequality is due to the assumption T ≥ β M ( G ) 3 . Now we readily conclude from (8) the desired lower bound R T = Ω( √ β M ( G ) T ) .

C Deferred Proofs for the Upper Bounds

Throughout the proofs, we will use α ( A ) ≜ α ( G | A ) (resp. δ ( A ) , m ( A ) ) to denote the independence number (resp. dominating number, MAS number) of the subgraph induced by A ⊆ V when the graph G is clear from the context.

C.1 Proof of Lemma 3.1

The lower bound U ⋆ 1 ( G,M ) ≥ β M ( G ) is easy: let I 1 , · · · , I M be M independent sets with I i ̸→ I j for all i < j . Then the choice A c = I c is always feasible for the adversary, for I c is disjoint from N out ( ∪ c ′ <c I c ′ ) . For the learner, the only subset D c ⊆ I c which dominates I c is D c = I c , hence U ⋆ 1 ( G,M ) ≥ ∑ M c =1 | I c | . Taking the maximum then gives U ⋆ 1 ( G,M ) ≥ β M ( G ) by (2).

To prove the upper bound U ⋆ 1 ( G,M ) ≤ Cβ M ( G ) log | V | , the learner chooses D c as follows. Given A c , the learner finds the smallest dominating set J c ⊆ A c and the largest independent set I c ⊆ A c , and sets D c = I c ∪ J c . Clearly this choice of D c is feasible for the learner, and since I i ̸→ A j for i < j , we have I i ̸→ I j as well. Consequently,

where (a) uses | J c | = δ ( A c ) = O ( α ( A c ) log | V | ) = O ( | I c | log | V | ) in Lemma A.1, and (b) follows from the definition of β M ( G ) in (2).

Since it is NP-hard to find either the smallest dominating set or the largest independent set (Karp, 2010; Grandoni, 2006), the above choice of D c is not computationally efficient. To arrive at a polynomial-time algorithm, we may use a greedy algorithm to find an O (log | V | ) -approximate smallest dominating set ˜ J c such that | ˜ J c | = O ( δ ( A c ) log | V | ) (cf. Lemma A.2). For I c , although finding the largest independent set is APX-hard (Feige et al., 1991), the constructive proof of (Alon et al., 2015, Lemma 8) gives a polynomial-time randomized algorithm which finds ˜ I c ⊆ A c such that | ˜ I c | = Ω( δ ( A c ) / log | V | ) and the average degree among ˜ I c is at most O (1) . The learner now chooses D c = ˜ I c ∪ ˜ J c . By the average degree constraint and Turán's theorem (Alon and Spencer, 2016, Theorem 3.2.1), each ˜ I c contains an independent subset I c with | I c | = Ω( | ˜ I c | ) . Since

C.2 Proof of Lemma 3.2

The second inequality is straightforward: β dom ( G,M ) ≤ m ( G ) since ∪ c B c is acyclic, and the other inequality follows from | B c | = O ( δ ( V c ) log | V | ) = O ( α ( V c ) log 2 | V | ) in Lemma A.1. To

prove the first inequality, we consider a simple greedy algorithm for the learner, where v t ∈ A c t is the vertex with the largest out-degree in the induced subgraph by A c t \ N out ( D t -1 ) . Intuitively, A c t \ N out ( D t -1 ) is the set of nodes in A c t that remain unexplored by the learner by time t . Under this greedy algorithm, for c ∈ [ M ] , define

We claim that V c are pairwise disjoint and | B c | ≤ δ ( V c )(1 + log | V | ) , and thereby complete the proof of ∑ M c =1 | B c | ≤ β dom ( G,M ) . The first claim simply follows from the pairwise disjointness of the sets N out ( v t ) ⋂ ( A c t \ N out ( D t -1 )) for different t . For the second claim, let t 1 < · · · < t n be all the time steps where c t = c , and

It is clear that V c,i +1 = V c,i \ N out ( v t i ) . Since v t i has the largest out-degree in the induced subgraph by A c \ N out ( D t i -1 ) ⊇ V c,i , and N out ( v t i ) ∩ ( A c \ N out ( D t i -1 )) = N out ( v t i ) ∩ V c,i , this is also the vertex with the largest out-degree in V c,i . Therefore, the sets { V c,i } n +1 i =1 evolve from V c, 1 = V c to V c,n +1 = ∅ as follows: one recursively picks the vertex with the largest out-degree in V c,i , and removes its out-neighbors to get V c,i +1 . This is a well-known approximate algorithm for computing δ ( V c ) , described above Lemma A.2, with

as desired.

C.3 Proof of Lemma 3.5

If G is undirected, then β dom ( G ) ≤ m ( G ) = α ( G ) = β M ( G ) easily holds. It remains to consider the case where G is transitively closed.

Note that in a transitively closed graph, every vertex set has an independent dominating subset (by tracing to the ancestors). Therefore, for the maximizing sets B 1 , · · · , B M in the definition of β dom , we may run the above procedure to find independent dominating subsets I 1 , · · · , I M of V 1 , . . . , V M respectively, with I c ⊆ B c and

Now consider the induced subgraph G ′ by ∪ c I c . Clearly G ′ is acyclic, and the length of longest path in G ′ is at most M (otherwise, two points on a path belong to the same I c , and transitivity will violate the independence). Invoking Lemma 4.1 now gives

and combining the above two inequalities completes the proof.

C.4 Proof of Theorem 3.3

The proof is decomposed into three claims: with probability at least 1 -δ ,

for every c ∈ [ M ] and ℓ ∈ N , when the confidence bound (5) is formed, either every action in A c,ℓ has been observed for at least ℓ times or | A c,ℓ | = 1 ;

for every c ∈ [ M ] and ℓ ∈ N , the best action a ⋆ ( c ) under context c belongs to A c,ℓ , and every action in A c,ℓ has suboptimality gap at most min { 1 , 4∆ ℓ -1 } , with ∆ ℓ ≜ √ log(2 MKT/δ ) /ℓ ;

for every ℓ ∈ N , the total number N ℓ of actions taken on layer ℓ in those subsets A c,ℓ with | A c,ℓ | > 1 is O ( β M ( G ) log 2 K ) .

The first claim simply follows from (1) when G is strongly observable, any subset of size > 1 is strongly observable, and (2) A c,ℓ ⊆ N out ( ∪ c ′ ≤ c D c ′ ,ℓ ) required by the sequential game I, so that on each layer ℓ ′ ∈ [ ℓ ] there is at least one action which observes a ∈ A c,ℓ ⊆ A c,ℓ ′ . For the second claim, the first claim, the usual Hoeffding concentration, and a union bound imply that | ¯ r c,a -µ c,a | ≤ ∆ ℓ in (5) for all c ∈ [ M ] and a ∈ A c,ℓ , when | A c,ℓ | > 1 and with probability at least 1 -δ . Conditioned on this event:

The best arm a ⋆ ( c ) is not eliminated by (5), because ¯ r c,a ⋆ ( c ) ≥ µ c,a ⋆ ( c ) -∆ ℓ ≥ max a ′ ∈ A c,ℓ µ c,a ′ -∆ ℓ ≥ max a ′ ∈ A c,ℓ ¯ r c,a ′ -2∆ ℓ ;

The instantaneous regret of choosing any action a ∈ A c,ℓ +1 is at most min { 1 , 4∆ ℓ } , for µ c,a ≥ ¯ r c,a -∆ ℓ ≥ ¯ r c,a ⋆ ( c ) -3∆ ℓ ≥ µ c,a ⋆ ( c ) -4∆ ℓ , and | µ c,a -µ c,a ⋆ ( c ) | ≤ 1 trivially holds.

Consequently the second claim holds for | A c,ℓ | > 1 . For the case | A c,ℓ | = 1 , note that starting from A c, 1 = [ K ] , the above argument implies that the best arm is never eliminated until | A c,ℓ ′ | = 1 at some layer ℓ ′ ≤ ℓ conditioned on the high probability event, which implies the single action in A c,ℓ is the best action and incurs 0 regret. The last claim is simply the reduction to the sequential game I, where Lemma 3.1 shows that N ℓ = ∑ M c =1 | D c,ℓ | = O ( β M ( G ) log 2 K ) .

Combining the above claims and that we incur 0 regret whenever | A c,ℓ | = 1 , with probability at least 1 -δ , we have

where N ℓ ≤ N := O ( β M ( G ) log 2 K ) , and ∑ ∞ ℓ =1 N ℓ = T . It is straightforward to see that the choice N 1 = · · · = N m = N and N m +1 = T -Nm for a suitable m ∈ N maximizes the above sum, and the maximum value is the target regret upper bound in Theorem 3.3.

C.5 Proof of Theorem 3.4

The proof follows verbatim the same lines in the proof of Theorem 3.3, except that the total number N ℓ of actions taken on layer ℓ is now at most β dom ( G,M ) in the third claim, by Lemma 3.2.

C.6 Proof of Lemma 4.1

Let V 1 ⊆ V be a maximum acyclic subset, then ρ ( G | V 1 ) ≤ ρ ( G ) . Consider the following recursive process: at time t = 1 , 2 , · · · , let J t be the set of vertices in V t with in-degree zero (which always exist as V t is acyclic), and V t +1 = V t \ J t . This recursion can only last for at most ρ ( G ) steps, for otherwise there is a path of length larger than ρ ( G ) . Then each J t is an independent set, for every vertex of J t has in-degree zero in V t ⊇ J t . For the same reason we also have J i ̸→ J j for i > j . This means that

where the last inequality follows from picking M largest sets among { J t } .

C.7 Proof of the statement in Section 4.2.2

In this section, we show that when G [ K ] and G [ M ] are either both undirected or both transitively closed, the product graph quantities β dom := β dom ( G [ K ] × G [ M ] ) and β M := β M ( G [ K ] × G [ M ] ) satisfy

Combining with the Ω( √ β M T ) lower bound, this shows the tightness of the upper bound ˜ O ( √ β dom T ) . The idea here is similar to Section C.3:

When G [ K ] and G [ M ] are both undirected, the union set ∪ c B c in the definition of β dom is an independent set thanks to the acyclic requirement. Thus β dom ≤ β M .

When G [ K ] and G [ M ] are both transitively closed, for the maximizing sets B 1 , . . . , B M in the definition of β dom , we can again find independent dominating subsets I c ⊆ B c (by transitive closure of G [ K ] ) with

Now it suffices to find independent subsets J c ⊆ [ K ] × { c } that satisfy J c ̸→ J c ′ when c < c ′ and ∑ c | J c | = ∑ c | I c | . Toward this end, we first suppose there are c and c ′ such that u c → u c ′ and v c ← v c ′ for u c , v c ∈ I c and u c ′ , v c ′ ∈ I c ′ . This implies c ↔ c ′ in G [ M ] by the product graph structure. Then

I c | [ K ] and I c ′ | [ K ] are disjoint since ⋃ c I c is acyclic;

by transitive closure of G [ K ] and that I c , I c ′ are independent, I c | [ K ] ∪ I c ′ | [ K ] has path length at most 1;

from above, there exist disjoint and independent sets S 1 and S 2 such that S 1 ∪ S 2 = I c | [ K ] ∪ I c ′ | [ K ] and S 1 ̸→ S 2 .

where we denote the set projection

Without loss of generality, assume c < c ′ . In this case, we can 'rearrange' them by letting J c = S 1 ×{ c } and J c ′ = S 2 ×{ c ′ } , so J c ̸→ J c ′ . Now suppose there is a loop on the set level, i.e. there are c 1 , . . . , c m ∈ [ M ] with I c 1 → ··· → I c m → I c 1 . Similarly, we must have that { c 1 , . . . , c m } form a clique in G [ M ] , I c 1 | [ K ] , . . . , I c m | [ K ] disjoint in G [ K ] , and the path length in ⋃ m j =1 I c j is at most m . Then we can again 'rearrange' them and get independent sets J c j ̸→ J c k for c j < c k and j, k ∈ [ m ] , and ∑ m j =1 | J c j | = ∑ m j =1 | I c j | . In other cases, we simply let J c = I c and arrive at J 1 , . . . , J M that are acyclic on the set level, i.e. up to reordering of the indices, we have J c ̸→ J c ′ when c < c ′ . Together with Eq (9), we have

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction include a lower bound and two upper bounds (and algorithms achieving them) for the considered setup, which are the contributions of this work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The gap between the lower bound and the upper bound under the most general assumptions is discussed with an example in Section 4.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The complete proof for the lower bound is in Appendix B and those for the upper bounds are in Appendix C. Auxilary lemmas are collected and referenced in Appendix A. The problem setup is in Section 1.2. Assumptions are stated in the theorem statements.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This work does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work aims to understand the theoretical limits of a family of contextual bandit problems and has no direct societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work is theoretical and has no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This work does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This work does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve or include such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve or include such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of contextual bandits with finitely many contexts, stochastic rewards and a directed feedback graph (assumed to contain all self loops) across actions. They study the setting of \"complete cross-learning\" where the reward feedback of the chosen action is observed across all contexts. In this setting, the authors establish a lower bound of $\\Omega(\\sqrt{\\beta_M(G) T})$ where $\\beta_M(G)$ is a graph-dependent quantity which lies between the independence number and the maximum acyclic subgraph of the feedback graph, and prove that this lower bound is tight for a specific class of context sequences by designing an efficient algorithm with a matching regret bound. Furthermore, the authors provide an upper bound for general context sequences with a graph dependence that improves upon the maximum acyclic subgraph, but in general may not be tight.\n\nSTRENGTHS:\n* The authors establish regret upper and lower bounds that constitute a considerable step towards characterizing the minimax regret in contextual bandits with feedback graphs in the complete cross-learning framework.\n* The authors provide an overview of the hard instance construction which helps the reader understand the difficulty of minimizing regret in this setting.\n* The idea of incorporating a sequential graph-theoretic zero sum game in a bandit algorithm seems novel and very interesting.\n* Even though the authors close the gap completely only for specific types of context sequences, they also provide an upper bound for general sequences which improves upon the known upper bounds in the literature.\n\nWEAKNESSES:\n* My main issue with the presentation of the main results is with the comparison with the previous related work of [1] (lines 147-148). While the authors do mention that this work considers the setting of feedback graphs over the contexts as well as over the arms, it should be mentioned that it is not how the problem is presented in [1]. Rather, they consider a tabular RL problem in which the states correspond to the contexts, with the crucial difference that the transition between states is governed by some stochastic process. Since in the authors' work there is no assumption regarding the transition between contexts (that is, it could be adversarial), it seems not quite fair to compare their results to those of [1], specifically the regret lower bound. Indeed, the lower bound described by the authors is easily seen to be inapplicable in the setting of [1] as they vary the contexts (or states) in a very controlled adversarial manner. Therefore, I think the authors should emphasize that their lower bound does not provide a strengthening of the lower bound given in [1] for the RL setup.\n\n\n\nReferences:\n[1] Christoph Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, and Karthik Sridharan. Reinforcement learning with feedback graphs.\n\nQUESTIONS:\nI would appreciate it if the authors could address my main concern under \"Weaknesses\" and provide a more careful comparison with the work of [1].\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider the problem of contextual bandits with a feedback graph, for finite context space.  In the presented setting, taking an action reveals the rewards for all neighboring actions in the feedback graph for all contexts. The authors propose $\\beta_M(G)$, a theoretical quantity in which $M$ is the number of contexts and $G$ is the feedback graph, with the goal of characterizing the hardness of learning for this class of problems. The authors prove a lower bound of  $\\Omega(\\sqrt{\\beta_M(G)T})$ where $T$ is the number of rounds.  The authors also present a near-optimal upper bound of $\\widetilde{O}(\\sqrt{\\min (\\bar{\\beta}(G),m(G))T})$.\n\nSTRENGTHS:\n1.\tThe insights in the paper are interesting, it characterizes the difference between MAB with graph feedback and Contextual MAB with graph feedback, for small and finite context space.\n\n2.\tThe authors present upper and lower bounds for the problem.\n\n3.\tThe algorithmic approach taken to produce both results is elegant, especially the use of the arm elimination technique.\n\nWEAKNESSES:\n1.\tIn contextual MAB literature, the main difficulty is that the context space is large, and in each episode, the feedback is revealed only for the current context. In the discussed setting, the feedback is revealed for all contexts. This significantly reduces the inherent difficulty of the contextual influence. Hence,\n\n(a)\tCan you please explain how having a context made the learning harder w.r.t the non-contextual MAB with feedback graph in this setting, conceptually?\n\n(b)\tHave you considered the standard Contextual setting in which only the feedback of the current context is released? Can you adjust your results to this setting as well? \n\n2.\tIt would benefit the reader if an additional explanation of equation (4) will be given. So is for the result stated in Lemma 3.1.  \n\n3.\tTypos - A comment that was left in the text: line 20: \"to name a few\".\n\nQUESTIONS:\n1.\tSee weakness 1.\n\n2.\tCan you please explain the conclusions to corollary 1.2? Specifically, the dependency of $I_c$ in the context is unclear to me, and why it implies that $\\beta_M(G) = m(G)$.\n\n3.\tCan you please provide some more intuition regarding the behavior of $\\beta_M(G)$, and an example for a calculation of it for simple graphs?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies contextual online learning when the feedback received by the learner is regulated by a feedback graph. The setting is as follows: the actions constitute the nodes of a directed graph $G$, and playing action $a$ at time $t$ when the context is $x_t$ reveals not only the loss incurred by that action at that time, for that context but the losses of all the neighboring actions, for all possible $m$ contexts. While the contexts are generated adversarially, the losses are i.i.d.. \n\nThe non-contextual problem is well understood, with tight minimax regret guarantees holding for both the adversarial and stochastic settings. These rates depend on both the time horizon $T$ and some graph parameters. In particular, for strongly observable graphs (as the ones studied in this paper), the rate is known to be $\\sqrt{T \\alpha}$, where $\\alpha$ is the independence number of the feedback graph. \n\nPrevious results achieve a regret bound of $O(\\sqrt{T m})$ for the contextual problem, where $m(G)$ is the maximum acyclic subgraph number, which is complemented by the abovementioned $\\Omega(\\sqrt{T \\alpha})$ lower bound. This paper investigates the gap in the graph-dependent parameter in the minimax rate. Note, $\\alpha$ = $m$ for undirected graphs. \n\nThe contribution of the paper are as follows: \nwhen the number of contexts is large ($\\ge m$), then the $\\sqrt{T m}$ rate is tight\nin general, a lower bound of $\\Omega(\\sqrt \\beta_m)$ is proved, where $\\beta_m$ is a new graph parameter which crucially depends on the number $m$ of contexts and gracefully interpolates between $\\alpha$ and $m$. \nimproved upper bounds are then proved for special context sequences and the general problem\n\nSTRENGTHS:\n1. Online learning with feedback graphs is a relevant problem with a long literature in NeurIPS and ICML\n2. Studying this problem with contexts is fairly natural, and has already been studied\n3. the paper presents a consistent set of results and manages to present them nicely in the intro. Due to space constraints, the technical parts are, however, only roughly sketched.\n\nWEAKNESSES:\n- the problem is not closed: there is still a significant gap in the right graph theoretic parameter\n- the result only holds in the stochastic setting. What can be said in the adversarial setting?\n- the lower-bound construction is fairly natural (at a high level)\n- the graph theoretic parameter introduced is artificial and way less natural than the ones present in the non-contextual settings.\n\nMinor comments\n- please update the references: e.g., Schneider and Zimmert and Zhang et al have been published\n- please consider adding some further references to the learning with feedback graph literature. (see also questions)\n\nQUESTIONS:\nWhat is the relationship between your graph parameter and the ones in Eldowa et al ”On the Minimax Regret for Online Learning with Feedback Graphs”. NeurIPS 2023?\n\nLIMITATIONS:\nNo potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors consider the problem of contextual bandits with feedback graphs and aim to achieve a tighter dependency on graph-dependent quantities.\nFiguring out the correct dependency on graph-dependent quantities is a notably challenging problem in the standard MAB framework, as aspects such as whether the feedback graph is directed, whether it has self-loops, or whether the time horizon is large compared to the size of the graphs are crucial components to figure out how much information can be extracted from the feedback graph.\\\\\n\nIn this work, the authors consider a contextual MAB problem where there is a feedback graph across actions and cross-learning between contexts. The authors propose a minimax lower bound for this context, which depends on a quantity $\\beta_M(G)$, where $M$ is the number of contexts. \nThe authors then show that this lower bound is tight for certain classes of problems such as self-avoiding contexts, which is a problem setting where the environment regularly switches from one context to the next but doesn't ever come back to contexts that have already been seen in the past. \nThe authors also derive an upper bound in the general setting using a different algorithm.\n\nSTRENGTHS:\nThe authors propose a detailed characterization of the challenges of contextual MAB with feedback graphs by focusing in deriving a lower bound and gaining a good sense of how challenging the problem is. They then provide both an algorithm in a setting where they can achieve a tight bound as well as a general algorithm.\n\nThe proofs are well detailed, and the authors properly study and discuss the gap between upper and lower bounds as well as possible extensions.\n\nWEAKNESSES:\nWhile the gap in terms of upper and lower bounds may be tight in terms of the graph-dependent quantities in settings with self-avoidant contexts, the upper bounds contain some superfluous logarithmic dependencies. (In particular the $log^2 K$ term in Theorem 3.3). Do you think that this could be avoided?\n\nQUESTIONS:\nSee weaknesses.\n\nYou propose a specific algorithm for the setting with self-avoidant contexts, but do you think it is a realistic assumption to make ahead of time?\nDo you think that it would be possible to get some sort of best-of-both-worlds guarantees, where the same algorithm could achieve tight bounds if we are in the self-avoidant setting while still ensuring worst-case guarantees in the general case?\n\nLIMITATIONS:\nTheoretical work, NA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of stochastic contextual bandits with graph feedback, in which a graph over actions models the feedback structure. The learner selects an action after observing the current context, and then receives the losses of the actions that are neighbors of the selected one in the feedback graph.\nThis work proposes a novel graph-theoretic quantity $\\\\beta\\_{M}(G)$ to characterize the statistical complexity of learning in this problem setting.\nThe authors establish a minimax regret lower bound $\\\\Omega(\\\\sqrt{\\\\beta\\_{M}(G)T})$, where $\\\\beta\\_{M}(G)$ interpolates between the independence number $\\\\alpha(G)$ and the maximum acyclic subgraph (MAS) number $m(G)$.\nSpecifically, $\\\\beta\\_{M}(G) = \\\\max \\\\{\\\\sum\\_{c=1}^{M} |I\\_c| : I\\_1, \\\\dots, I\\_M \\\\text{ are independent sets in } G, I\\_i \\\\nrightarrow I\\_j \\\\text{ for } i < j\\\\}$.\nThis result implies that, while $\\\\alpha(G)$ dictates the complexity in multi-armed bandits (i.e., $M=1$), $m(G)$ becomes the relevant parameter as the number of contexts increases.\nThe paper further provides algorithms that achieve near-optimal regret bounds $\\\\tilde{O}(\\\\sqrt{\\\\beta\\_{M}(G)T})$ for self-avoiding context sequences and $\\\\tilde{O}(\\\\sqrt{\\\\min\\\\{m(G),\\\\bar\\\\beta\\_{M}(G)\\\\} T})$ for general context sequences (where $\\\\bar\\\\beta\\_{M}(G)$ is a larger but similarly defined graph parameter than $\\\\beta(G)$), leveraging carefully designed arm elimination techniques.\nThese algorithms are polynomial-time and demonstrate tight regret bounds for special families of context sequences and feedback graphs, namely undirected or transitively closed graphs.\n\nSTRENGTHS:\nThe most interesting contribution of this work is probably the connection between the learnability of the problem and the novel (at least to the best of my knowledge) graph-theoretic parameter $\\\\beta_{M}(G)$.\nThanks to this, the authors are able to provide further insights into the contextual bandit problem with feedback graphs (with complete cross-learning and under the assumption), showing that the number of contexts can influence the dependence of the regret on the structure of the feedback graph $G$, interpolating between the independence number $\\\\alpha(G)$ (when $M=1$) and $m(G)$ (e.g., when $M \\\\ge m(G)$.\nThe way the regret analysis shows the dependence on such a parameter, via the computation of the value of the sequential game described in Section 3, is also interesting and nontrivial.\n\nWEAKNESSES:\nWhat the authors consider in this work is not the entire family of strongly observable feedback graphs (the one known to correspond with minimax regret of order $\\\\sqrt{T}$ in the non-contextual case), but only a subset of those graphs, i.e., that contain all self-loops. This excludes relevant feedback graphs such as the loopless clique and the apple tasting one. I think a discussion about these missing graphs, e.g., in Section 4.1 would give a clearer picture of the contributions of this work and how they compare with previous relevant work.\n\nMore importantly, the only nearly tight bounds for the regret are provided for self-avoiding contexts.\nMoreover, the setting of complete cross-learning studied in this work seems quite restrictive as it imposes the assumption of observing the reward of the chosen action under any context.\nThis feels like a somewhat limiting assumption, as in real-world scenarios such a reward is often observed for the current context only, and the same context could reappear in non-contiguous time steps.\nThe applicability of the results is nevertheless sufficient, especially given some applications of interest and the extension of their results for the more general setting, albeit lacking a nearly optimal characterization for general context sequences and any strongly observable feedback graphs.\n\nA further limitation of the applicability of the results is the fact that the feedback graph is assumed to be fixed.\nThis might not be the case generally speaking, as feedback graphs could be time-varying (as assumed in most of the recent literature on bandits with feedback graphs).\nTime-varying feedback graphs could also be found in applications such as repeated first-price auctions (e.g., see “The Role of Transparency in Repeated First-Price Auctions with Unknown Valuations” by Cesa-Bianchi, Cesari, Colomboni, Fusco, and Leonardi, STOC 2024), which is one of the applications mentioned in the related work within this paper.\n\nQUESTIONS:\n- Please, address any relevant doubt that might have arisen from what is written above.\n- Can the results be extended to the other strongly observable feedback graphs (i.e., the ones not containing all self-loops)? What could be the technical limitations, if you see any?\n- Do you think it could be possible to adapt the algorithmic techniques in this work for time-varying graphs?\n\nMinor comments/typos:\n- Throughout the paper, use “domination number” instead of “dominating number”\n- Line 84: “In what follows” instead of “In the sequel”\n- Some references mention the arXiv, while they might already be published in some conference or journal\n- I think the introduction would benefit from a more thorough comparison with the literature on bandits with feedback graphs. For instance, detailed studies on the minimax regret for bandits with feedback graphs have been pursued in:\n  - Chen, Huang, Li, and Zhang. “Understanding bandits with graph feedback”, NeurIPS 2021\n  - Eldowa, Esposito, Cesari, and Cesa-Bianchi. “On the minimax regret for online learning with feedback graphs”, NeurIPS 2023\n  - Chen, He, and Zhang. “On interpolating experts and multi-armed bandits”, ICML 2024\n- Lines 157-159: the following paper also fits with that description:\n  - Zhang, Zhang, Luo, and Mineiro. “Efficient contextual bandits with uninformed feedback graphs”, ICML 2024\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}