Title: Practical Contextual Bandits with Feedback Graphs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online learning with feedback graphs

Contextual Bandits

Practical algorithms
Abstract: While contextual bandit has a mature theory, effectively leveraging different feedback patterns to enhance the pace of learning remains unclear. Bandits with feedback graphs, which interpolates between the full information and bandit regimes, provides a promising framework to mitigate the statistical complexity of learning. In this paper, we propose and analyze an approach to contextual bandits with feedback graphs based upon reduction to regression.  The resulting algorithms are computationally practical and achieve established minimax rates, thereby reducing the statistical complexity in real-world applications.

Practical Contextual Bandits with Feedback Graphs

Abstract

While contextual bandit has a mature theory, effectively leveraging different feedback patterns to enhance the pace of learning remains unclear. Bandits with feedback graphs, which interpolates between the full information and bandit regimes, provides a promising framework to mitigate the statistical complexity of learning. In this paper, we propose and analyze an approach to contextual bandits with feedback graphs based upon reduction to regression. The resulting algorithms are computationally practical and achieve established minimax rates, thereby reducing the statistical complexity in real-world applications.

1 Introduction

This paper is primarily concerned with increasing the pace of learning for contextual bandits [Auer et al., 2002, Langford and Zhang, 2007]. While contextual bandits have enjoyed broad applicability [Bouneffouf et al., 2020], the statistical complexity of learning with bandit feedback imposes a data lower bound for application scenarios [Agarwal et al., 2012]. This has inspired various mitigation strategies, including exploiting function class structure for improved experimental design [Zhu and Mineiro, 2022], and composing with memory for learning with fewer samples [Rucker et al., 2022]. In this paper we exploit alternative graph feedback patterns to accelerate learning: intuitively, there is no need to explore a potentially suboptimal action if a presumed better action, when exploited, yields the necessary information.

The framework of bandits with feedback graphs is mature and provides a solid theoretical foundation for incorporating additional feedback into an exploration strategy [Mannor and Shamir, 2011, Alon et al., 2015, 2017]. Succinctly, in this framework, the observation of the learner is decided by a directed feedback graph G : when an action is played, the learner observes the loss of every action to which the chosen action is connected. When the graph only contains self-loops, this problem reduces to the classic bandit case. For non-contextual bandits with feedback graphs, [Alon et al., 2015] provides a full characterization on the minimax regret bound with respect to different graph theoretic quantities associated with G according to the type of the feedback graph.

However, contextual bandits with feedback graphs have received less attention [Singh et al., 2020, Wang et al., 2021]. Specifically, there is no prior work offering a solution for general feedback graphs

and function classes. In this work, we take an important step in this direction by adopting recently developed minimax algorithm design principles in contextual bandits, which leverage realizability and reduction to regression to construct practical algorithms with strong statistical guarantees [Foster et al., 2018, Foster and Rakhlin, 2020, Foster et al., 2020, Foster and Krishnamurthy, 2021, Foster et al., 2021, Zhu and Mineiro, 2022]. Using this strategy, we construct a practical algorithm for contextual bandits with feedback graphs that achieves the optimal regret bound. Moreover, although our primary concern is accelerating learning when the available feedback is more informative than bandit feedback, our techniques also succeed when the available feedback is less informative than bandit feedback, e.g., in spam filtering where some actions generate no feedback. More specifically, our contributions are as follows.

Contributions. In this paper, we extend the minimax framework proposed in [Foster et al., 2021] to contextual bandits with general feedback graphs, aiming to promote the utilization of different feedback patterns in practical applications. Following [Foster and Rakhlin, 2020, Foster et al., 2021, Zhu and Mineiro, 2022], we assume that there is an online regression oracle for supervised learning on the loss. Based on this oracle, we propose SquareCB . G , the first algorithm for contextual bandits with feedback graphs that operates via reduction to regression (Algorithm 1). Eliding regression regret factors, our algorithm achieves the matching optimal regret bounds for deterministic feedback graphs, with ˜ O ( √ αT ) regret for strongly observable graphs and ˜ O ( d 1 3 T 2 3 ) regret for weakly observable graphs, where α and d are respectively the independence number and weakly domination number of the feedback graph (see Section 3.2 for definitions). Notably, SquareCB . G is computationally tractable, requiring the solution to a convex program (Theorem 3.6), which can be readily solved with off-the-shelf convex solvers (Appendix A.3). In addition, we provide closed-form solutions for specific cases of interest (Section 4), leading to a more efficient implementation of our algorithm. Empirical results further showcase the effectiveness of our approach (Section 5).

2 Problem Setting and Preliminary

Throughout this paper, we let [ n ] denote the set { 1 , 2 , . . . , n } for any positive integer n . We consider the following contextual bandits problem with informed feedback graphs. The learning process goes in T rounds. At each round t ∈ [ T ] , an environment selects a context x t ∈ X , a (stochastic) directed feedback graph G t ∈ [0 , 1] A×A , and a loss distribution P t : X → ∆([ -1 , 1] A ) ; where A is the action set with finite cardinality K . For convenience, we use A and [ K ] interchangeably for denoting the action set. Both G t and x t are revealed to the learner at the beginning of each round t . Then the learner selects one of the actions a t ∈ A , while at the same time, the environment samples a loss vector ℓ t ∈ [ -1 , 1] A from P t ( ·| x t ) . The learner then observes some information about ℓ t according to the feedback graph G t . Specifically, for each action j , she observes the loss of action j with probability G t ( a t , j ) , resulting in a realization A t , which is the set of actions whose loss is observed. With a slight abuse of notation, denote G t ( ·| a ) as the distribution of A t when action a is picked. We allow the context x t , the (stochastic) feedback graphs G t and the loss distribution P t ( ·| x t ) to be selected by an adaptive adversary. When convenient, we will consider G to be a K -byK matrix and utilize matrix notation.

Other Notations. Let ∆( K ) denote the set of all Radon probability measures over a set [ K ] . conv( S ) represents the convex hull of a set S . Denote I as the identity matrix with an appropriate dimension. For a K -dimensional vector v , diag( v ) denotes the K -byK matrix with the i -th diagonal entry v i and other entries 0 . Weuse R K ≥ 0 to denote the set of K -dimensional vectors with non-negative entries. For a positive definite matrix M ∈ R K × K , we define norm ∥ z ∥ M = √ z ⊤ Mz for any vector z ∈ R K . We use the ˜ O ( · ) notation to hide factors that are polylogarithmic in K and T .

Realizability. We assume that the learner has access to a known function class F ⊂ ( X × A ↦→ [ -1 , 1]) which characterizes the mean of the loss for a given context-action pair, and we make the following standard realizability assumption studied in the contextual bandit literature [Agarwal et al., 2012, Foster et al., 2018, Foster and Rakhlin, 2020, Simchi-Levi and Xu, 2021].

Assumption 1 (Realizability) . There exists a regression function f ⋆ ∈ F such that E [ ℓ t,a | x t ] = f ⋆ ( x t , a ) for any a ∈ A and across all t ∈ [ T ] .

Two comments are in order. First, we remark that, similar to [Foster et al., 2020], misspecification can be incorporated while maintaining computational efficiency, but we do not complicate the exposition here. Second, Assumption 1 induces a 'semi-adversarial' setting, wherein nature is completely free to determine the context and graph sequences; and has considerable latitude in determining the loss distribution subject to a mean constraint.

Regret. For each regression function f ∈ F , let π f ( x t ) := argmin a ∈A f ( x t , a ) denote the induced policy, which chooses the action with the least loss with respective to f . Define π ⋆ := π f ⋆ as the optimal policy. We measure the performance of the learner via regret to π ⋆ : Reg CB := ∑ T t =1 ℓ t,a t -∑ T t =1 ℓ t,π ⋆ ( x t ) , which is the difference between the loss suffered by the learner and the one if the learner applies policy π ⋆ .

Regression Oracle We assume access to an online regression oracle Alg Sq for function class F , which is an algorithm for online learning with squared loss. We consider the following protocol. At each round t ∈ [ T ] , the algorithm produces an estimator ̂ f t ∈ conv( F ) , then receives a set of context-action-loss tuples { ( x t , a, ℓ t,a ) } a ∈ A t where A t ⊆ A . The goal of the oracle is to accurately predict the loss as a function of the context and action, and we evaluate its performance via the square loss ∑ a ∈ A t ( ̂ f t ( x t , a ) -ℓ t,a ) 2 . We measure the oracle's cumulative performance via the following square-loss regret to the best function in F .

Assumption 2 (Bounded square-loss regret) . The regression oracle Alg Sq guarantees that for any (potentially adaptively chosen) sequence { ( x t , a, ℓ t,a ) } a ∈ A t ,t ∈ [ T ] in which A t ⊆ A ,

For finite F , Vovk's aggregation algorithm yields Reg Sq = O (log |F| ) [Vovk, 1995]. This regret is dependent upon the scale of the loss function, but this need not be linear with the size of A t , e.g., the loss scale can be bounded by 2 in classification problems. See Foster and Krishnamurthy [2021] for additional examples of online regression algorithms.

3 Algorithms and Regret Bounds

In this section, we provide our main algorithms and results.

3.1 Algorithms via Minimax Reduction Design

Our approach is to adapt the minimax formulation of [Foster et al., 2021] to contextual bandits with feedback graphs. In the standard contextual bandits setting (that is, G t = I for all t ), Foster et al. [2021] define the Decision-Estimation Coefficient (DEC) for a parameter γ > 0 as dec γ ( F ) := sup dec γ ( F ; ̂ f, x ) , where

Their proposed algorithm is as follows. At each round t , after receiving the context x t , the algorithm first computes ̂ f t by calling the regression oracle. Then, it solves the solution p t of the minimax problem defined in Eq. (1) with ̂ f and x replaced by ̂ f t and x t . Finally, the algorithm samples an action a t from the distribution p t and feeds the observation ( x t , a t , ℓ t,a t ) to the oracle. Foster et al. [2021] show that for any value γ , the algorithm above guarantees that

However, the minimax problem Eq. (1) may not be solved efficiently in many cases. Therefore, instead of obtaining the distribution p t which has the exact minimax value of Eq. (1), Foster et al.

Algorithm 1 SquareCB . G . Note Theorem 3.6 provides an efficient implementation of Eq. (4). Input: parameter γ ≥ 4 , a regression oracle Alg Sq

Receive context x t and directed feedback graph G t .

Obtain an estimator ̂ f t from the oracle Alg Sq .

Compute the distribution p t ∈ ∆( K ) such that p t = argmin p ∈ ∆( K ) dec γ ( p ; ̂ f t , x t , G t ) , where

and

Φ :=

Sample

a

t

X ×

from

[

K

p

t

]

R

.

and observe

{

ℓ

t,j

}

j

∈

A

t

where

A

t

∼

Feed the tuples { ( x t , j, ℓ t,j ) } j ∈ A t to the oracle Alg Sq .

end

[2021] show that any distribution that gives an upper bound C γ on dec γ ( p, F ; ̂ f, x ) also works and enjoys a regret bound with dec γ ( F ) replaced by C γ in Eq. (2).

To extend this framework to the setting with feedback graph G , we define dec γ ( F ; ̂ f, x, G ) as follows

Compared with Eq. (1), the difference is that we replace the squared estimation error on action a by the expected one on the observed set A ∼ G ( ·| a ) , which intuitively utilizes more feedbacks from the graph structure. When the feedback graph is the identity matrix, we recover Eq. (1). Based on dec γ ( F ; ̂ f, x, G ) , our algorithm SquareCB . G is shown in Algorithm 1. As what is done in [Foster et al., 2021], in order to derive an efficient algorithm, instead of solving the distribution p t with respect to the supremum over f ⋆ ∈ F , we solve p t that minimize dec γ ( p ; ̂ f, x t , G t ) (Eq. (4)), which takes supremum over f ⋆ ∈ ( X × [ K ] ↦→ R ) , leading to an upper bound on dec γ ( F ; ̂ f, x t , G t ) . Then, we receive the loss { ℓ t,j } j ∈ A t and feed the tuples { ( x t , j, ℓ t,j ) } j ∈ A t to the regression oracle Alg Sq . Following a similar analysis to [Foster et al., 2021], we show that to bound the regret Reg CB , we only need to bound dec γ ( p t ; ̂ f t , x t , G t ) .

The proof is deferred to Appendix A. In Section 3.3, we give an efficient implementation for solving Eq. (4) via reduction to convex programming.

3.2 Regret Bounds

In this section, we derive regret bounds for Algorithm 1 when G t 's are specialized to deterministic graphs, i.e., G t ∈ { 0 , 1 } A×A . We utilize discrete graph notation G = ([ K ] , E ) , where E ⊆ [ K ] × [ K ] ; and define N in ( G,j ) ≜ { i ∈ A : ( i, j ) ∈ E } as the set of nodes that can observe node j . In this case, at each round t , the observed node set A t is a deterministic set which contains any node i satisfying a t ∈ N in ( G t , i ) . In the following, we introduce several graph-theoretic concepts for deterministic feedback graphs [Alon et al., 2015].

G

t

(

·|

a

t

)

.

̸

Strongly/Weakly Observable Graphs. For a directed graph G = ([ K ] , E ) , a node i is observable if N in ( G,i ) = ∅ . An observable node is strongly observable if either i ∈ N in ( G,i ) or N in ( G,i ) = [ K ] \{ i } , and weakly observable otherwise. Similarly, a graph is observable if all its nodes are observable. An observable graph is strongly observable if all nodes are strongly observable, and weakly observable otherwise. Self-aware graphs are a special type of strongly observable graphs where i ∈ N in ( G,i ) for all i ∈ [ K ] .

Independent Set and Weakly Dominating Set. An independence set of a directed graph is a subset of nodes in which no two distinct nodes are connected. The size of the largest independence set of a graph is called its independence number. For a weakly observable graph G = ([ K ] , E ) , a weakly dominating set is a subset of nodes D ⊆ [ K ] such that for any node j in G without a self-loop, there exists i ∈ D such that directed edge ( i, j ) ∈ E . The size of the smallest weakly dominating set of a graph is called its weak domination number. Alon et al. [2015] show that in non-contextual bandits with a fixed feedback graph G , the minimax regret bound is ˜ Θ( √ αT ) when G is strongly observable and ˜ Θ( d 1 3 T 2 3 ) when G is weakly observable, where α and d are the independence number and the weak domination number of G , respectively.

3.2.1 Strongly Observable Graphs

In the following theorem, we show the regret bound of Algorithm 1 for strongly observable graphs.

Theorem 3.2 (Strongly observable graphs) . Suppose that the feedback graph G t is deterministic and strongly observable with independence number no more than α . Then Algorithm 1 guarantees that

In contrast to existing works that derive a closed-form solution of p t in order to show how large the DEC can be [Foster and Rakhlin, 2020, Foster and Krishnamurthy, 2021], in our case we prove the upper bound of dec γ ( p t ; ̂ f t , x t , G t ) by using the Sion's minimax theorem and the graph-theoretic lemma proven in [Alon et al., 2015]. The proof is deferred to Appendix A.1. Combining Theorem 3.2 and Theorem 3.1, we directly have the following corollary:

Corollary 3.3. Suppose that G t is deterministic, strongly observable, and has independence number no more than α for all t ∈ [ T ] . Algorithm 1 with choice γ = max { 4 , √ αT/ Reg Sq } guarantees that

For conciseness, we show in Corollary 3.3 that the regret guarantee for Algorithm 1 depends on the largest independence number of G over t ∈ [ T ] . However, we in fact are able to achieve a move t adaptive regret bound of order ˜ O (√ ∑ T t =1 α t Reg Sq ) where α t is the independence number of G t .

It is straightforward to achieve this by applying a standard doubling trick on the quantity ∑ T t =1 α t , assuming we can compute α t given G t , but we take one step further and show that it is in fact unnecessary to compute α t (which, after all, is NP-hard [Karp, 1972]): we provide an adaptive tuning strategy for γ by keeping track the the cumulative value of the quantity min p ∈ ∆( K ) dec γ ( p ; ̂ f t , x t , G t ) and show that this efficient method also achieves the adaptive ˜ O (√ ∑ t t =1 α t Reg Sq ) regret guarantee; see Appendix D for details.

3.2.2 Weakly Observable Graphs

For the weakly observable graph, we have the following theorem.

Theorem 3.4 (Weakly observable graphs) . Suppose that the feedback graph G t is deterministic and weakly observable with weak domination number no more than d . Then Algorithm 1 with γ ≥ 16 d guarantees that

where ˜ α is the independence number of the subgraph induced by nodes with self-loops in G t .

The proof is deferred to Appendix A.2. Similar to Theorem 3.2, we do not derive a closed-form solution to the strategy p t but prove this upper bound using the minimax theorem. Combining Theorem 3.4 and Theorem 3.1, we are able to obtain the following regret bound for weakly observable graphs, whose proof is deferred to Appendix A.2.

Corollary 3.5. Suppose that G t is deterministic, weakly observable, and has weak domination number no more than d for all t ∈ [ T ] . In addition, suppose that the independence number of the subgraph induced by nodes with self-loops in G t is no more than ˜ α for all t ∈ [ T ] . Then, Algorithm 1 with γ = max { 16 d, √ ˜ αT/ Reg Sq , d 1 3 T 2 3 Reg -2 3 Sq } guarantees that

Similarly to the strongly observable graph case, we also derive an adaptive tuning strategy for γ to achieve a more refined regret bound ˜ O (√ ∑ T t =1 ˜ α t Reg Sq + ( ∑ T t =1 √ d t ) 2 3 Reg 1 3 Sq ) where ˜ α t is the independence number of the subgraph induced by nodes with self-loops in G t and d t is the weakly domination number of G t . This is again achieved without explicitly computing ˜ α t and d t ; see Appendix D for details.

3.3 Implementation

In this section, we show that solving argmin p ∈ ∆( K ) dec γ ( p ; ̂ f, x, G ) in Algorithm 1 is equivalent to solving a convex program, which can be easily and efficiently implemented in practice.

Theorem 3.6. Solving argmin p ∈ ∆( K ) dec γ ( p ; ̂ f, x, G ) is equivalent to solving the following convex optimization problem.

where ̂ f in the objective is a shorthand for ̂ f ( x, · ) ∈ R K , e a is the a -th standard basis vector, and ≻ means element-wise greater.

The proof is deferred to Appendix A.4. Note that this implementation is not restricted to the deterministic feedback graphs but applies to the general stochastic feedback graph case. In Appendix A.3, we provide the 20 lines of Python code that solves Eq. (5).

4 Examples with Closed-Form Solutions

In this section, we present examples and corresponding closed-form solutions of p that make the value dec γ ( p ; ̂ f, x, G ) upper bounded by at most a constant factor of min p dec γ ( p ; ̂ f, x, G ) . This offers an alternative to solving the convex program defined in Theorem 3.6 for special (and practically relevant) cases, thereby enhancing the efficiency of our algorithm. All the proofs are deferred to Appendix B.

Cops-and-Robbers Graph. The 'cops-and-robbers' feedback graph G CR = 11 ⊤ -I , also known as the loopless clique, is the full feedback graph removing self-loops. Therefore, G CR is strongly observable with independence number α = 1 . Let a 1 be the node with the smallest value of ̂ f and a 2 be the node with the second smallest value of ̂ f . Our proposed closed-form distribution p is only supported on { a 1 , a 2 } and defined as follows:

In the following proposition, we show that with the construction of p in Eq. (6), dec γ ( p ; ̂ f, x, G CR ) is upper bounded by O ( 1 / γ ) , which matches the order of min p dec γ ( p ; ̂ f, x, G ) based on Theorem 3.2 since α = 1 .

Proposition 1. When G = G CR , given any ̂ f , context x , the closed-form distribution p in Eq. (6) guarantees that dec γ ( p ; ̂ f, x, G CR ) ≤ O ( 1 γ ) .

Apple Tasting Graph. The apple tasting feedback graph G AT = [ 1 1 0 0 ] consists of two nodes, where the first node reveals all and the second node reveals nothing. This scenario was originally proposed by Helmbold et al. [2000] and recently denoted the spam filtering graph [van der Hoeven et al., 2021]. The independence number of G AT is 1 . Let ̂ f 1 be the oracle prediction for the first node and let ̂ f 2 be the prediction for the second node. We present a closed-form solution p for Eq. (4) as follows:

We show that this distribution p satisfies that dec γ ( p ; ̂ f, x, G AT ) is upper bounded by O ( 1 / γ ) in the following proposition. We remark that directly applying results from [Foster et al., 2021] cannot lead to a valid upper bound since the second node does not have a self-loop.

Proposition 2. When G = G AT , given any ̂ f , context x , the closed-form distribution p in Eq. (7) guarantees that dec γ ( p ; ̂ f, x, G AT ) ≤ O ( 1 γ ) .

Inventory Graph. In this application, the algorithm needs to decide the inventory level in order to fulfill the realized demand arriving at each round. Specifically, there are K possible chosen inventory levels a 1 < a 2 < . . . < a K and the feedback graph G inv has entries G ( i, j ) = 1 for all 1 ≤ j ≤ i ≤ K and G ( i, j ) = 0 otherwise, meaning that picking the inventory level a i informs about all actions a j ≤ i . This is because items are consumed until either the demand or the inventory is exhausted. The independence number of G inv is 1 . Therefore, (very) large K is statistically tractable, but naively solving the convex program Eq. (5) requires superlinear in K computational cost. We show in the following proposition that there exists an analytic form of p , which guarantees that dec γ ( p ; ̂ f, x, G inv ) can be bounded by O ( 1 / γ ) .

Proposition 3. When G = G inv , given any ̂ f , context x , there exists a closed-form distribution p ∈ ∆( K ) guaranteeing that dec γ ( p ; ̂ f, x, G inv ) ≤ O ( 1 γ ) , where p is defined as follows: p j = max { 1 1+ γ ( ̂ f j -min i ̂ f i ) -∑ j ′ >j p j ′ , 0 } for all j ∈ [ K ] .

Undirected Self-Aware Graph. For the undirected and self-aware feedback graph G , which means that G is symmetric and has diagonal entries all 1 , we also show that a certain closed-form solution of p satisfies that dec γ ( p ; ̂ f, x, G ) is bounded by O ( α γ ) .

Proposition 4. When G is an undirected self-aware graph, given any ̂ f , context x , there exists a closed-form distribution p ∈ ∆( K ) guaranteeing that dec γ ( p ; ̂ f, x, G ) ≤ O ( α γ ) .

5 Experiments

In this section, we use empirical results to demonstrate the significant benefits of SquareCB . G in leveraging the graph feedback structure and its superior effectiveness compared to SquareCB . Following Foster and Krishnamurthy [2021], we use progressive validation (PV) loss as the evaluation metric, defined as L pv ( T ) = 1 T ∑ T t =1 ℓ t,a t . All the feedback graphs used in the experiments are deterministic. We run experiments on CPU Intel Xeon Gold 6240R 2.4G and the convex program solver is implemented via Vowpal Wabbit [Langford et al., 2007].

1.0

SquareCB.G

0.6

SquareCB

0.9

0.8

0.5

S

0.7

10

0.6

Los

0.4

2

N

0.5

0.3

0.4

0.3

BanditGraph

0.2

FullInformationGraph

0.2

CopsandRobbersGraph

102

103

104

103

104

Iterations

Iterations

5.1 SquareCB . G under Different Feedback Graphs

In this subsection, we show that our SquareCB . G benefits from considering the graph structure by evaluating the performance of SquareCB . G under three different feedback graphs. We conduct experiments on RCV1 dataset and leave the implementation details in Appendix C.1.

The performances of SquareCB . G under bandit graph, full information graph and cops-and-robbers graph are shown in the left part of Figure 1. We observe that SquareCB . G performs the best under full information graph and performs worst under bandit graph. Under the cops-and-robbers graph, much of the gap between bandit and full information is eliminated. This improvement demonstrates the benefit of utilizing graph feedback for accelerating learning.

5.2 Comparison between SquareCB . G and SquareCB

In this subsection, we compare the effectiveness of SquareCB . G with the SquareCB algorithm. To ensure a fair comparison, both algorithms update the regressor using the same feedbacks based on the graph. The only distinction lies in how they calculate the action probability distribution. We summarize the main results here and leave the implementation details in Appendix C.2.

5.2.1 Results on Random Directed Self-aware Graphs

We conduct experiments on RCV1 dataset using random directed self-aware feedback graphs. Specifically, at round t , the deterministic feedback graph G t is generated as follows. The diagonal elements of G t are all 1 , and each off-diagonal entry is drawn from a Bernoulli ( 3 / 4 ) distribution. The results are presented in the right part of Figure 1. Our SquareCB . G consistently outperforms SquareCB and demonstrates lower variance, particularly when the number of iterations was small. This is because when there are fewer samples available to train the regressor, it is more crucial to design an effective algorithm that can leverage the graph feedback information.

5.2.2 Results on Synthetic Inventory Dataset

In the inventory graph experiments, we create a synthetic inventory dataset and design a loss function for each inventory level a t ∈ [0 , 1] with demand d t ∈ [0 , 1] . Since the action set [0 , 1] is continuous, we discretize the action set in two different ways to apply the algorithms.

Fixed discretized action set. In this setting, we discretize the action set using fixed grid size ε ∈ { 1 100 , 1 300 , 1 500 } , which leads to a finite action set A of size 1 ε + 1 . Note that according to Theorem 3.2, our regret does not scale with the size of the action set (to within polylog factors), as the independence number is always 1. The results are shown in the left part of Figure 2.

Fixed Action Set

Adaptive ActionSet

SquareCB,k=101

SquareCB

SquareCB,k=301

0.14

SquareCB.G

0.14

SquareCB.G,k=101

0.13

SquareCB.G,k=301

0.12

SquareCB.G,k=501

0.12

Loss

Loss

0.11

PV

0.10

N

0.10

0.09

0.08

0.08

0.07

0.06

0.06

10²

103

10°

10²

103

10

Iterations

Iterations

We remark several observations from the results. First, our algorithm SquareCB . G outperforms SquareCB for all choices K ∈ { 101 , 301 , 501 } . This indicates that SquareCB . G utilizes a better exploration scheme and effectively leverages the structure of G inv . Second, we observe that SquareCB . G indeed does not scale with the size of the discretized action set A , since under different discretization scales, SquareCB . G has similar performances and the slight differences are from the improved approximation error with finer discretization. This matches the theoretical guarantee that we prove in Theorem 3.2. On the other hand, SquareCB does perform worse when the size of the action set increases, matching its theoretical guarantee which scales with the square root of the size of the action set.

Adaptively changing action set. In this setting, we adaptively discretize the action set [0 , 1] according to the index of the current round. Specifically, for SquareCB . G , we uniformly discretize the action set [0 , 1] with size ⌈ √ t ⌉ , whose total discretization error is O ( √ T ) due to the Lipschitzness of the loss function. For SquareCB , to optimally balance the dependency on the size of the action set and the discretization error, we uniformly discretize the action set [0 , 1] into ⌈ t 1 3 ⌉ actions. The results are illustrated in the right part of Figure 2. We can observe that SquareCB . G consistently outperforms SquareCB by a clear margin.

6 Related Work

Multi-armed bandits with feedback graphs have been extensively studied. An early example is the apple tasting problem of Helmbold et al. [2000]. The general formulation was introduced by Mannor and Shamir [2011]. Alon et al. [2015] characterized the minimax rates in terms of graphtheoretic quantities. Follow-on work includes relaxing the assumption that the graph is observed prior to decision [Cohen et al., 2016]; analyzing the distinction between the stochastic and adversarial settings [Alon et al., 2017]; considering stochastic feedback graphs [Li et al., 2020, Esposito et al., 2022]; instance-adaptivity [Ito et al., 2022]; data-dependent regret bound [Lykouris et al., 2018, Lee et al., 2020]; and high-probability regret under adaptive adversary [Neu, 2015, Luo et al., 2023].

The contextual bandit problem with feedback graphs has received relatively less attention. Wang et al. [2021] provide algorithms for adversarial linear bandits with uninformed graphs and stochastic contexts. However, this work assumes several unrealistic assumptions on both the policy class and the context space and is not comparable to our setting, since we consider the informed graph setting with adversarial context. Singh et al. [2020] study a stochastic linear bandits with informed feedback graphs and are able to improve over the instance-optimal regret bound for bandits derived in [Lattimore and Szepesvari, 2017] by utilizing the additional graph-based feedbacks.

Our work is also closely related to the recent progress in designing efficient algorithms for classic contextual bandits. Starting from [Langford and Zhang, 2007], numerous works have been done to the development of practically efficient algorithms, which are based on reduction to either cost-sensitive classification oracles [Dudik et al., 2011, Agarwal et al., 2014] or online regression oracles [Foster and Rakhlin, 2020, Foster et al., 2020, 2021, Zhu and Mineiro, 2022]. Following the latter trend, our work assumes access to an online regression oracle and extends the classic bandit problems to the bandits with general feedback graphs.

7 Discussion

In this paper, we consider the design of practical contextual bandits algorithm with provable guarantees. Specifically, we propose the first efficient algorithm that achieves near-optimal regret bound for contextual bandits with general directed feedback graphs with an online regression oracle.

While we study the informed graph feedback setting, where the entire feedback graph is exposed to the algorithm prior to each decision, many practical problems of interest are possibly uninformed graph feedback problems, where the graph is unknown at the decision time. It is unclear how to formulate an analogous minimax problem to Eq. (1) under the uninformed setting. One idea is to consume the additional feedback in the online regressor and adjust the prediction loss to reflect this additional structure, e.g., using the more general version of the E2D framework which incorporates arbitrary side observations [Foster et al., 2021]. Cohen et al. [2016] consider this uninformed setting in the non-contextual case and prove a sharp distinction between the adversarial and stochastic settings: even if the graphs are all strongly observable with bounded independence number, in the adversarial setting the minimax regret is Θ( T ) whereas in the stochastic setting the minimax regret is Θ( √ αT ) . Intriguingly, our setting is semi-adversarial due to realizability of the mean loss, and therefore it is apriori unclear whether the negative adversarial result applies.

In addition, bandits with graph feedback problems often present with associated policy constraints, e.g., for the apple tasting problem, it is natural to rate limit the informative action. Therefore, another interesting direction is to combine our algorithm with the recent progress in contextual bandits with knapsack [Slivkins and Foster, 2022], leading to more practical algorithms.

Acknowledgments HL and MZ are supported by NSF Awards IIS-1943607.

References

Alekh Agarwal, Miroslav Dudík, Satyen Kale, John Langford, and Robert Schapire. Contextual bandit learning with predictable rewards. In Artificial Intelligence and Statistics , pages 19-26. PMLR, 2012.

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646. PMLR, 2014.

Noga Alon, Nicolo Cesa-Bianchi, Claudio Gentile, and Yishay Mansour. From bandits to experts: A tale of domination and independence. Advances in Neural Information Processing Systems , 26, 2013.

Noga Alon, Nicolo Cesa-Bianchi, Ofer Dekel, and Tomer Koren. Online learning with feedback graphs: Beyond bandits. In Conference on Learning Theory , pages 23-35. PMLR, 2015.

Noga Alon, Nicolo Cesa-Bianchi, Claudio Gentile, Shie Mannor, Yishay Mansour, and Ohad Shamir. Nonstochastic multi-armed bandits with graph-structured feedback. SIAM Journal on Computing , 46(6):1785-1826, 2017.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Djallel Bouneffouf, Irina Rish, and Charu Aggarwal. Survey on applications of multi-armed and contextual bandits. In 2020 IEEE Congress on Evolutionary Computation (CEC) , pages 1-8. IEEE, 2020.

Alon Cohen, Tamir Hazan, and Tomer Koren. Online learning with feedback graphs without the graphs. In International Conference on Machine Learning , pages 811-819. PMLR, 2016.

Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , pages 169-178, 2011.

Emmanuel Esposito, Federico Fusco, Dirk van der Hoeven, and Nicolò Cesa-Bianchi. Learning on the edge: Online learning with stochastic feedback graphs. In Advances in Neural Information Processing Systems , 2022.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

Dylan Foster, Alekh Agarwal, Miroslav Dudik, Haipeng Luo, and Robert Schapire. Practical contextual bandits with regression oracles. In International Conference on Machine Learning , pages 1539-1548. PMLR, 2018.

Dylan J Foster and Akshay Krishnamurthy. Efficient first-order contextual bandits: Prediction, allocation, and triangular discrimination. Advances in Neural Information Processing Systems , 34: 18907-18919, 2021.

Dylan J Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33:11478-11489, 2020.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

David P Helmbold, Nicholas Littlestone, and Philip M Long. Apple tasting. Information and Computation , 161(2):85-139, 2000.

Shinji Ito, Taira Tsuchiya, and Junya Honda. Nearly optimal best-of-both-worlds algorithms for online learning with feedback graphs. In Advances in Neural Information Processing Systems , 2022.

Richard M Karp. Reducibility among combinatorial problems. page 85-103, 1972.

John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems , 20(1):96-1, 2007.

John Langford, Lihong Li, and Alex Strehl. Vowpal wabbit online learning project, 2007.

Tor Lattimore and Csaba Szepesvari. The end of optimism? an asymptotic analysis of finite-armed linear bandits. In Artificial Intelligence and Statistics , pages 728-737. PMLR, 2017.

Chung-Wei Lee, Haipeng Luo, and Mengxiao Zhang. A closer look at small-loss bounds for bandits with graph feedback. In Conference on Learning Theory , pages 2516-2564. PMLR, 2020.

David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research , 5(Apr):361-397, 2004.

Shuai Li, Wei Chen, Zheng Wen, and Kwong-Sak Leung. Stochastic online learning with probabilistic graph feedback. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4675-4682, 2020.

Haipeng Luo, Hanghang Tong, Mengxiao Zhang, and Yuheng Zhang. Improved high-probability regret for adversarial bandits with time-varying feedback graphs. In International Conference on Algorithmic Learning Theory , pages 1074-1100. PMLR, 2023.

Thodoris Lykouris, Karthik Sridharan, and Éva Tardos. Small-loss bounds for online learning with partial information. In Conference on Learning Theory , pages 979-986. PMLR, 2018.

Shie Mannor and Ohad Shamir. From bandits to experts: On the value of side-observations. Advances in Neural Information Processing Systems , 24, 2011.

Gergely Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. Advances in Neural Information Processing Systems , 28, 2015.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Mark Rucker, Joran T Ash, John Langford, Paul Mineiro, and Ida Momennejad. Eigen memory tree. arXiv preprint arXiv:2210.14077 , 2022.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research , 2021.

Rahul Singh, Fang Liu, Xin Liu, and Ness Shroff. Contextual bandits with side-observations. arXiv preprint arXiv:2006.03951 , 2020.

Aleksandrs Slivkins and Dylan Foster. Efficient contextual bandits with knapsacks via regression. arXiv preprint arXiv:2211.07484 , 2022.

Dirk van der Hoeven, Federico Fusco, and Nicolò Cesa-Bianchi. Beyond bandit feedback in online multiclass classification. Advances in Neural Information Processing Systems , 34:13280-13291, 2021.

Vladimir G Vovk. A game of prediction with expert advice. In Proceedings of the eighth annual conference on Computational learning theory , pages 51-60, 1995.

Lingda Wang, Bingcong Li, Huozhi Zhou, Georgios B Giannakis, Lav R Varshney, and Zhizhen Zhao. Adversarial linear contextual bandits with graph-structured side observations. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 10156-10164, 2021.

Yinglun Zhu and Paul Mineiro. Contextual bandits with smooth regret: Efficient learning in continuous action spaces. In International Conference on Machine Learning , pages 27574-27590. PMLR, 2022.

A Omitted Details in Section 3

Theorem 3.1. Suppose dec γ ( p t ; ̂ f t , x t , G t ) ≤ Cγ -β for all t ∈ [ T ] and some β > 0 , Algorithm 1 with γ = max { 4 , ( CT ) 1 β +1 Reg -1 β +1 Sq } guarantees that E [ Reg CB ] ≤ O ( C 1 β +1 T 1 β +1 Reg β β +1 Sq ) .

Proof. Following [Foster et al., 2020], we decompose Reg CB as follows: E [ Reg CB ]

Next, since E [ ℓ t,a | x t ] = f ⋆ ( x t , a ) for all t ∈ [ T ] and a ∈ A , we know that

where the final inequality is due to Assumption 2.

Therefore, we have

Picking γ = max { 4 , ( CT Reg Sq ) 1 β +1 } , we obtain that

A.1 Proof of Theorem 3.2

Before proving Theorem 3.2, we first show the following key lemma, which is useful in proving that dec γ ( p ; ̂ f, x, G ) is convex for both strongly and weakly observable feedback graphs G . We

highlight that the convexity of dec γ ( p ; ̂ f, x, G ) is crucial for both proving the upper bound of min p ∈ ∆( K ) dec γ ( p ; ̂ f, x, G ) and showing the efficiency of Algorithm 1.

LemmaA.1. Suppose u, v, x ∈ R d with ⟨ v, x ⟩ > 0 . Then both g ( x ) = ⟨ u,x ⟩ 2 ⟨ v,x ⟩ and h ( x ) = (1 -⟨ u,x ⟩ ) 2 ⟨ v,x ⟩ are convex in x .

Proof. The function f ( x, y ) = x 2 /y is convex for y > 0 due to

By composition with affine functions, both g ( x ) = f ( ⟨ u, x ⟩ , ⟨ v, x ⟩ ) and h ( x ) = f (1 -⟨ u, x ⟩ , ⟨ v, x ⟩ ) are convex.

Theorem 3.2 (Strongly observable graphs) . Suppose that the feedback graph G t is deterministic and strongly observable with independence number no more than α . Then Algorithm 1 guarantees that

Proof. For conciseness, we omit the subscript t . Direct calculation shows that for all a ⋆ ∈ [ K ] ,

where W a = ∑ a ′ ∈ N in ( G,a ) p a ′ . Therefore, taking the gradient over f ∗ ( x, · ) and we know that

Then, denote ̂ f ∈ R K to be ̂ f ( x, · ) and consider the following minimax form:

̸

where the last equality is due to Sion's minimax theorem and the fact that Eq. (10) is convex in p ∈ ∆( K ) by applying Lemma A.1 with u = e a and v = g a for each a ∈ [ K ] , where g a ∈ { 0 , 1 } K is defined as g a,i = { ( i, a ) ∈ E } , G = ([ K ] , E ) , ∀ i ∈ [ K ] .

✶ Choose p a = (1 -1 γ ) q a + 1 γK for all a ∈ [ K ] . Let S be the set of nodes in [ K ] that have a self-loop. Then we can upper bound the value above as follows:

Next we bound 2 q a (1 -q a ) W a for each a ∈ [ K ] . If a ∈ [ K ] \ S , we have W a = 1 -p a and

If a ∈ S , we know that

where the last inequality is due to Lemma 5 in Alon et al. [2015]. We include this lemma (Lemma E.1) for completeness. Combining all the above inequalities, we obtain that

A.2 Proof of Theorem 3.4

Theorem 3.4 (Weakly observable graphs) . Suppose that the feedback graph G t is deterministic and weakly observable with weak domination number no more than d . Then Algorithm 1 with γ ≥ 16 d guarantees that

where ˜ α is the independence number of the subgraph induced by nodes with self-loops in G t .

Proof. Similar to the strongly observable graphs setting, for weakly observable graphs, we know that

Choose p a = (1 -1 γ -ηd ) q a + 1 γK + η ✶ { a ∈ D } where D with | D | = d is the minimum weak dominating set of G and 0 < η ≤ 1 4 d is some parameter to be chosen later. Substituting the form of p to Eq. (16) and using the fact that | ̂ f a | ≤ 1 for all a ∈ [ K ] , we can obtain that

Then we can upper bound the value above as follows:

Now consider a ∈ D . If a ∈ S , then we know that W a ≥ η ; Otherwise, we know that this node can be observed by at least one node in D , meaning that W a ≥ η . Combining the two cases above, we know that

where the last inequality is because η ≤ 1 4 d and γ ≥ 16 d . Consider a / ∈ D . Let S 0 be the set of nodes which either have a self loop or can be observed by all the other node. Recall that S represents the set of nodes with a self-loop. Then similar to the derivation of Eq. (13), we know that for a ∈ S 0 ,

For a / ∈ S 0 , we know that W a ≥ η . Therefore,

Plugging Eq. (18), Eq. (19), and Eq. (20) to Eq. (17), we obtain that

Consider the last term. If a ∈ S 0 \ S , similar to Eq. (14), we know that

where the last inequality is due to γ ≥ 16 d and η ≤ 1 4 d . If a ∈ S , similar to Eq. (15), we know that

where the last inequality is again due to Lemma 5 in [Alon et al., 2015] and ˜ α is the independence number of the subgraph induced by nodes with self-loops in G . Plugging Eq. (22) to Eq. (21) gives

Next, we prove Corollary 3.5 by combining Theorem 3.4 and Theorem 3.1.

Corollary 3.5. Suppose that G t is deterministic, weakly observable, and has weak domination number no more than d for all t ∈ [ T ] . In addition, suppose that the independence number of the subgraph induced by nodes with self-loops in G t is no more than ˜ α for all t ∈ [ T ] . Then, Algorithm 1 with γ = max { 16 d, √ αT/ Reg Sq , d 1 3 T 2 3 Reg -2 3 Sq } guarantees that

Proof. Combining Eq. (8), Eq. (9) and Theorem 3.4, we can bound Reg CB as follows:

A.3 Python Solution to Eq. (5)

This particular formulation multiplies both sides of the constraint in Eq. (5) by √ γ while scaling the objective by √ γ . While mathematically equivalent to Eq. (5), empirically it has superior numerical stability for large γ . For additional stability, when using this routine we recommend subtracting off the minimum value from ̂ f , which is equivalent to making the substitutions √ γ ̂ f ← √ γ ̂ f - √ γ min a ̂ f a and z ← z + √ γ min a ̂ f a and then exploiting the 1 ⊤ p = 1 constraint.

A.4 Proof of Theorem 3.6

Theorem 3.6. Solving argmin p ∈ ∆( K ) dec γ ( p ; ̂ f, x, G ) is equivalent to solving the following convex optimization problem.

where ̂ f in the objective is a shorthand for ̂ f ( x, · ) ∈ R K , e a is the a -th standard basis vector, and ≻ means element-wise greater.

Proof. Denote f ⋆ = f ⋆ ( x, · ) ∈ R K . Note that according to the definition of G , we know that ( G ⊤ p ) i denotes the probability that action i 's loss is revealed when the selected action a is sampled from distribution p . Then, we know that

where the third equality is by picking f ⋆ to be the maximizer and introduces a constraint. Therefore, the minimization problem min p ∈ ∆( K ) dec γ ( p ; ̂ f, x, G ) can be written as the following constrained optimization by variable substitution:

The convexity of the constraints follows from Lemma A.1.

B Omitted Details in Section 4

In this section, we provide proofs for Section 4. We define W a := ∑ a ′ ∈ N in ( G,a ) p a ′ to be the probability that the loss of action a is revealed when selecting an action from distribution p . Let ̂ f = ̂ f ( x, · ) ∈ R K and f = f ( x, · ) ∈ R K . Direct calculation shows that for any a ⋆ ∈ [ K ] ,

Therefore, substituting f ⋆ into Eq. (4), we obtain that

Without loss of generality, we assume the min i ∈ [ K ] ̂ f i = 0 . This is because shifting ̂ f by min i ∈ [ K ] ̂ f i does not change the value of 〈 p -e a ⋆ , ̂ f 〉 . In the following sections, we provide proofs showing that a certain closed-form of p leads to optimal dec γ ( p ; ̂ f, x, G ) up to constant factors for several specific types of feedback graphs, respectively.

B.1 Cops-and-Robbers Graph

Proposition 1. When G = G CR , given any ̂ f , context x , the closed-form distribution p in Eq. (6) guarantees that dec γ ( p ; ̂ f, x, G CR ) ≤ O ( 1 γ ) .

Proof. We use the following notation for convenience: p 1 := p a 1 , p 2 := p a 2 , ̂ f 1 := ̂ f a 1 = 0 , ̂ f 2 := ̂ f a 2 . For the cops-and-robbers graph and closed-form solution p in Eq. (6), Eq. (23) becomes:

̸

If a ⋆ = a 1 and a ⋆ = a 2 , we know that

̸

If a ⋆ = a 2 , we can obtain that

If a ⋆ = a 1 , we have

Putting everything together, we prove that dec γ ( p ; ̂ f, x, G CR ) ≤ 6 γ ≤ O ( 1 γ ) .

B.2 Apple Tasting Graph

Proposition 2. When G = G AT , given any ̂ f , context x , the closed-form distribution p in Eq. (7) guarantees that dec γ ( p ; ̂ f, x, G AT ) ≤ O ( 1 γ ) .

Proof. For the apple tasting graph and closed-form solution p in Eq. (7), Eq. (23) becomes:

Suppose ̂ f 1 = 0 , we know that p 1 = 1 , p 2 = 0 and

If a ⋆ = 1 , we have

If a ⋆ = 2 , direct calculation shows that

Suppose ̂ f 2 = 0 , we know that p 1 = 2 4+ γ ̂ f 1 , p 2 = 1 -p 1 and

1.

If a ⋆ = 2 , direct calculation shows that

B.3 Inventory Graph

Proposition 3. When G = G inv , given any ̂ f , context x , there exists a closed-form distribution p ∈ ∆( K ) guaranteeing that dec γ ( p ; ̂ f, x, G inv ) ≤ O ( 1 γ ) , where p is defined as follows: p j = max { 1 1+ γ ( ̂ f j -min i ̂ f i ) -∑ j ′ >j p j ′ , 0 } for all j ∈ [ K ] .

Proof. Based on the distribution defined above, define A ⊆ [ K ] to be the set such that for all i ∈ A , p i > 0 and denote N = | A | . We index each action in A by k 1 < k 2 < · · · < k N = K . According to the definition of p i , we know that p i is strictly positive only when ̂ f i < ̂ f j for all j > i and specifically, when p i > 0 , we know that W i = ∑ j ≥ i p j = 1 1+ γ ̂ f i (recall that min i ̂ f i = 0 since we shift ̂ f ). Therefore, define W k N +1 = 0 and we know that

According to Lemma 9 of [Alon et al., 2013] (included as Lemma E.2 for completeness), we know that

where G A is the subgraph of G restricted to node set A and mas ( G ) is the size of the maximum acyclic subgraphs of G . It is direct to see that any subgraph G of G inv has mas ( G ) = 1 .

̸

Next, consider the value of a ⋆ ∈ [ K ] that maximizes 1 -2 p a ⋆ γW a ⋆ -̂ f a ⋆ . If a ⋆ ≤ k 1 , then we know that W a ⋆ = 1 and 1 -2 p a ⋆ γW a ⋆ -̂ f a ⋆ ≤ 1 γ . Otherwise, suppose that k i < a ⋆ ≤ k i +1 for some i ∈ [ N -1] . According to the definition of p , if a ⋆ = k i +1 we know that p a ⋆ = 0 and

Otherwise, W a ⋆ = W k i +1 and 1 -2 p a ⋆ γW a ⋆ -̂ f a ⋆ ≤ 1 γW k i +1 -̂ f k i +1 = 1 γ . Combining the two cases above and Eq. (24), we obtain that

Therefore,

B.4 Undirected and Self-Aware Graphs

Proposition 4. When G is an undirected self-aware graph, given any ̂ f , context x , there exists a closed-form distribution p ∈ ∆( K ) guaranteeing that dec γ ( p ; ̂ f, x, G ) ≤ O ( α γ ) .

̸

Proof. We first introduce the closed-form of p and then show that dec γ ( p ; ̂ f, x, G ) ≤ O ( α γ ) . Specifically, we first sort ̂ f a in an increasing order and choose a maximal independent set by choosing the nodes in a greedy way. Specifically, we pick k 1 = argmin i ∈ [ K ] ̂ f i . Then, we ignore all the nodes that are connected to k 1 and select the node a with the smallest ̂ f a in the remaining node set. This forms a maximal independent set I ⊆ [ K ] , which has size no more than α and is also a dominating set. Set p a = 1 α + γ ̂ f a for a ∈ I \{ k 1 } and p k 1 = 1 -∑ a = k 1 ,a ∈ I p a . This is a valid distribution as we only choose at most α nodes and p a ≤ 1 /α for all a ∈ I \{ k 1 } . Now we show that dec γ ( p ; ̂ f, x, G ) ≤ O ( α γ ) . Specifically, we only need to show that with this choice of p , for any a ⋆ ∈ [ K ] ,

Plugging in the form of p , we know that

If a ⋆ = k 1 , then we can obtain that 1 -2 p a ⋆ γW a ⋆ ≤ 1 γW k 1 ≤ α γ as p k 1 ≥ 1 α according to the definition of p . Otherwise, note that according to the choice of the maximal independent set I , W a ⋆ ≥ 1 α + γ ̂ f a ′ for some a ′ ∈ I such that ̂ f a ′ ≤ ̂ f a ⋆ . Therefore,

Combining the two inequalities above together proves the bound.

C Implementation Details in Experiments

C.1 Implementation Details in Section 5.1

We conduct experiments on RCV1 [Lewis et al., 2004], which is a multilabel text-categorization dataset. We use a subset of RCV1 containing 50000 samples and K = 50 sub-classes. Therefore, the feedback graph in our experiment has K = 50 nodes. We use the bag-of-words vector of each sample as the context with dimension d = 47236 and treat the text categories as the arms. In each round t , the learner receives the bag-of-words vector x t and makes a prediction a t ∈ [ K ] as the text category. The loss is set to be ℓ t,a t = 0 if the sample belongs to the predicted category a t and ℓ t,a t = 1 otherwise.

The function class we consider is the following linear function class:

where Sigmoid ( u ) = 1 1+ e -u for any u ∈ R . The oracle is implemented by applying online gradient descent with learning rate η searched over { 0 . 1 , 0 . 2 , 0 . 5 , 1 , 2 , 4 } . As suggested by [Foster and

Krishnamurthy, 2021], we use a time-varying exploration parameter γ t = c · √ αt , where t is the index of the iteration, c is searched over { 8 , 16 , 32 , 64 , 128 } , and α is the independence number of the corresponding feedback graph. Our code is built on PyTorch framework [Paszke et al., 2019]. We run 5 independent experiments with different random seeds and plot the mean and standard deviation value of PV loss.

C.2 Implementation Details in Section 5.2

C.2.1 Details for Results on Random Directed Self-aware Graphs

We conduct experiments on a subset of RCV1 containing 10000 samples with K = 10 sub-classes. Our code is built on Vowpal Wabbit [Langford and Zhang, 2007]. For SqaureCB, the exploration parameter γ t at round t is set to be γ t = c · √ Kt , where t is the index of the round and c is the hyperparameter searched over set { 8 , 16 , 32 , 64 , 128 } . The remaining details are the same as described in Appendix C.1.

C.2.2 Details for Results on Synthetic Inventory Dataset

In this subsection, we introduce more details in the synthetic inventory data construction, loss function constructions, oracle implementation, and computation of the strategy at each round.

Dataset. In this experiment, we create a synthetic inventory dataset constructed as follows. The dataset includes T data points, the t -th of which is represented as ( x t , d t ) where x t ∈ R m is the context and d t is the realized demand given context x t . Specifically, in the experiment, we choose m = 100 and x t 's are drawn i.i.d from Gaussian distribution with mean 0 and standard deviation 0 . 1 . The demand d t is defined as

where θ ∈ R m is an arbitrary vector and ε t is a one-dimensional Gaussian random variable with mean 0 . 3 and standard deviation 0 . 1 . After all the data points { ( x t , d t ) } T t =1 are constructed, we normalize d t to [0 , 1] by setting d t ← d t -min t ′ ∈ [ T ] d t ′ max t ′ ∈ [ T ] d t ′ -min t ′ ∈ [ T ] d t ′ . In all our experiments, we set T = 10000 .

Loss construction. Next, we define the loss at round t when picking the inventory level a t with demand d t , which is defined as follows:

where h > 0 is the holding cost per remaining items and b > 0 is the backorder cost per remaining items. In the experiment, we set h = 0 . 25 and b = 1 .

Regression oracle. The function class we use in this experiment is as follows:

This ensures the realizability assumption according to the definition of our loss function shown in Eq. (25). The oracle uses online gradient descent with learning rate η searched over { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 } .

Calculation of p t . To make SquareCB . G more efficient, instead of solving the convex program defined in Eq. (5), we use the closed-form of p t derived in Proposition 3, which only requires O ( K ) computational cost and has the same theoretical guarantee (up to a constant factor) as the one enjoyed by the solution solved by Eq. (5). Similar to the case in Appendix C.1, at each round t , we pick γ t = c · √ t with c searched over the set { 0 . 25 , 0 . 5 , 1 , 2 , 3 , 4 } . Note again that the independence number for inventory graph is 1 .

We run 8 independent experiments with different random seeds and plot the mean and standard deviation value of PV loss.

D Adaptive Tuning of γ without the Knowledge of Graph-Theoretic Numbers

In this section, we show how to adaptively tune the parameter γ in order to achieve ˜ O (√ ∑ T t =1 α t Reg Sq ) regret in the strongly observable graphs case and ˜ O ( ( ∑ T t =1 √ d t ) 2 3 Reg 1 3 Sq + √ ∑ T t =1 ˜ α t Reg Sq ) in the weakly observable graphs case.

D.1 Strongly Observable Graphs

In order to achieve ˜ O ( ∑ T t =1 α t Reg Sq ) regret guarantee without the knowledge of α t , we apply a doubling trick on γ based on the value of min p ∈ ∆( K ) dec γ ( p ; ̂ f t , x t , G t ) . Specifically, our algorithm goes in epochs with the parameter γ being γ s in the s -th epoch. We initialize γ 1 = √ T Reg Sq . As proven in Theorem 3.2, we know that

Therefore, within each epoch s (with start round b s ), at round t , we calculate the value

which is bounded by ˜ O ( 1 γ s ∑ t τ = b s α τ ) and is in fact obtainable by solving the convex program. Then, we check whether Q t ≤ γ s Reg Sq . If this holds, we continue our algorithm using γ s ; otherwise, we set γ s +1 = 2 γ s and restart the algorithm.

Now we analyze the performance of the above described algorithm. First, note that for any t , within epoch s ,

meaning that the number of epoch S is bounded by S = log 2 C 1 +log 4 ∑ T t =1 α t T for certain C 1 > 0 which only contains constant and log terms.

Next, consider the regret in epoch s with I s = [ b s , e s ] . According to Eq. (8), we know that the regret within epoch s is bounded as follows:

where the last inequality is because at round t = e s -1 , Q t ≤ γ s Reg Sq is satisfied. Taking summation over all S epochs, we know that the overall regret is bounded as

which finishes the proof.

D.2 Weakly Observable Graphs

For the weakly observable graphs case, to achieve the target regret without the knowledge of ˜ α t and d t , which are the independence number of the subgraph induced by nodes with self-loops in G t and the weak domination number of G t , we apply the same approach as the one applied in the strongly observable graph case. Note that according to Theorem 3.4, within epoch s , we for certain

C

2

>

1

only containing constants and log

factors. In the weakly observable graphs case, we know that the number of epoch is bounded by

and at round t in epoch S ,

meaning that epoch S will never end. Therefore, following Eq. (27) and Eq. (28), we can obtain that

which finishes the proof.

E Auxiliary Lemmas

Lemma E.1 (Lemma 5 in [Alon et al., 2015]) . Let G = ( V, E ) be a directed graph with | V | = K , in which i ∈ N in ( G,i ) for all vertices i ∈ [ K ] . Assign each i ∈ V with a positive weight w i such that ∑ n i =1 w i ≤ 1 and w i ≥ ε for all i ∈ V for some constant 0 < ε < 1 2 . Then

where α ( G ) is the independence number of G .

Lemma E.2 (Lemma 9 in [Alon et al., 2013]) . Let G = ( V, E ) be a directed graph with vertex set | V | = K , in which i ∈ N in ( G,i ) for all i ∈ [ K ] . Let p be an arbitrary distribution over [ K ] . Then, we have

where mas ( G ) is the size of the maximum acyclic subgraphs of G .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online learning in a contextual setting when a feedback graph determines the feedback received by the learner. Namely, the learner observes all the losses experienced by the actions in the graph-neighborhood of the action it played. \n\nFeedback graphs are a well-known feedback model for online learning, and their study in the contextual setting is natural. The paper's main result is a general framework (Theorem 3.1) that reduces the problem to the solution at each time step of a convex program. Then the authors instantiate this general theorem for the strongly observable and weakly observable case, obtaining the same regret rates that characterize the non-contextual version of the problem. \n\nThe primary technical tool of the paper is the parameter defined in equation (3). Following Foster et al. 2021, the authors use this new formula instead of the bandit one (equation 1) to prove their results.\n\nSTRENGTHS:\n- The problem studied is natural and well-motivated.\n- The regret bounds match the lower bounds in Alon et al., 2015, so they are tight. \n- The experimental results and the theoretical guarantees support the claim that a richer feedback structure improves the regret bounds w.r.t. the bandit case. \n- It is nice that the authors devoted some time to presenting the detailed results for some famous classes of feedback graphs.\n\nWEAKNESSES:\n- Once equation 3 is designed, the rest of the paper seems incremental to Foster et al.  \n- Equation 3 needs knowledge of the feedback graph. Therefore the paper works only in the informed setting. \n- The introductory model entails time-varying feedback graphs, while the results for strong and weakly observable feedback graphs need the graph to be deterministic and known up-front. \n\nMinor comment:\n- uniform the \\citet and \\citep notation\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the adversarial contextual bandit problem with feedback graphs, in the finite function class setting with a realizability assumption, with an access to an online regression oracle. They extend previous approaches for the vanilla contextual MAB setting in order to obtain regret bounds of $\\tilde{O}(\\sqrt{\\alpha T})$ with fully observable graphs and $\\tilde{O}(d^{\\frac13} T^{\\frac23})$ for weakly observable graphs, where $\\alpha$ and $d$ denote the graph's independence number and weak domination number respectively. The authors demonstrate their results empirically and show that their algorithm performs better than existing approaches when side observations are available.\n\nSTRENGTHS:\n* The authors exhibit the first efficient algorithm for contextual bandits with feedback graphs, which obtains near-optimal regret guarantees both in the strongly observable and weakly observable settings.\n* The techniques utilized by the authors seem to neatly generalize the approach of Foster et al. ('21) to the feedback graph setting, by generalizing the inverse gap technique to solving a convex problem together with utilizing a combinatorial property of the feedback graphs.\n* The authors demonstrate in several empirical experiments that their algorithm out-performs SquareCB when feedback graphs are present, thus strengthening the idea that better performance can be obtained when side observations are available.\n\nWEAKNESSES:\n* This is not a very major issue, but the algorithm suggested by the authors requires knowledge of the feedback graphs' independence number (or a good bound on it), which is a hard quantity to compute in general. This is only a minor point because many previous works in the feedback graphs literature also require knowing such a parameter, but I will remark that some results can be obtained without this knowledge.\n\nQUESTIONS:\nGiven the result presented in the paper, a very natural question that arises is whether or not for stochastic contexts and losses, the approach of [2] can be generalized to the feedback setting in order to obtain similar regret bounds, but with access to an **offline** regression oracle. I think such a result would be also very interesting, and I would expect that similar approaches to those that the authors used in this paper would work for extending [2], as their approach also extends SquareCB of [1]. I'd appreciate it if the authors could comment on whether or not they considered the stochastic setting as well, and if they think that they're approach could be used in order to obtain results with an offline oracle.\n\n[1] Foster, Dylan, and Alexander Rakhlin. \"Beyond ucb: Optimal and efficient contextual bandits with regression oracles.\" _International Conference on Machine Learning_. PMLR, 2020.\n\n[2] Simchi-Levi, David, and Yunzong Xu. \"Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability.\" _Mathematics of Operations Research_ 47.3 (2022): 1904-1931.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is concerned with the problem of contextual bandits with graph feedback. The authors consider a setting where the contexts and the graphs are generated in an arbitrary manner and revealed to the learner at the beginning of each round. For a given context, the mean loss of each arm is assumed to be fixed. Furthermore, it is assumed that the learner is provided with a class of functions mapping context-action pairs to their mean loss, and that this class contains the true function. Following prior works in contextual bandits, the authors use an online regression oracle to estimate the true mean losses. These estimates are then used within the estimation-to-decision framework of  Foster et al. (2021).  Within this framework, the proposed algorithm requires solving a convex program at each round, for which closed form solutions are provided for special cases of interest. The authors prove regret bounds that depend on the structure of the feedback graphs and the regret of the online regression oracle. Additionally, empirical evaluations are carried out to showcase the algorithm's ability to take advantage of the side observations provided via the feedback graphs.\n\nSTRENGTHS:\nAlthough the adopted approach relies on existing techniques, the application of the estimation-to-decision framework of Foster et al. (2021) for bandits with graph feedback is still an interesting contribution. Most notably, in Theorems 3.2 and 3.4, the authors bound the decision estimation coefficient for their algorithm in terms of the independence number for strongly observable graphs and the weak domination number for weakly observable graphs, which respectively are the graph theoretic quantities that characterize the minimax regret for these settings. Overall, the paper is well written, the setting is adequately motivated, and the clarity of the presentation is decent.\n\nWEAKNESSES:\n- The regret bounds in Corollaries 3.3 and 3.5 are stated in terms of a uniform upper bound on the independence number or the weak domination number of the observed graphs. This is unsatisfactory since a single sparse graph could render the bound vacuous.\n- In the formulated setting, the graphs are allowed to be stochastic, where each edge is realized with a certain probability which is revealed to the learner at the beginning of the round. However, all the provided theoretical results are for deterministic graphs.\n- One point in need of clarification concerning the experiments is that it is mentioned in the beginning of Section 5 that all the graphs used in the experiments are deterministic, while the experiment described in Section 5.2.1 seems to involve stochastic graphs.\n- For the experiment of Section 5.1, it might be more informative to include more intermediate cases between bandits and full information.\n- A minor correction: In the discussion section, the authors address the limitation of their approach in the uniformed setting, where the feedback graph is only revealed to the learner after making a decision. The authors then cite results for the non-contextual case from (Cohen et al, 2016). However, in that work, the graph is never revealed to the learner, only the actions in the neighbourhood of the played action and their losses are observed.\n\nQUESTIONS:\n- Is it possible to obtain bounds that scale with the average independence number (or weak domination number) of the graphs perhaps via an adaptive choice of the value of gamma at each round?\n- What kind of regret bounds can we obtain for stochastic graphs using this approach?\n- Does the proposed approach offer an advantage over existing algorithms for learning with graph feedback in the non-contextual case?\n\nLIMITATIONS:\nThe authors did address some of the limitations of their work. Notably the fact that their approach requires the knowledge of the feedback graph before choosing the action.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the contextual bandits problem in the presence of a feedback graph G_t. An edge (i -> j) in G_t means that taking action a_i allows us to observe the loss for action a_j. The work extends the SquareCB algorithm to this setting, the primary difference being the way the action sampling probability p_t is learned (Eq. (1) vs Eq (4)). The authors also present regret bounds for strongly observable and weakly observable graphs where the bounds improve over the standard contextual bandits setting when the independence number and cardinality of the dominating set is small.\n\nSTRENGTHS:\n- Overall I found the paper to be well-written where the setup was explained well and the notation was easy to follow.\n - This paper builds on the SquareCB algorithm. I appreciated the fact that the work makes an effort to try to separate their own contributions from the SquareCB paper.\n- The paper covers both strongly and weakly observable graphs. In general, the statements in Theorems 3.2 and 3.4 are fairly intuitive.\n\nWEAKNESSES:\n- By and large, the paper relies heavily on the SquareCB paper's analysis. Although the setting of feedback graphs is new, the analysis seems derivative.\n- I would suggest that the authors provide more intuition behind the proofs of the key theorems. Even though the theorem statements are easy to understand, it would be nice to get some insight into the proofs, and specifically what are steps different from the standard CB regret analysis.\n\nQUESTIONS:\n- Are there any intermediate characterizations of graphs between strongly and weakly observable? In the strongly observable case, for nodes without self-loops, you require an edge from all other nodes. What if there are edges from most nodes (but not all)? Or what if most nodes are strongly observable but not all? Broadly, I am trying to understand how conservative the bounds are when the graphs are weakly observable (but perhaps not too weak). Is the algorithm expected to the complexity of the graph somehow?\n - Follow-up to the above, the regret bounds take the worst-case graph over all the rounds. You allow the graph to change over t. Does the regret of your proposed algorithm also improve when the graphs are favorable for a large number of time steps?\n\n(CB is not my area of expertise so I understand that both of the above assumptions are probably standard for doing regret analysis, but I think experimentally it would be nice to get some insight into how the actual algorithm performs relative to the theoretical regret).\n\nLIMITATIONS:\nSee weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

comments: SUMMARY:
This work studied contextual bandits with feedback graphs. The authors provided an algorithm based on the recent Decision-Estimation Coefficient (DEC) framework that finds the next-step action distribution by solving a minimax optimization problem. To address the issue that the minimax problem is hard to solve, the authors also showed that there exists an efficient implementation of the solver, and the closed-form solution also exists for some special cases. For the experiment, the authors compared the proposed algorithm with a vanilla baseline algorithm that does not utilize the feedback graph information.

STRENGTHS:
The presentation is clear. The overall approach to solving the contextual bandits with feedback graph is convincing. The theoretical results are sound.

WEAKNESSES:
The importance of this work remains unclear. The proposed algorithm is very similar to the original E2D algorithm proposed by Foster et al. 2021 with an additional expectation over the feedback graph. The analysis technique is also very similar. The key difficulty in proving the regret of SquareCBG is to build an effective estimate the DEC constant. However, from the proof of Theorem 3.2, it seems that the proof is pretty standard by following the decomposition technique developed in Foster et al. 2021 (like their Proposition 5.1), while utilizing the graph node estimation lemma proposed by Alon et al. 2015. Therefore, I would recommend the authors highlight the main challenge for them to derive the theoretical results. 


A typo. Line 62, for some action $j$-> $a_t$.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
The authors addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-28