Title: Towards Efficient and Optimal Covariance-Adaptive Algorithms for Combinatorial Semi-Bandits
Venue: NeurIPS 2024
Accepted: yes
Keywords: Combinatorial Semi-Bandit

Upper Confidence Bound

Thompson Sampling

Confidence Ellipsoid
Abstract: We address the problem of stochastic combinatorial semi-bandits, where a player selects among $P$ actions from the power set of a set containing $d$ base items. Adaptivity to the problem's structure is essential in order to obtain optimal regret upper bounds. As estimating the coefficients of a covariance matrix can be manageable in practice, leveraging them should improve the regret. We design ``optimistic''  covariance-adaptive algorithms relying on online estimations of the covariance structure, called OLS-UCB-C and COS-V (only the variances for the latter). They both yields improved gap-free regret. Although COS-V can be slightly suboptimal, it improves on computational complexity by taking inspiration from Thompson Sampling approaches. It is the first sampling-based algorithm satisfying a $\sqrt{T}$ gap-free regret (up to poly-logs). We also show that in some cases, our approach efficiently leverages the semi-bandit feedback and outperforms bandit feedback approaches, not only in exponential regimes where $P\gg d$ but also when $P\leq d$, which is not covered by existing analyses.

Towards Efficient and Optimal Covariance-Adaptive Algorithms for Combinatorial Semi-Bandits

Abstract

We address the problem of stochastic combinatorial semi-bandits, where a player selects among P actions from the power set of a set containing d base items. Adaptivity to the problem's structure is essential in order to obtain optimal regret upper bounds. As estimating the coefficients of a covariance matrix can be manageable in practice, leveraging them should improve the regret. We design 'optimistic' covariance-adaptive algorithms relying on online estimations of the covariance structure, called OLS-UCB-C and COS-V (only the variances for the latter). They both yield improved gap-free regret. Although COS-V can be slightly suboptimal, it improves on computational complexity by taking inspiration from Thompson Sampling approaches. It is the first sampling-based algorithm satisfying a O ( √ T ) gap-free regret (up to poly-logs). We also show that in some cases, our approach efficiently leverages the semi-bandit feedback and outperforms bandit feedback approaches, not only in exponential regimes where P ≫ d but also when P ≤ d , which is not covered by existing analyses.

1 Introduction

In sequential decision-making, the bandit framework has been extensively studied and was instrumental to several applications, e.g. A/B testing (Guo et al., 2020), online advertising and recommendation services (Zeng et al., 2016), network routing (Tabei et al., 2023), demand-side management (Brégère et al., 2019), etc. Its popularity stems from its relative simplicity, allowing it to model and analyze a wide range of challenging real-world settings. Reference books like Bubeck and Cesa-Bianchi (2012) or Lattimore and Szepesvári (2020) offer a wide perspective on the subject.

In this framework, a decision-maker or player must make choices and receives associated rewards, but it lacks prior knowledge of its environment. This naturally leads to an exploration-exploitation trade-off: the player must explore different actions to determine the best one, but an inefficient exploration strategy may harm the cumulative rewards. Efficient algorithms rely on exploiting the

environment's structure, such as estimating the parameters of a reward function instead of exploring every action.

This paper focuses on the stochastic combinatorial semi-bandit framework. At each round, the player chooses a subset of base items and receives a feedback for each item chosen. The action set is included in the base items' power set, and can therefore be exponentially big and difficult to explore. The main challenge in this framework is to effectively combine the information collected through different actions (that may share common base items).

Problem formulation. We consider a set of d ∈ N ∗ base items , each item i ∈ [ d ] = { 1 , . . . , d } yielding stochastic rewards. A player accesses these rewards through a set A ⊆ { 0 , 1 } d of P ∈ N ∗ actions , each corresponding to a subset of at most m ≥ 5 items (a) . We refer to actions a ∈ A using their components vector a = ( a i ) i ∈ [ d ] ∈ { 0 , 1 } d where for all j ∈ [ d ] , a j = 1 if and only if action a contains base item j .

The player interacts with an environment over a sequence of T ∈ N ∗ rounds . At each round t ∈ [ T ] , the player chooses an action A t ∈ A , the environment samples a reward vector Y t ∈ R d , the player observes the realization for every item contained in A t , and receives their sum. The interactions between the player and the environment are summarized in Framework 1.

Framework 1 Stochastic Combinatorial Semi-Bandit

For each t ∈ { 1 , . . . , T } :

The player chooses an action A t ∈ A .

The environment samples a vector of rewards Y t ∈ R d from a fixed unknown distribution.

The player receives the reward ⟨ A t , Y t ⟩ = ∑ i A t,i Y t,i .

The player observes Y t,i for all i ∈ [ d ] s.t. A t,i = 1 .

Assumptions. We make the following assumptions. For all t ∈ [ T ] , Y t is independent of the past rewards and the player's decision σ ( A 1 , Y 1 , . . . , A t -1 , Y t -1 , A t ) . There exists a mean reward vector E [ Y t ] = µ ∈ R d and a second order moment matrix S = E [ Y t Y ⊤ t ] ∈ M d ( R ) . The positive semidefinite covariance matrix is denoted Σ ∈ M d ( R ) , with Σ = S -µµ ⊤ .There exists a known vector B ∈ R d + such that for all t ∈ [ T ] and i ∈ [ d ] , | Y t,i | ≤ B i / 2 almost surely (and | Y t,i -µ i | ≤ B i ) .

The objective of the decision-maker is to minimize the expected cumulative pseudo-regret defined as:

where ⟨· , ·⟩ denotes the usual inner product in R d , a ∗ ∈ arg max a ∈A ⟨ a, µ ⟩ is an optimal action, and ∆ a = ⟨ a ∗ -a, µ ⟩ is the sub-optimality gap for action a ∈ A .

1.1 Existing work and limitations

Combinatorial semi-bandit problems have been extensively studied by the bandit community since their introduction by Chen et al. (2013). Here, we only highlight key earlier works related to this paper. For a comprehensive introduction to this literature, we refer the interested reader to the monograph by Lattimore and Szepesvári (2020).

A first line of works considers deterministic algorithms based on the optimistic principle and upper confidence bounds (UCBs). Chen et al. (2013) first designed CUCB , computing UCBs for the items' average rewards, converting these into UCBs for the actions' rewards, and choosing the action with the highest one. It was later analyzed by Kveton et al. (2015), who proved a regret upper bound uniform over all possible covariance matrices Σ (hence, paying the worst-case). Combes et al. (2015) highlighted the importance of designing Σ -adaptive algorithms by showing that the regret could be improved by a factor of m when the items' average rewards are independent. Subsequently, Degenne and Perchet (2016) developed OLS-UCB , an algorithm intended to leverage the covariance structure. However, OLS-UCB requires prior knowledge of a positive semi-definite covariance-proxy matrix Γ , such that for all t ≥ 1 and for all u ∈ R d , E [exp( ⟨ u, Y t -µ ⟩ )] ≤ exp( 1 2 ∥ u ∥ 2 Γ ) . Estimating Γ in

practice is challenging and leads to regret bounds depending on it instead of the 'true' covariance matrix Σ , potentially resulting in significantly looser bounds. This issue was addressed by Perrault et al. (2020b), who proposed a covariance-adaptive algorithm, ESCB-C , with asymptotically optimal gap-dependent regret upper bounds. Yet, it suffers from an additive constant of order ∆ -2 min , which prevents its conversion into an ˜ O ( √ T ) (b) gap-free bound. Thus, none of the above works proposes a ˜ O ( √ T ) gap-free and covariance-adaptive regret bound, which is one of the key contributions of this paper. A common drawback of these works is also their potentially prohibitive computational complexity, due to the need to solve a maximization step over a large action set A ⊆ { 0 , 1 } d that can be exponentially large. Some works, such as Cuvelier et al. (2021) or Liu et al. (2022), propose solutions to achieve polynomial time complexity, for example by applying UCB at the item level only rather than the action level. However, these approaches only work for independent rewards or under specific assumptions on their distribution, making the analysis for generic and unknown distributions extremely challenging. Another approach to tackle the computational burden in combinatorial semi-bandits is to resort to sampling algorithms, which we detail below.

A second line of works for stochastic combinatorial semi-bandits considers randomized algorithms inspired by Thompson Sampling ( TS ) for multi-armed bandits (Thompson, 1933). These algorithms involve sampling a random vector ˜ µ t ∈ R d at each round t + 1 ∈ [ T -1] from a distribution representing a 'belief' over the parameter µ , taking a decision A t +1 ∈ arg max a ∈A ⟨ a, ˜ µ t ⟩ , and updating the belief distribution using the observations. The main appeal of these approaches lies in their computational complexity, especially when solving a linear maximization problem in particular action spaces (such as matroids). Recent works have designed and analyzed such algorithms. Notably, Wang and Chen (2018) consider independent item's rewards. Perrault et al. (2020a) refine it and assume a known variance-proxy Γ and therefore suffers from the same drawbacks as Degenne and Perchet (2016). Their technical analysis also yields a gap-dependent regret bound with an undesirable ∆ -m min term, preventing a ˜ O ( √ T ) gap-free rate. A central contribution in our paper is the combination of the computational efficiency for sampling algorithms with the covariance-adaptivity ˜ O ( √ T ) gap-free from our UCB approach.

Besides, the literature concerning our setting has historically mostly focused on cases where the action set is exponentially large, namely P ≫ d , and the way to get quasi-optimal regret rates in these instances. However, outside of these regimes, the commonly derived regret bounds are too rough and fail to show the benefit of the semi-bandit feedback. While the conventional stochastic combinatorial semi-bandit regret upper bound grows as ˜ O ( √ mdT ) (Kveton et al., 2015), a ˜ O ( √ mPT ) could be achieved using bandit feedback only (Auer et al., 2002b). Intriguingly, the latter appears to outperform the semi-bandit rate as soon as P < d , making the extra information obtained through a richer feedback seemingly useless. Fine-grained analyses, clearly taking the structure into account, are therefore needed.

1.2 Contributions

A new deterministic optimism-based algorithm (Section 2). We present OLS-UCB-C (Online Least Squares Upper Confidence Bound with Covariance estimation), relying on the optimism principle. The analysis of OLS-UCB-C sketched in Section 5.2 shows the following properties:

First optimal gap-free regret upper bound . OLS-UCB-C yields a similar gap-dependent regret bound as ESCB-C from Perrault et al. (2020b) up to logarithmic factors, and the first optimal covariance-adaptative gap-free ˜ O ( √ T ) regret bound (Theorem 1).

Improved performance over UCB in all regimes of P/d . Under some conditions on the covariance matrix Σ , we prove that OLS-UCB has a uniformly better regret than UCB , showing that properly leveraging semi-bandit feedback indeed consistently offers an advantage on (simple) bandit algorithms, which is not straightforward from existing analyses.

Improved complexity over concurrent algorithms . OLS-UCB-C circumvents the convex optimization problem that ESCB-C requires to solve at each round and is therefore more efficient, despite suffering in the very large P regime as many other deterministic algorithms.

The first stochastic optimism-based algorithm (Section 3). We introduce COS-V (Combinatorial Optimistic Sampling with Variance estimation), a TS -inspired algorithm exploiting the 'frequentist' confidence regions derived in Section 4. It satisfies the following:

Improved complexity for P ≫ 1 compared to other deterministic semi-bandit algorithms . COS-V can be efficient in the very large P regime, which is the main blind spot of OLS-UCB-C . √

First gap-free ˜ O ( T ) regret upper bound for a sampling algorithm . The analysis we provide in Section 5.3 exploits the common structure of the OLS-UCB-C and COS-V algorithms. It enables the derivation of a gap-dependent bound for COS-V that does not involve the ∆ -m term we typically find in the analysis for other TS algorithms (Wang and Chen, 2018; Perrault et al., 2020a), consequently leading to a new ˜ O ( √ T ) variance-adaptive gap-free regret upper bound for a sampling algorithm.

A novel gap-free lower bound (Section 2.2). We show a gap-free lower bound on the regret for stochastic combinatorial semi-bandits, explicitly involving the structure of the problem (the items forming each action) and the covariance matrix Σ . This lower bound highlights the optimality of the gap-free upper bound we establish for OLS-UCB-C .

Technical details are deferred to Section 4, Section 5, and the Appendix.

˜

step needed in ESCB-C .

2 Covariance-adaptative deterministic algorithm: OLS-UCB-C

In this section, we design a new algorithm that efficiently leverages the semi-bandit feedback. It approximates the coefficients of the covariance matrix Σ online. The approximation is symmetric by construction and yields a coefficient-wise upper bound of Σ , but it is not necessarily positive semi-definite, a constraint that can be challenging to impose in practice.

2.1 Algorithm: OLS-UCB-C

We present OLS-UCB-C described in Alg. 2 and detail below the successive steps it performs.

Initial exploration. The algorithm first explores by choosing every base item i ∈ [ d ] and every 'reachable' couple ( i, j ) ∈ [ d ] 2 at least once.

Rewards means estimation. At each round t + 1 ∈ [ T -1] , the algorithm uses an empirical mean ˆ µ t for µ defined as

where d a = diag( a ) ∈ M d ( R ) is the diagonal matrix of the elements in a ∈ A ; n t, ( i,j ) is the number of times items i and j (with possibly i = j ) have been chosen together; N t = diag(( n t, ( i,i ) ) i ∈ [ d ] ) ∈ M d ( R ) is the diagonal matrix of item counts.

Rewards covariances estimation. The covariances are estimated by

The algorithm uses ˆ Σ t , a coefficient-wise upper-confidence bound of Σ whose coefficients are defined for a fixed δ > 0 as

Optimistic action choice. Following the 'optimistic' principle of UCB -like algorithms, the estimated rewards ( ⟨ ˆ µ t , a ⟩ ) a ∈A are inflated by bonuses, yielding corresponding upper confidence bounds. The bonuses involve the history through a regularized empirical design matrix (with empirical covariances):

where d ˆ Σ t = diag( ˆ Σ t ) ∈ M d ( R ) , I is the identity matrix and ˆ Σ t is the coefficient-wise upper bound for the covariance matrix defined in (4). Formally, OLS-UCB-C chooses

Efficiency improvement. While Perrault et al. (2020b) use an axis-realignment technique to derive their confidence regions, our approach builds ellipsoidal confidence regions. This simplifies the computation of an upper confidence bound for each action as we have a closed-form expression. In comparison, Perrault et al. (2020b) need to solve linear programs in convex sets at each iteration.

2.2 Regret upper bounds

Theorem 1. Let T ∈ N ∗ and δ > 0 .

Then, OLS-UCB-C (Alg. 2) satisfies the gap-dependent regret upper bound

where σ 2 a,i = ∑ j ∈ a max { Σ i,j , 0 } , and the gap-free regret upper bound

The proof is outlined in Section 5 and the specific details are presented in Appendix E.

Algorithm 2

̸

Optimal gap-free bound. This result shows that OLS-UCB-C yields the same gap-dependent regret upper bound as ESCB-C (Perrault et al., 2020b) (up to poly-logarithmic factors) and more importantly yields a novel covariance-adaptive and optimal ˜ O ( √ T ) gap-free bound, as shown by the following lower-bound proven in Appendix A. Unfortunately, only the positive coefficients of Σ are considered in our bound but the inclusion of negative correlations could be advantageous to reduce the rate at which the regret increases. However, it could complicate the analysis greatly and is thus deferred to future research.

Theorem 2. Let d, m ∈ N ∗ such that d/m ≥ 2 is an integer, T ∈ N ∗ , and Σ ⪰ 0 a covariance matrix. Then, there exists a stochastic combinatorial semi-bandit with d base items, and a reward distribution with covariance matrix Σ on which for any policy π , the pseudo regret satisfies

Improvement over CUCB . Our gap-free and gap-dependent bounds outperform those of CUCB (Kveton et al., 2015) no matter the covariance structure, as ( Tmd ) 1 / 2 ∥ B ∥ ∞ ≳ (Tr( Σ ) T ) 1 / 2 . (c) Besides, in the particular case of a diagonal Σ , our gap-free upper bound gains a factor at least √ m over the one of CUCB . In this scenario, σ 2 a,i = Σ i,i for all a ∈ A and i ∈ a . Our gap-dependent and gap-free upper bounds are then roughly bounded as

respectively.

Improvement over UCBV . Assuming that Σ i,j ≥ 0 for all i, j , our upper-bound uniformly improves the one of UCBV of order ( T ∑ a a ⊤ Σ a ) 1 / 2 , since in this case ∑ d i =1 max a ∈A\ i ∈A σ 2 a,i ≤ ∑ a ∥ a ∥ Σ . Existing semi-bandit analyses could only leverage semi-bandit feedback in the regime P ≫ d , which is natural in combinatorial bandits but not systematic in real-world applications.

3 New sampling algorithm for combinatorial semi-bandits: COS-V

In this section, we introduce a randomized algorithm inspired from TS , enabling to get potentially computational complexity at the cost of not leveraging off-diagonal covariances.

The difficulty in designing and analysing TS algorithms generally stems from controlling the random exploration. To that end, we parametrize the exploration distribution using the same estimators as OLS-UCB-C .

3.1 Algorithm: COS-V

Wepropose a sampling strategy using 'frequentist' estimators, COS-V , described in Algorithm 3.

The algorithm begins with the same exploration phase as OLS-UCB-C . Thereafter at each round t +1 ∈ [ T -1] , we sample parameters (˜ µ i,t ) i ∈ [ d ] using 1 -dimensional normal distributions biased

toward the positive orthant. Formally, for all i ∈ [ d ] , we sample

where g t,δ = ( 1 + 2 log ( 2 dt log( t ) 2 /δ ) ) 1 / 2 and f t,δ is the same as for OLS-UCB-C .

̸

3.2 Regret upper bound

Theorem 3. Let T ∈ N ∗ , and δ > 0 .

Then, COS-V (Alg. 3) satisfies the gap-dependent regret upper bound

where ∆ i, min = min { ∆ a , a ∈ A such that i ∈ a } , and the gap-free regret upper bound

The proof is outlined in Section 5 and the specific details can be found in Appendix F.

Novel variance-dependent bound. Theorem 3 presents the first variance-dependent bound for a sampling-based semi-bandit algorithm. Unfortunately, integrating the covariances Σ i,j in the leading term is still an open problem. Possible leads include exploring other biasing strategies for sampling, or using oversampling approaches like Abeille and Lazaric (2017) which inflate the confidence regions in the linear bandits setting.

Novel gap-free regret bound. An important novelty of our gap-dependent bound Eq. (16) is the absence of ∆ -m min terms present in the previous analyses of CTS (Wang and Chen, 2018; Perrault et al., 2020a). In particular, this improvement yields the first ˜ O ( √ T ) gap-free regret upper bound for a sampling strategy.

4 Mean and covariance estimation

In this section, we present concentration results for ˆ µ t (rewards means) and ˆ Σ t (rewards covariances, estimated with ˆ χ t ) used in OLS-UCB-C and COS-V , which are central to prove Theorem 1 and Theorem 3 (sketched in Section 5).

4.1 Covariance-aware confidence region for the average reward

Average reward estimation. Let a ∈ A , t ≥ d ( d +1) / 2 , as introduced in Section 2.1, the least square estimator for the mean reward vector µ using all the data gathered after round t is

where the η s denote the deviations Y s -µ .

Confidence region design. We design confidence regions inspired from LinUCB literature (Rusmevichientong and Tsitsiklis, 2010; Filippi et al., 2010; Abbasi-Yadkori et al., 2011) and the work of Degenne and Perchet (2016). Major differences with those works include using Bernstein's style concentration inequalities involving the covariance matrix Σ , assuming a multidimensional noise term, and combining them with a covering argument to relax dependence in d (peeling trick from Degenne and Perchet, 2016). We introduce the regularized design matrix defined by

where V t = ∑ t s =1 d A s Σd A s is the design matrix (of which the OLS-UCB-C and COS-V use an empirical version). Let S t = N t (ˆ µ t -µ ) , the deviations of ⟨ a, ˆ µ t ⟩ are bounded as

Designing a confidence region for ∥ S t ∥ Z -1 t therefore allows to control the deviations |⟨ a, ˆ µ t -µ ⟩| uniformly on A . Let δ > 0 , we define the event

with f t,δ = 6log(1 /δ ) + 6[log( t ) + ( d +2)log(log( t ))] + 3 d [2 log(2) + log(1 + e )] .

This event can also be written G t = { ∥ ˆ µ t -µ ∥ N t Z -1 t N t ≤ f t,δ } and is therefore equivalent to ˆ µ t belonging to an ellipsoid around the true reward mean vector µ .

Confidence region probability. The following result proven in Appendix B presents an upper bound for P ( G c t ) .

Proposition 1. Let t ≥ d ( d +1) / 2 and δ > 0 . Then, P ( G c t ) ≤ δ/ ( t log( t ) 2 ) .

Proving this result relies on an argument adapted from Faury et al. (2020) and a covering trick from Degenne and Perchet (2016).

4.2 Confidence interval for covariances estimator

Rewards covariances estimator. Let t ≥ d ( d +1) / 2 and a 'reachable' couple ( i, j ) ∈ [ d ] 2 . The coefficients of Σ can be estimated online by ˆ χ t as introduced in Section 2.1

Rewards covariances upper confidence bound. Let δ > 0 . We use the following coefficient-wise upper estimates of Σ in our algorithms

Favorable event design. We define C t as the event where all the coefficients of ˆ Σ t are indeed upper bounding those of Σ :

Favorable event probability. The following result proven in Appendix C presents an upper bound for P ( C c t ) .

Proposition 2. Let t ≥ d ( d +1) / 2 and δ > 0 . Then, P ( C c t ) ≤ δ/ ( t log( t ) 2 ) .

5 Regret upper bounds

In this section, we provide a sketch of the proof for Theorem 1 and Theorem 3. For both OLS-UCB-C and COS-V , the idea to bound the regret is to find a sequence of favorable events ( E t ) t ≥ d ( d +1) / 2 that are true with high probability, and under which the regret grows logarithmically with time.

5.1 Template bound

Let ( E t ) t ∈ [ T ] be a sequence of events, then for both OLS-UCB-C and COS-V standard derivations yield

Assuming that the sequence of events ( E t ) t ≥ d ( d +1) / 2 happens with high enough probability, it is sufficient to control what happens conditionally to it. In particular, Proposition 6 in Appendix D states that if we can bound ∆ 2 A t +1 with a linear combination of terms evolving as n -k t, ( i,j ) for every couple ( i, j ) ∈ A t +1 and different k ≥ 1 , then we can infer a worst-case behaviour, which yields Theorem 1 and Theorem 3.

In the following, we will refer to the term ∑ T -1 t = d ( d +1) / 2 P ( E c t ) as the unfavorable event probability and to the term E [ ∑ T t = d ( d +1) / 2 1 {E t } ∆ A t +1 ] as the high-probability regret .

5.2 Regret of OLS-UCB-C

For OLS-UCB-C we consider the sequence of events E t = {G t ∩ C t } for all t ≥ d ( d + 1) / 2 , corresponding the confidence regions of (ˆ µ t ) t ≥ d ( d +1) / 2 and of ( ˆ Σ t, ( i,j ) ) t ≥ d ( d +1) / 2 , ( i,j ) ∈ [ d ] 2 defined in Section 4. Under these events, we can upper-bound the high-probability regret from Eq. (13) with the following proposition (proven in Appendix E.1).

Proposition 3. Let δ > 0 . Then, OLS-UCB-C yields

as T →∞ , where σ 2 a,i = ∑ j ∈ a ( Σ i,j ) +

Conclusion of the proof. Injecting results from Proposition 3 (high-probability regret) as well as Proposition 1 and Proposition 2 (unfavorable event probability) into the template bound (13), we get

for OLS-UCB-C as T → ∞ . This provides the gap-dependent bound of Theorem 1. The gap-free bound is detailed in Appendix E.4. It is enabled by the fact that our gap-dependent bound does not incur any term in ∆ -2 min , unlike Perrault et al. (2020b); Degenne and Perchet (2016).

5.3 Regret of COS-V

For the analysis of our stochastic algorithm COS-V , we need to consider events related to the sampling distributions in addition to the events G ′ t and C t introduced in the precedent section. For this purpose, we denote the event H t defined as

The high-level idea of the event H t is to ensure that the sampled rewards ˜ µ t,i upper-bound the true mean µ i while not being too far for all the items i ∈ a ∗ . Showing that the event H t indeed occurs with high-probability (Lemma 7 in Appendix F) and setting the events E t = {G t ∩ C t ∩ H t } , we can upper-bound the high-probability regret in the following proposition (proof is in Appendix F.2).

Proposition 4. Let δ > 0 . Then COS-V yields

Conclusion of the proof. Injecting results from Proposition 4 (high-probability regret) as well as Lemma 7, Proposition 1 and Proposition 2 (unfavorable event probability) into the template bound (13) yields

as T →∞ . This provides the gap-dependent bound of Theorem 3. As it does not incur any term in ∆ -m min as in Wang and Chen (2018); Perrault et al. (2020a), this result can be used to derive a ˜ O ( √ T ) gap-free bound for a sampling-based combinatorial semi-bandit algorithm.

6 Concluding remarks

Wepropose and analyze two algorithms for combinatorial semi-bandits. OLS-UCB-C is a deterministic, covariance-adaptive algorithm. Compared to other existing approaches, our algorithm is typically less computationally demanding and yields the first ˜ O ( √ T ) gap-free regret rate that explicitly depends on the covariance of the base item rewards and the structure. COS-V is a variance-adaptive, TS -like algorithm. Its complexity is significantly lower under certain types of constraints, but its regret is suboptimal as it assumes worst-case correlations. However, leveraging the analysis of OLS-UCB-C , it also yields the first ˜ O ( √ T ) gap-free regret upper bound among sampling-based approaches.

References

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems .

Abeille, M. and Lazaric, A. (2017). Linear thompson sampling revisited. In Artificial Intelligence and Statistics , pages 176-184. PMLR.

Audibert, J.-Y., Munos, R., and Szepesvári, C. (2009). Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science , 410(19):1876-1902.

Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002a). Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47:235-256.

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. (2002b). The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77.

Brégère, M., Gaillard, P., Goude, Y., and Stoltz, G. (2019). Target tracking for contextual bandits: Application to demand side management. In International Conference on Machine Learning .

Bubeck, S. and Cesa-Bianchi, N. (2012). Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning , 5(1):1-122.

Chen, W., Wang, Y., and Yuan, Y. (2013). Combinatorial multi-armed bandit: General framework and applications. In International Conference on Machine Learning .

Combes, R., Talebi Mazraeh Shahi, M. S., Proutiere, A., et al. (2015). Combinatorial bandits revisited. Advances in Neural Information Processing Systems .

Cuvelier, T., Combes, R., and Gourdin, E. (2021). Statistically efficient, polynomial-time algorithms for combinatorial semi-bandits. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 5(1):1-31.

Degenne, R. and Perchet, V. (2016). Combinatorial semi-bandit with known covariance. Advances in Neural Information Processing Systems .

Faury, L., Abeille, M., Calauzènes, C., and Fercoq, O. (2020). Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning , pages 3052-3060. PMLR.

Filippi, S., Cappe, O., Garivier, A., and Szepesvári, C. (2010). Parametric bandits: The generalized linear case. Advances in Neural Information Processing Systems .

Guo, D., Ktena, S. I., Myana, P. K., Huszar, F., Shi, W., Tejani, A., Kneier, M., and Das, S. (2020). Deep Bayesian bandits: Exploring in online personalized recommendations. In Conference on Recommender Systems .

Kveton, B., Wen, Z., Ashkan, A., and Szepesvári, C. (2015). Tight regret bounds for stochastic combinatorial semi-bandits. In International Conference on Artificial Intelligence and Statistics .

Lattimore, T. and Szepesvári, C. (2020). Bandit algorithms . Cambridge University Press.

Liu, X., Zuo, J., Wang, S., Joe-Wong, C., Lui, J., and Chen, W. (2022). Batch-size independent regret bounds for combinatorial semi-bandits with probabilistically triggered arms or independent arms. Advances in Neural Information Processing Systems .

Perrault, P., Boursier, E., Valko, M., and Perchet, V. (2020a). Statistical efficiency of thompson sampling for combinatorial semi-bandits. Advances in Neural Information Processing Systems , 33:5429-5440.

Perrault, P., Valko, M., and Perchet, V . (2020b). Covariance-adapting algorithm for semi-bandits with application to sparse outcomes. In Conference on Learning Theory .

Rusmevichientong, P. and Tsitsiklis, J. N. (2010). Linearly parameterized bandits. Mathematics of Operations Research , 35(2):395-411.

Tabei, G., Ito, Y ., Kimura, T., and Hirata, K. (2023). Design of multi-armed bandit-based routing for in-network caching. IEEE Access .

Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294.

Wang, S. and Chen, W. (2018). Thompson sampling for combinatorial semi-bandits. In International Conference on Machine Learning .

Zeng, C., Wang, Q., Mokhtari, S., and Li, T. (2016). Online context-aware recommendation with time varying multi-armed bandit. In International Conference on Knowledge Discovery and Data Mining .

Appendix

The Supplementary is organized as follows:

-Appendix A proves the lower bound from Theorem 2,

-Appendix B outlines proofs concerning the concentration of the average estimator (Propositions 1)

-Appendix C presents those for the covariance estimator (Proposition 2),

-Appendix D establishes general propositions used to upper-bound the number of times each item is chosen,

-Appendix E and Appendix F detail proofs for OLS-UCB-C and COS-V ,

-Appendix G presents some experimental results.

A Proof of the lower bound (Theorem 2)

Theorem 2. Let d, m ∈ N ∗ such that d/m ≥ 2 is an integer, T ∈ N ∗ , and Σ ⪰ 0 a covariance matrix. Then, there exists a stochastic combinatorial semi-bandit with d base items, and a reward distribution with covariance matrix Σ on which for any policy π , the pseudo regret satisfies

Proof. We follow the methodology of Auer et al. (2002b), modifying it to account for the different variances among actions.

Let d, m ∈ N ∗ such that d/m ≥ 2 is an integer, T ∈ N ∗ , and a covariance matrix Σ ⪰ 0 . Weconsider the structure where A = { a 1 , . . . , a d/m } ⊂ { 0 , 1 } d contains d/m disjoint actions each having m base elements. We consider that for all p ∈ [ d/m ] , ( a p ) i ∈ [ d ] = ( 1 { ( p -1) m<i ≤ pm }) i ∈ [ d ] . Let π be a policy. As all the actions are disjoints, we can reduce ourselves to a multi-armed bandit with d/m actions, where for all p ∈ [ d/m ] the variance of the p -th action is ⟨ a p , Σ p ⟩ .

Let Σ ′ ∈ M d/m ( R ) be the diagonal matrix where for all p ∈ [ d/m ] , Σ ′ p,p = a ⊤ p Σ a p . Let c > 0 , and

where Σ ′ min = min p ∈ [ d/m ] Σ ′ p,p .

We denote G 0 ∼ N (0 , Σ ′ ) a ( d/m ) -dimensional centered Gaussian distribution with covariance matrix Σ ′ . Let p ∈ [ d/m ] , we consider the mean vector µ ( p ) ∈ R d/m having coordinate 0 everywhere and ∆ at coordinate p , for all i ∈ [ d/m ] , µ ( p ) i = ∆ 1 { i = p } . We introduce the Gaussian reward distributions G p ∼ N ( µ ( p ) , Σ ′ ) and denote T p = ∑ T t =1 1 { A t = p } . Then, using policy π , and considering the reward distributions G p and G 0 , the average number of times action p has been chosen satisfies

where TV denotes the total variation distance, KL denotes the Kullback-Leibler divergence and the last inequality uses Pinsker's inequality. Then, using the divergence decomposition between multi-armed bandits (Lemma 15.1 in Lattimore and Szepesvári, 2020),

Reinjecting this expression into Eq. (18), we get

Now, summing over the actions p ,

We denote R ( p ) T the average cumulative regret incurred with the reward distribution G p , then

Therefore, there exists at least one instance p ∗ ∈ [ d/m ] such that

Now, decomposing we get

B Concentration of the average rewards estimations (Proposition 1)

Proposition 1. Let t ≥ d ( d +1) / 2 and δ > 0 . Then, P ( G c t ) ≤ δ/ ( t log( t ) 2 ) .

Proof. Let t ≥ d ( d +1) / 2 and δ > 0 .

We have

Covering argument ( Peeling trick ). The peeling trick consists in separating the space of trajectories up to round t into an exponentially large number of parts, each having an exponentially small probability.

Formally, let 0 < ϵ < 1 . For each p ∈ N d we associate the set

As an abuse of notation, we denote by ( t ∈ D p ) the event ( ( n t, ( i,i ) +1 ) i ∈ [ d ] ∈ D p ) .

Setting P t,ϵ = ⌊ log( t ) log(1+ ϵ ) ⌋ , we define for each p ∈ [ P t,ϵ ] d

In particular, under the event ( t ∈ D p ) , N t ⪯ (1 + ϵ ) ˜ N p .

Using this covering, we decompose

We now apply the following Lemmas.

Lemma 1. Let t ≥ d ( d +1) / 2 , 0 < ϵ < 1 , p ∈ [ P t,ϵ ] d , and δ > 0 . Then, P ( ∥ S t ∥ Z -1 t,p > 6 log ( Norm p Norm t,p ) +6log(1 /δ ) ) ≤ δ , (22) where Norm p = ∫ λ ∈ R d 1 { ∥ Z 1 / 2 0 ,p λ ∥ ≤ 1 2 } exp {-∥ λ ∥ 2 Z 0 ,p } dλ, Norm t,p = ∫ λ ∈ R d 1 { ∥ Z 1 / 2 t,p λ ∥ ≤ 1 2 } exp {-∥ λ ∥ 2 Z t,p } dλ.

Lemma 2. Let t ≥ d ( d +1) / 2 , 0 < ϵ < 1 and p ∈ [ P t,ϵ ] d . Then,

Moreover, under event ( t ∈ D p ) ,

In our case, setting ϵ = e -1 , they yield

B.1 Proof of Lemma 1

Lemma 1. Let t ≥ d ( d +1) / 2 , 0 < ϵ < 1 , p ∈ [ P t,ϵ ] d , and δ > 0 . Then, P ( ∥ S t ∥ Z -1 t,p > 6 log ( Norm p Norm t,p ) +6log(1 /δ ) ) ≤ δ , (22) where Norm p = ∫ λ ∈ R d 1 { ∥ Z 1 / 2 0 ,p λ ∥ ≤ 1 2 } exp {-∥ λ ∥ 2 Z 0 ,p } dλ, Norm t,p = ∫ λ ∈ R d 1 { ∥ Z 1 / 2 t,p λ ∥ ≤ 1 2 } exp {-∥ λ ∥ 2 Z t,p } dλ.

Proof. We adapt the proofs from Faury et al. (2020), which adapts Abbasi-Yadkori et al. (2011) itself. Let t ≥ d ( d +1) / 2 , 0 < ϵ < 1 , p ∈ [ P t,ϵ ] d , and δ > 0 .

Let λ ∈ R d such that ∥ λ ∥ ≤ 1 2 ∥ B ∥ and s ∈ [ t ] . We denote F ′ t -1 = σ ( A 1 , Y 1 , . . . , A t -1 , Y t -1 , A t ) .

Then, ∥ λ ⊤ d A s η s ∥ ≤ 1 / 2 and

which yields that ( M k ( λ ) ) k ∈ N ∗ = ( exp ( λ ⊤ S k -∥ λ ∥ 2 V k ) ) k ∈ N ∗ is a F ′ k -supermartingale.

Let p ∈ [ P t,ϵ ] d , we consider the density g p of a d -dimensional Gaussian with covariance matrix 1 2 ( ˜ N p d Σ + ∥ B ∥ 2 I ) -1 = 1 2 Z -1 0 ,p , truncated in the ellipsoid { x ∈ R d , ∥ Z 1 / 2 0 ,p x ∥ ≤ 1 2 } ,

where Norm p is the normalisation constant.

We integrate ( M k ( λ ) ) k ∈ N ∗ for λ ∼ g p , and define ( ¯ M p,k ) k ∈ N ∗ as

which is still a supermartingale.

Let λ ∗ t,p ∈ arg max { Z 1 / 2 0 ,p ∥ λ ∥≤ 1 4 } ( λ ⊤ S t -∥ λ ∥ 2 Z t,p ) . Then

where we can recognize g k,p the density of a d -dimensional Gaussian with covariance matrix 1 2 Z -1 k,p , truncated in the ellipsoid { x ∈ R d , ∥ Z 1 / 2 0 ,p x ∥ ≤ 1 4 } ,

with Norm k,p the normalisation constant.

Besides, Jensen's inequality yields

Therefore, for all k ∈ N ∗

Markov's inequality yields

Taking k = t in particular gives

The constraint on λ in the inner expression prevent to use the usual optimal value for subgaussian r.v. which could give a bound for ∥ S t ∥ 2 Z -1 t,p . Instead, we introduce

for which

Then

B.2 Proof of Lemma 2

Lemma 2. Let t ≥ d ( d +1) / 2 , 0 < ϵ < 1 and p ∈ [ P t,ϵ ] d . Then,

Moreover, under event ( t ∈ D p ) ,

Proof. Let t ≥ d ( d + 1) / 2 , 0 < ϵ < 1 and p ∈ [ P t,ϵ ] d . Then, following steps from Faury et al. (2020) yields

and

Noting that ∥ Z 1 / 2 0 ,p Z -1 / 2 t,p ∥ ≤ 1 , we deduce

Therefore,

We treat the integrals as

Thus

In particular under event ( t ∈ D p ) ,

C Concentration of the covariances estimations (Proposition 2)

Proposition 2. Let t ≥ d ( d +1) / 2 and δ > 0 . Then, P ( C c t ) ≤ δ/ ( t log( t ) 2 ) .

It is a direct application of the following proposition:

Proposition 5. Let δ ∈ (0 , 1) . Then with probability 1 -δ , for all t ≥ d ( d +1) / 2 and ( i, j ) ∈ [ d ] 2 'reachable',

where h t,δ = (1 + 2 log(1 /δ ) + 2 log( d ( d +1)) + log(1 + t )) 1 / 2 .

Proof. Let δ > 0 , t ≥ d ( d + A ) / 2 . We remind

Let ( i, j ) ∈ [ d ] 2 'reachable'. Then

And,

A triangle inequality yields

We make repeated use of the following Lemma

Lemma 3. Let ( H t ) t ∈ N ∗ be a filtration, ( U t ) t ∈ N ∗ be an H t adapted martingales bounded by C ∈ R ∗ + with E [ U 1 ] = 0 , and ( 1 { V t } ) t ∈ N ∗ be a predictable process and δ > 0 .

Then with probability at least 1 -δ , for all t

Therefore, with probability at least 1 -δ/ 2 , for all ( i, j ) and t ,

With probability at least 1 -δ/ 2 , for all i and t ,

To simplify this expression, using n t, ( i,j ) ≤ min { n t, ( i,i ) , n t, ( j,j ) } and n t, ( i,j ) ≤ √ n t, ( i,i ) n t, ( j,j ) yields

Denoting h t,δ = ( 1 + 2 log(1 /δ ) + 2 log ( t log( t ) 2 d ( d + 1) ) + log(1 + t ) ) 1 / 2 , we have with probability at least 1 -2 δ d ( d +1) t log( t ) 2

A union bound yields the desired results.

C.1 Proof for Lemma 3

Lemma 3. Let ( H t ) t ∈ N ∗ be a filtration, ( U t ) t ∈ N ∗ be an H t adapted martingales bounded by C ∈ R ∗ + with E [ U 1 ] = 0 , and ( 1 { V t } ) t ∈ N ∗ be a predictable process and δ > 0 .

Then with probability at least 1 -δ , for all t

Proof. Let t ≥ 2 . Then U t is C sub-Gaussian and for all λ ∈

Then ( W t ( λ )) t ∈ N ∗ = ( exp( λ ∑ t s =1 1 { V s } U s -λ 2 C 2 2 ∑ t s =1 1 { V s } ) ) t ∈ N ∗ is a supermatringale. We use the Method of Mixtures by integrating for a λ ∼ N (0 , 1 /C 2 ) . This yield

≤ 1 .

Therefore,

Using the stopping time construction from Abbasi-Yadkori et al. (2011) yields the property for all t .

D Behaviour in the high-probability events (Section 5)

The following proposition states that under some assumptions on the sequence of events ( E t ) , the regret can be bounded by problem-dependent quantities (including Σ , T , or d ). They are not all explicitly stated in Proposition 6 to make it adaptive to both algorithms but are hidden in the constants.

Proposition 6. Let r ∈ N , e ∈ (1 , + ∞ ) r . Let ( E t ) t ≥ d ( d +1) / 2 be a sequence of events such that for all t ≥ d ( d +1) / 2 , under E t ,

where C and ( C s ) s ∈ [ r ] are problem-dependent positive constants. C A t +1 ,i is a positive constant depending on A t +1 and i so that, for all a ∈ A , C a,i ≤ 2 m Σ i,i . Let c ∈ R ∗ + and ( c s ) s ∈ [ r ] ∈ ( R ∗ + ) r be positive constants such that 1 /c + ∑ s ∈ [ r ] 1 /c s = 1 .

Then,

where ( α k ) k ∈ N ∗ , ( β k ) k ∈ N ∗ and k 0 ∈ N ∗ are defined in Appendix D.1.

Proof. The proof is classical and involves a decomposition of the events E t (see Kveton et al. (2015); Degenne and Perchet (2016); Perrault et al. (2020b)). By considering each of the r sub-sum in Eq. (25) and designing sets of event that can happen only a finite number of times.

We introduce two sequences ( α k ) k ∈ N ∗ and ( β k ) k ∈ N ∗ , both begin at 1 and strictly decrease to 0 (see Appendix D.1 for their definitions). These sequences are introduced to be able to consider the different terms of Eq. (25) separately.

Let t ≥ d ( d +1) / 2 , k ∈ N ∗ . We define the set

and the event

A notable difference from previous approaches is the use of Σ ∗ i,i /C a,i in A 1 t,k instead of set cardinals. This enables the explicit appearance of the C a,i coefficients, which will involve the σ 2 a,i for the application of this proposition to our algorithms.

For s ∈ [ r ] , we define

and the events

The following Lemma, proven in Appendix D.2, decomposes ( E t ) t ≥ d ( d +1) / 2 using these events.

Lemma 4. Let's consider the assumptions of Proposition 6. Let A t,k and ( A s t,k ) s ∈ [ r ] be the events defined in Eq. (28) and Eq. (30) . Let k 0 ∈ N ∗ such that 0 < mβ k 0 < 1 2 m and t ≥ d ( d +1) / 2 .

Using it, we decompose

We begin with the first term of Eq. (31). Let t ≥ d ( d +1) / 2 , and k ∈ [ k 0 ] . Then,

Therefore,

Summing over t and k , and including the gaps yields

Let i ∈ [ d ] , we consider all the actions associated to it. Let q i ∈ N ∗ be the number of actions associated to item i . Let l ∈ [ q i ] , we denote e l i ∈ A the l -th action associated to item i , sorted by decreasing ∆ e l i C e l i ,i , with C e 0 i ,i ∆ e 0 i = 0 by convention. Then

Reinjecting Eq. (35) into Eq. (34) yields

We treat the r other terms in a similar way. Let s ∈ [ r ] , t ≥ d ( d +1) / 2 , and k ∈ [ k ] ,

Summing over t and k yields

Let ( i, j ) ∈ [ d ] 2 , we consider all the actions which are associated to it. Let q ( i,j ) ∈ N ∗ be the number of actions associated to the tuple ( i, j ) . Let l ∈ [ q ( i,j ) ] , this time, we denote e l ( i,j ) ∈ A the l -th action associated to tuple ( i, j ) , sorted by decreasing ∆ e l ( i,j ) , with 1 ∆ e 0 ( i,j ) = 0 by convention. Then,

If e s = 2 , then

Reinjecting this expression into yields Eq. (38), for e s = 2

This yield

Finally, for e s > 2 ,

and

All in all, we reinject Eq. (36), Eq. (39), Eq. (40) and Eq. (41) into Eq. (31), yielding

D.1 Definition of the sequences ( α k ) and ( β k )

Let β = 1 / 5 , x > 0 . We define β 0 = α 0 = 1 . For k ≥ 1 , we define

Let's first look for an adequate k 0 for Lemma 4, taking 1 ≤ k 0 = ⌈ 2 log( √ 2 m ) log(1 /β ) +1 ⌉ ≤ (2 log( m ) + 3) is sufficient to have 0 < mβ k 0 < 1 2 m . This choice particularly yields

for x = 4 k 0 +2 .

Besides, as m ≥ 5 . Let c ∈ R , c > 1 . Then

D.2 Proof of Lemma 4

Lemma 4. Let's consider the assumptions of Proposition 6. Let A t,k and ( A s t,k ) s ∈ [ r ] be the events defined in Eq. (28) and Eq. (30) . Let k 0 ∈ N ∗ such that 0 < mβ k 0 < 1 2 m and t ≥ d ( d +1) / 2 .

Proof. Let's consider the assumptions of 6, A t,k and ( A s t,k ) s ∈ [ r ] be the events defined in Eq. (28) and Eq. (30). Let k 0 ∈ N ∗ such that 0 < mβ k 0 < 1 2 m and t ≥ d ( d +1) / 2 .

We first prove that the events for k ≥ k 0 cannot happen. Let k ≥ k 0 ,

As β k m < β k 0 m < 1 2 m ≤ min i,a Σ i,i C a,i and ( S 1 t,l ) l is a decreasing sequence of sets, ∑ i ∈ S t,k 0 Σ i,i C At +1 ,i < β k 0 m imply S t,k 0 = ∅ and ∑ i ∈ S t,k Σ ∗ i,i C At +1 = 0 < β k m . Therefore, A t,k cannot happen and we denote

Likewise, for k > k 0 and s ∈ [ r ] ,

As β k 0 m 2 < 1 / 2 < 1 and ( S s t,l ) l is a decreasing sequence of sets, then | S s t,k 0 | < β k 0 m 2 imply S s t,k 0 = ∅ and | S s t,k | = 0 < β k m 2 . Therefore, A s t,k cannot happen and we denote

The idea is now to prove that

We begin by considering ( A 1 t ) c ,

Then, under ( A t ) c , denoting S t, 0 = A t +1 , as S t,k 0 = ∅ and the sets S t,k are decreasing with respect to k ,

Likewise for s ∈ [ r ] ,

Denoting S s t, 0 = A t +1 × A t +1 , as S s t,k 0 = ∅ ,

Therefore, under A c t ∩ ∩ r s =1 ( A s t ) c , summing Eq. (47) and Eq. (49)

which contradict Eq. (25) and thus imply E c t . By contraposition, we have proved that E t imply A t ∩ ⋃ r s =1 ( A s t ) . Therefore,

E Details for OLS-UCB-C (Section 5.2)

E.1 Proof of Proposition 3

Proposition 3. Let δ > 0 . Then, OLS-UCB-C yields

Proof. The objective is to use Proposition 6.

Proposition 6. Let r ∈ N , e ∈ (1 , + ∞ ) r . Let ( E t ) t ≥ d ( d +1) / 2 be a sequence of events such that for all t ≥ d ( d +1) / 2 , under E t ,

where C and ( C s ) s ∈ [ r ] are problem-dependent positive constants. C A t +1 ,i is a positive constant depending on A t +1 and i so that, for all a ∈ A , C a,i ≤ 2 m Σ i,i . Let c ∈ R ∗ + and ( c s ) s ∈ [ r ] ∈ ( R ∗ + ) r be positive constants such that 1 /c + ∑ s ∈ [ r ] 1 /c s = 1 . Then,

where ( α k ) k ∈ N ∗ , ( β k ) k ∈ N ∗ and k 0 ∈ N ∗ are defined in Appendix D.1.

We need to check that its hypotheses are satisfied. Let t ≥ d ( d +1) / 2 and δ > 0 , then we have the Lemma.

Lemma 5. Let δ > 0 and t ≥ d ( d +1) / 2 . Then under {G t ∩ C t } , OLS-UCB-C satisfies

Therefore, we can choose r = 3 , e = (2 , 3 / 2 , 3) and ( E t ) = ( G t ∩ C t ) . Taking c = (4 , 4 , 4 , 4) and identifying the rest of the coefficients yields that OLS-UCB-C satisfies

As

we deduce

E.2 Proof of Lemma 5

Lemma 5. Let δ > 0 and t ≥ d ( d +1) / 2 . Then under {G t ∩ C t } , OLS-UCB-C satisfies

Proof. Let t ≥ d ( d +1) / 2 and δ > 0 . OLS-UCB-C statisfies the following Lemma.

Lemma 6. Let t ≥ d ( d +1) / 2 and δ > 0 . Then for OLS-UCB-C , under the event {G t ∩ C t } ,

Therefore, under {G t ∩ C t } , and

From, here, we develop the right-hand side,

As Z t = ∑ t s =1 d A s Σ ∗ d A s + d Σ ∗ N t + ∥ B ∥ 2 I , we get

by rearranging terms.

Now as for all ( i, j ) ∈ [ d ] 2 , n t, ( i,j ) ≤ min { n t, ( i,i ) , n t, ( j,j ) } , then

Denoting ¯ σ 2 A t +1 ,i = 2 ∑ j ∈ A t +1 / Σ j,j ≤ Σ i,i ( Σ i,j ) + yields

The second term from the right-hand side of Eq. (51) is developed in the same manner but involves more terms.

We remind that ˆ Z t = ∑ t s =1 d A s ˆ Σ t d A s + d ˆ Σ t N t + ∥ B ∥ 2 I where for all ( i, j ) ∈ [ d ] 2 ,

Being under the event C , Proposition 2 yields ˆ Σ t, ( i,j ) ≤ Σ i,j + B i B j 4 ( 5 h t,δ √ n t, ( i,j ) + h 2 t,δ n t, ( i,j ) + 1 n 2 t, ( i,j ) ) . Then,

Reinjecting Eq. (52) and Eq. (53) into Eq. (51) yields

The desired inequality just comes from f T,δ ≥ f t,δ

E.3 Proof of Lemma 6

Lemma 6. Let t ≥ d ( d +1) / 2 and δ > 0 . Then for OLS-UCB-C , under the event {G t ∩ C t } ,

Proof. Let t ≥ d ( d +1) / 2 and δ > 0 . The error in estimating the mean reward for action a with ⟨ a, ˆ µ t ⟩ is bounded as

The definition of G t = { ∥ ∥ ∑ t s =1 d A s η s ∥ ∥ Z -1 t ≤ f t,δ } yields that in this event,

and

By the definition of A t +1 for OLS-UCB-C in (6),

Combining the expressions gives

Now, we use the fact that under C t , ˆ Z t uses coefficient-wise upper bounds of Σ , which yields that ∥ N -1 t a ∗ ∥ 2 Z t ≤ ∥ N -1 t a ∗ ∥ 2 ˆ Z t .

Rearranging terms the desired result.

E.4 Proof of the gap-free bound

Theorem 1. Let T ∈ N ∗ and δ > 0 .

Then, OLS-UCB-C (Alg. 2) satisfies the gap-dependent regret upper bound

where σ 2 a,i = ∑ j ∈ a max { Σ i,j , 0 } , and the gap-free regret upper bound

Proof. Let ∆ > 0 , then

Adapting Proposition 6 to account for ∆ A t +1 > ∆ yields

where ≲ is an inequality up to constant factors (when T varies).

Balancing T ∆ and

F Details for COS-V (Section 5.3)

F.1 Proof for Lemma 7

Lemma 7. Let δ > 0 . Then COS-V satisfies

Proof. Let δ > 0 , t ≥ d ( d +1) / 2 . We remind

where g t,δ = ( 2 log ( 2 dt log( t ) 2 ) +log(1 /δ ) ) 1 / 2 .

Conditionally to F t = σ ( A 1 , Y 1 , . . . , A t , Y t ) , for all i ∈ [ d ]

Let i ∈ a ∗ . Then Gaussian concentration yields

by integration.

A union bound on i ∈ [ d ] and t ≥ d ( d +1) / 2 yields the result

F.2 Proof for Proposition 4

Proposition 4. Let δ > 0 . Then COS-V yields

Proof. Let δ > 0 . We first make use of the following Lemma.

Lemma 8. Let t ≥ d ( d +1) / 2 , δ > 0 . Then for COS-V , under {G t ∩ C t ∩ H t } ,

This enables to use a 'modified' version of Proposition 6, which do not consider covariances.

Proposition 7. Let r ∈ N , e ∈ (1 , + ∞ ) r . Let ( E t ) t ≥ d ( d +1) / 2 be a sequence of events such that for all t ≥ d ( d +1) / 2 , under E t ,

where C and ( C s ) s ∈ [ r ] are problem-dependent positive constants. C i is a positive constant depending on i so that, C i ≤ 2 m Σ i,i . Let c ∈ R ∗ + and ( c s ) s ∈ [ r ] ∈ ( R ∗ + ) r be positive constants such that 1 /c + ∑ s ∈ [ r ] 1 /c s = 1 .

Then,

where ( α k ) k ∈ N ∗ , ( β k ) k ∈ N ∗ and k 0 ∈ N ∗ are defined in Appendix D.1.

Applied to COS-V , this yields

where

We deduce

F.3 Proof for Lemma 8

Lemma 8. Let t ≥ d ( d +1) / 2 , δ > 0 . Then for COS-V , under {G t ∩ C t ∩ H t } ,

Proof. Let t ≥ d ( d +1) / 2 and δ > 0 . Then

by definition of A t +1 . Besides,

Under G t ∩ C t , for all i ∈ a ∗ ,

Under H t , for all i ∈ a ∗ ,

Therefore, under {G t ∩ H t ∩ C t } ,

We now develop the expression

As and

Therefore,

This finally yields

G Experimental results

This section outlines some experimental results.

G.1 Theoretical regret upper bound

In this experiment, the objective is to show the effect of the smallest suboptimality gap ∆ min over theoretical gap-dependent regret upper bounds for ESCB-C and OLS-UCB-C . To that end, we sampled 100 environments with different ∆ min , with a constant number of items d = 20 , a horizon of T = 10 5 rounds, and randomly sampled structures. We represent theoretical upper bounds with respect to 1 / ∆ min in Fig. 1.

For readability reasoning, we have rescaled and reweighted the different components of the sums so that the leading term in the upper-bounds ( 1 / ∆ min or 1 / ∆ 2 min for ESCB-C or OLS-UCB-C ) is greater/smaller than the rest, in a significant number of cases. In particular, all the theoretical upper bounds have the form

where C, C ′ and C r are the tuned constants. For OLS-UCB-C ,

For ESCB-C ,

When the minimal gap is too small (right part of Fig. 1), both upper-bounds are of the magnitude of either 1 / ∆ 2 min or 1 / ∆ min (depending on the algorithm). In this case, the theoretical regret bound of OLS-UCB-C outperforms the one of ESCB-C (green dots vs. blue dots). On the other side, when the gap is big enough, the remaining terms have more impact. In this case, ESCB-C has a better theoretical guarantee (orange dots vs. red dots).

G.2 Comparison between ESCB-C and OLS-UCB-C

We evaluate ESCB-C (approximated as proposed in Perrault et al., 2020b) and OLS-UCB-C on d = 5 items, P = 10 actions, T = 10 5 rounds and randomly sampled structures.

We represent the pseudo-regret evolutions in Fig. 2. The evolutions remain the same until 10 3 rounds. After that, ESCB-C seemingly performs better than OLS-UCB-C which has a supplementary log( t ) factor and is more conservative. However, just before 10 5 rounds, we can observe a slight regime change for ESCB-C while the pseudo-regret of OLS-UCB-C continues to increase smoothly. The average regret of ESCB-C seems to have an inflexion point upward to meet the q75 curve.

When observing the final regret with respect to 1 / ∆ min in Fig. 3, overall ESCB-C seems to outperform OLS-UCB-C except on some corner cases. Those cases skew the distribution for ESCB-C . Especially, for the case with the smallest suboptimality gap (the rightmost part of the figure), OLS-UCB-C outperforms ESCB-C .

The evolution of the pseudo-regret in this case with the smallest suboptimality gap is presented in Fig. 4. While ESCB-C seems to fare better in the beginning, we actually see a sharp increase in its pseudo-regret before 10 5 rounds. It could have been caused by the computational approximation of ESCB-C (described in Perrault et al. (2020b)), and/or it could be the impact of the 1 / ∆ 2 min term.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims of the paper are stated in Theorems and Propositions which for which the technical proofs are in the Appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The results and theorems are discussed with comments made especially about some limitations. In particular, computational complexity is addressed.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The Theorems and proofs are numbered and cross-referenced. The proofs for the main results are outlined in the paper and formally written in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper is mainly states theoretical results and include no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The answer NA means that the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics. The paper does not involve human subjects or participants, and the data-related concerns are not applicable.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is mainly theoretical and is not directly tied to an application.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper focuses on stochastic combinatorial semi-bandit problems where a player selects from a power set of actions comprising subsets of d base items. It underscores the importance of adapting to the problem structure to achieve optimal regret bounds, emphasizing the use of covariance matrix estimation to enhance performance. The proposed "optimistic" algorithms, OLS-UCB-C and COS-V, employ online covariance estimation. While COS-V prioritizes improved computational efficiency over slight optimality, inspired by Thompson Sampling, it marks the first sampling-based algorithm to achieve a $T^.5$ gap-free regret bound.

STRENGTHS:
The paper is well-written.

The worst-case guarantees and instance-dependent guarantees represent a significant enhancement over existing algorithms that adapt to covariance. 

2 different and relevant algorithms and analysis are proposed.

The comparison of regret rates and computational complexity between different algorithms is clear.

WEAKNESSES:
The log(T) term has a power 3 in the regret bound, which can be improved.

The approach of estimating the covariance matrix online and using it for confidence bounds is not new.

No empirical evaluation.

QUESTIONS:
Do you think the log(T)^3 factor can be reduced ? Why is there such a factor, whereas usually there is simply a log(T) factor ?

LIMITATIONS:
A sharp dependence on the horizon $T$ is more important for the instance-dependent guarantees.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Towards Efficient and Optimal Covariance-Adaptive Algorithms for Combinatorial Semi-Bandits\" addresses the challenge of designing algorithms that adapt to covariance in the context of stochastic combinatorial semi-bandits, where decision-makers face a set of actions exponentially large in the number of base items.\n\nSTRENGTHS:\n1. The paper presents new algorithms that improve on both the computational efficiency and regret bounds by adapting to the covariance structure of the environment, which is crucial for applications where the rewards are not independent across actions.\n\n2. It provides the first gap-free regret bounds (regret bounds that do not depend on the minimum sub-optimality gap between the best and second-best actions) for these types of problems, extending the theory beyond independent rewards assumptions.\n\nWEAKNESSES:\n1. The assumption that the reward (Lines 45-46) for each base item \\( Y_{t,i} \\) is bounded by \\( \\frac{B_i}{2} \\) may be strong, where reward distributions can exhibit significant variability and are not tightly bounded. \n\n2. While the algorithms' theoretical framework and mathematical formulation are well-articulated, the lack of experimental results or simulations leaves a crucial gap.\n\nQUESTIONS:\n1. Concerns regarding the role of `f_t,δ` and `g_t,δ` functions in Eq. 6 and 7.\nThe algorithms utilize the functions `f_t,δ` and `g_t,δ` to dynamically adjust the exploration strategy. The paper implies an increase in `f_t,δ` and `g_t,δ` as \\( t \\) increases, which leads to an increasing exploration bonus and variance over time. This design choice raises several concerns and questions, especially from an intuitive standpoint:\n\n     a) Increasing Exploration Over Time: Typically, one might expect that as an algorithm learns more about the environment, the need for exploration would decrease, reducing the exploration bonus and variance. The paper’s approach where `f_t,δ` increases with \\( t \\) seems counterintuitive. An explanation or justification for why increased exploration is necessary or beneficial as the algorithm progresses would be crucial. \n\n     b) Design Justification for `g_t,δ` and `f_t,δ`: The paper lacks a detailed explanation for the design of `g_t,δ` and `f_t,δ`. Are there specific characteristics of the problem domain or empirical observations that suggest this approach? \n\n   c) Impact on Algorithm’s Performance: How does the increasing trend of `f_t,δ` and `g_t,δ` impact the algorithm's overall performance across different learning stages? Some insights into how this approach compares to more traditional methods where exploration decreases over time would be beneficial. \n\n2. The paper specifies using a Gaussian distribution in Eq. 7 for parameter sampling within the COS-V algorithm. Could the authors provide further justification for the choice of a Gaussian distribution in this context?\n\nLIMITATIONS:\nAs described in the paper, the results and theorems are discussed with comments made especially about some limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the combinatorial semi-bandits problem and provides many theoretical results, including gap-free variance-dependent upper bounds for both deterministic and stochastic sampling strategies, and for the least, they also provide a corresponding lower bound.\n\nSTRENGTHS:\nThe paper is well-written in a reader-friendly manner. It contributes a  variety of highly non-trivial theoretical results that improve upon the current state-of-the-art, with extensive proofs in the appendix.\n\nWEAKNESSES:\nThe paper does not provide any empirical evaluation, it would have been interesting to see how the presented algorithms perform empirically against its competitors. But given the theoretical contributions and nature of this paper, this is okay for me.\n\nApart of that I have only the following rather minor issues and typos/suggestions:\n- Be more precise in the formulation of the lower bound (Thm. 2): In which sense does the inequality hold, in expectation? Also, is policy restricted deterministic policies here?\n- Some of the notations may be clear for most readers, but should for the sake of completeness still be properly introduced. E.g., $\\mathbb{N}^\\ast$, $\\mathbf{I}$ and which logarithm (base) $\\log$ refers to.\n- In Algo 2, line 3,  does it have to be $\\leq 2$ instead of $\\leq 1$?\n- 8: yield[s]\n- 10: $\\mathcal{O}(\\sqrt{T})$ instead of $\\sqrt{T}$?\n- 28: At each round\n- 108: adaptative\n- 139: positive semi-definite\n- 269: $\\mathbb{P}(\\mathcal{C}^c)$\n- 272: sketch of the proof\n- You oftentimes write $A\\bigcap B$ instead of $A\\cap B$, why?\n\nQUESTIONS:\n- Do you have a feeling how sensible the regret is w.r.t. the vector $B$? You assume this to be known, but I could imagine that in practice it's oftentimes unknown.\n- In Table 1, do you know how $C_{1/T}^{opt}$ scales in comparison to $d^2$? That is, do you know that $OLS-UCB-C$ outperforms $ESCB-C$ regarding time complexity?\n- Both your algorithms have a parameter $\\delta > 0$, which is not contained in the upper bounds. Could you say how the behaviour of the regrets are w.r.t. this parameter?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}