Title: Nearly Minimax Optimal Submodular Maximization with Bandit Feedback
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bandits

Submodular optimization

Minimax optimal
Abstract: We consider maximizing an unknown monotonic, submodular set function $f: 2^{[n]} \rightarrow [0,1]$ with cardinality constraint under stochastic bandit feedback. 
    At each time $t=1,\dots,T$ the learner chooses a set $S_t \subset [n]$ with $|S_t| \leq k$ and receives reward $f(S_t) + \eta_t$ where $\eta_t$ is mean-zero sub-Gaussian noise. 
    The objective is to minimize the learner's regret with respect to an approximation of the maximum $f(S_*)$ with $|S_*| = k$, obtained through robust greedy maximization of $f$. 
    To date, the best regret bound in the literature scales as $k n^{1/3} T^{2/3}$. 
    And by trivially treating every set as a unique arm one deduces that $\sqrt{ {n \choose k} T }$ is also achievable using standard multi-armed bandit algorithms. 
    In this work, we establish the first minimax lower bound for this setting that scales like $\tilde{\Omega}(\min_{L \le k}(L^{1/3}n^{1/3}T^{2/3} + \sqrt{{n \choose k - L}T}))$.  For a slightly restricted algorithm class, we prove a stronger regret lower bound of $\tilde{\Omega}(\min_{L \le k}(Ln^{1/3}T^{2/3} + \sqrt{{n \choose k - L}T}))$.
    Moreover, we propose an algorithm Sub-UCB that achieves regret $\tilde{\mathcal{O}}(\min_{L \le k}(Ln^{1/3}T^{2/3} + \sqrt{{n \choose k - L}T}))$ capable of matching the lower bound on regret for the restricted class up to logarithmic factors.

Nearly Minimax Optimal Submodular Maximization with Bandit Feedback

Abstract

We consider maximizing an unknown monotonic, submodular set function f : 2 [ n ] → [0 , 1] with cardinality constraint under stochastic bandit feedback. At each time t = 1 , . . . , T the learner chooses a set S t ⊂ [ n ] with | S t | ≤ k and receives reward f ( S t ) + η t where η t is mean-zero sub-Gaussian noise. The objective is to minimize the learner's regret with respect to an approximation of the maximum f ( S ∗ ) with | S ∗ | = k , obtained through robust greedy maximization of f . To date, the best regret bound in the literature scales as kn 1 / 3 T 2 / 3 . And by trivially treating every set as a unique arm one deduces that √ ( n k ) T is also achievable using standard multi-armed bandit algorithms. In this work, we establish the first minimax lower bound for this setting that scales like ˜ Ω(min L ≤ k ( L 1 / 3 n 1 / 3 T 2 / 3 + √( n k -L ) T )) . For a slightly restricted algorithm class, we prove a stronger regret lower bound of ˜ Ω(min L ≤ k ( Ln 1 / 3 T 2 / 3 + √( n k -L ) T )) . Moreover, we propose an algorithm Sub-UCB that achieves regret ˜ O (min L ≤ k ( Ln 1 / 3 T 2 / 3 + √( n k -L ) T )) capable of matching the lower bound on regret for the restricted class up to logarithmic factors.

1 INTRODUCTION

Optimizing over sets of n ground items given noisy feedback is a common problem. For example, when a patient comes into the hospital with sepsis (bacterial infection of the blood), it is common for a cocktail of 1 < k ≤ n antibiotics to be prescribed. This can be attractive for reasons including 1) the set could be as effective (or more) than a single drug alone, but each unit of the cocktail could be administered at a far lower dosage to avoid toxicity, or 2) could be more robust to resistance by blocking a number of different pathways that would have to be overcome simultaneously, or 3) could cover a larger set of pathogens present in the population. In this setting the prescriber wants to balance exploration with exploitation over different subsets to maximize the number of patients that survive. As a second example, we consider factorial optimization of web-layouts: you have n pieces of content and k locations on the webpage to place them-how do you choose subsets to maximize metrics like click-through rate or engagement?

Given there are ≈ n k ways to choose k items amongst a set of n , this optimization problem is daunting. It is further complicated by the fact that for any set S t that we evaluate at time t , we only get to observe a noisy realization of f , namely y t = f ( S t )+ η t where η t is mean-zero, sub-Gaussian noise. In the antibiotics case, this could be a Bernoulli indicating whether the patient recovered or not, and in the web-layout case this could be a Bernoulli indicating a click or a (clipped) real number to represent the engagement time on the website. To make this problem more tractable, practitioners make structural assumptions about f . A common assumption is to assume that higher-

order interaction terms are negligible [20, 12]. For example, assuming only interactions up to the second degree would mean that there exist parameters θ (0) ∈ R , θ (1) ∈ R n , and θ (2) ∈ R ( n 2 ) such that

̸

However, this model can be very restrictive and even if true, the number of unknowns scales like n 2 which could still be intractably large.

An alternative strategy is to remain within a non-parametric class, but reduce our ambitions to measuring performance relative to a different benchmark which is easier to optimize. We say a set function f : 2 [ n ] → R is increasing and submodular if for all A ⊂ B ⊂ [ n ] we have f ( A ) ≤ f ( B ) and

Such a condition limits how quickly f can grow and captures some notion of diminishing returns. Diminishing returns is reasonable in both the antibiotics and webpage optimization examples. It is instructive to note that a sufficient condition for the parametric form of (1) to be submodular is for max i,j θ (2) i,j ≤ 0 . But in general, f still has ≈ n k degrees of freedom even if it is monotonic and submodular. And it is known that for an unknown f , identifying S ∗ := arg max S ⊂ [ n ]: | S | = k f ( S ) may require evaluating f as many as n k times.

The power of submodularity is made apparent through the famous result of [26] which showed that the greedy algorithm which grows a set one item at a time by adding the item with the highest marginal gain returns a solution that is within a (1 -e -1 ) -multiplicative factor of the optimal solution. That is, if we begin with S f gr = ∅ and set S f gr ← arg max i ∈ [ n ] \ S f gr f ( S f gr ∪{ i } ) until | S f gr | = k , then f ( S f gr ) ≥ (1 -1 /e ) f ( S f ∗ ) where S f ∗ := arg max S ∈ [ n ]: | S |≤ k f ( S ) if f is increasing and submodular. This result is complemented by [13] which shows achieving any (1 -e -1 + ϵ ) -approximation is NP-Hard. Under additional assumptions like curvature, this guarantee can be strengthened.

Due to the centrality of the greedily constructed set to the optimization of a submodular function, it is natural to define a performance measure relative to the greedily constructed set. However, as discussed at length in the next section, because we only observe noisy observations of the underlying function, recovering the set constructed greedily from noiseless evaluations is too much to hope for. Consequently, there is a more natural notion of regret against a noisy greedy solution, denoted R gr , that actually appears in the proofs of all upper bounds found in the literature for this setting (see the next section for a definition).

For this notion of regret, previous works have demonstrated that a regret bound of R gr = O ( poly ( k ) n 1 / 3 T 2 / 3 ) is achievable ([28], [36]). This T 2 / 3 rate is unusual in multi-armed bandits, where frequently we expect a regret bound to scale as T 1 / 2 . On the other hand, by treating each k -subset as a separate arm, one can easily adapt existing algorithms to achieve a regret bound of √ ( n k ) T . This leads to the following question:

Does there exist an algorithm that obtains √ n r T regret for r = o ( k ) on every instance? And if not, what is the optimal dependence on k and n for a bound scaling like T 2 / 3 ?

To address these questions, we prove a minimax lower bound and complement the result with an algorithm achieving a matching upper bound. To be precise, the contributions of this paper include:

A minimax lower bound demonstrating that R gr = ˜ Ω ( min 0 ≤ L ≤ k ( L 1 / 3 n 1 / 3 T 2 / 3 + √( n k -L ) T ) ) . In words, for small T , a T 2 / 3 regret bound is inevitable, for large T the √ ( n k ) T bound is optimal, with an interpolating regret bound for in between.

-For slightly restricted class of algorithms with non-adaptive greedy error threshold, we have the improved R gr = ˜ Ω( ( min 0 ≤ L ≤ k ( Ln 1 / 3 T 2 / 3 + √( n k -L ) T ) ) .

where

We propose an algorithm that for any increasing, submodular f , we have R gr = ˜ O min 0 ≤ L ≤ k ( Ln 1 / 3 T 2 / 3 + √( n k -L ) T ) . As this matches our lower bound, we conclude that this is the first provably tight algorithm for optimizing increasing, submodular functions with bandit feedback. Existing algorithms construct a set by greedily adding k items. Our main insight is that it is actually optimal to build up a set up to a size i ∗ and then for the remaining stages play sets of size k that include the initial set of size i ∗ . Our choice of i ∗ is directly motivated by our lower bound.

In what remains, we will formally define the problem, discuss the related work, and then move on to the statement of the main theoretical results. Experiments and conclusions follow.

1.1 Problem Statement

Let [ n ] = { 1 , . . . , n } denote the set of base arms, T be the time horizon, and k be a given cardinality constraint. At time t , the agent selects a set S t ⊂ [ n ] where | S t | ≤ k , and observes reward f ( S t )+ η t where η t is i.i.d. mean-zero 1-sub-Gaussian noise, and f : 2 [ n ] → [0 , 1] is an unknown monotone non-decreasing submodular function defined for all sets of cardinality at most k .

Ideally, our goal would be to minimize the regret relative to pulling the best set S ∗ := arg max | S |≤ k f ( S ) at each time. In general, even if we had the ability to evaluate the true function f ( · ) (i.e. without noise), maximizing a submodular function with a cardinality constraint is NP-hard. However, greedy algorithms which sequentially add points, i.e. S ( i +1) = arg max a ̸∈ S ( i ) f ( S ( i ) ∪ a ) , 1 ≤ i ≤ k guarantee that f ( S ( k ) ) ≥ αf ( S ⋆ ) with α ≥ 1 -1 /e in worst-case. Unfortunately, since we do not know f ( · ) and instead only have access to noisy observations, running the greedy algorithm on any estimate ˆ f ( · ) may not necessarily guarantee an α = (1 -1 /e ) -approximation to f ( S ∗ ) 1 .

Consequently, a natural notion to address noisy observations is an ϵ -approximate greedy set for ϵ ∈ [0 , 1] k . We define the following collection of sets of size k

Intuitively, any S ∈ S k, ϵ can be thought of as being constructed from a process that adds an element at stage i which is ϵ i -optimal compared to the Greedy algorithm run on f . Such a set naturally arises as the output of the Greedy algorithm run on an approximation ˆ f . This set enjoys the following guarantee.

Lemma 1.1. (Theorem 6 in [36]) For any ϵ ≥ 0 ∈ R k , and S k, ϵ gr ∈ S k, ϵ , we have

Lemma 1.1 is a noise-robust analogous result to the approximation ratio of the perfect greedy algorithm of [26] that says f ( S k, 0 gr ) ≥ (1 -e -1 ) f ( S ∗ ) . Note that |S k, ϵ | is non-decreasing in ϵ i for all i ∈ [ k ] , so identifying a set in S k, ϵ is in some sense easier for a larger 1 T ϵ . Thus, to define an appropriate definition of regret, the measure must balance the facts that comparing with the noiseless greedy approximation in S k, 0 may be impossible, but should account for identifying a set in S k, ϵ is strictly easier for larger 1 T ϵ . Inspired by the above lemma we define robust greedy regret

This notion of regret captures the fact that if the algorithm plays a set in S k, ϵ then they may be incurring up to 1 T ϵ extra regret. Note that when ϵ = 0 achieves the minimum (which can happen if the 'gaps' between the greedily added element and all other elements at each stage is large) then this notion of regret is relative to the greedy set constructed in the noiseless setting.

The definition of regret in (3) is not novel to our paper. This notion is implicitly used in [36] in the proofs of Lemma 3 for the full-feedback setting and Theorem 13 for the bandit feedback setting, [28] in Theorem 4.1, [31] in Theorem 1, [27] in Theorem 2 for the full-feedback setting and Theorem 4 for bandit feedback, and [29] in Theorem 1. However, readers of these papers will note that they report their results not in terms of R gr , but α -Regret: for an α ∈ [0 , 1] , define α -regret by, R α := ∑ T t =1 αf ( S ∗ ) -f ( S t ) where S ∗ := arg max | S |≤ k f ( S ) . Using Lemma 1, one immediately has that R α ≤ R gr for α = (1 -e -1 ) . Thus, an upper bound on (3) immediately results in an upper bound on R α , which is precisely what previous works exploit to obtain their upper bounds on R α .

To summarize: all the analyses of these previous works concentrate on showing an upper bound on R gr , and only at the last step argue that R α ≤ R gr , and report an upper bound on R α . But R α can be a very loose lower bound on R gr ! For instance, when the function is modular (the inequalities of submodularity are tight), and the gap between the best set and worst set is equal to ∆ < e -1 , then a random selection algorithm would get zero or even negative R α regret, while R gr would be linear ∆ T , which is more natural. Thus, in studying regret against approximations attained by an offline step-wise greedy procedure, R gr can be a more appropriate measure than R α

1.2 Related Work

There has been several works on combinatorial multi-armed bandits with submodular assumptions and different feedback assumptions. Table 1 summarizes of the most relevant results as well as the results of this paper. For monotonic submodular maximization specifically, previous work use Lemma 1.1 with appropriate ϵ to prove an upper bound on expected R α -regret when the greedy result with perfect information gives an α -approximation of the actual maximum value.

Stochastic In the stochastic setting, when the expected reward function is submodular and monotonic, [28] proposed an explore-then-commit algorithm with full-bandit feedback that achieves R gr = O ( k 4 / 3 T 2 / 3 n 1 / 3 ) 2 . Recently, [31] showed with the same explore-then-commit algorithm with different parameters, R gr = O ( kn 1 / 3 T 2 / 3 + kn 2 / 3 T 1 / 3 d ) is possible with delay feedback parameter of d . Without the monotonicity, [15] achieves R α = O ( nT 2 / 3 ) with bandit feedback for α = 1 / 2 . There have also been several works in the semi-bandit feedback setting ([42], [45]), and others such as getting the marginal gain of each element after each query.

Adversarial In the adversarial setting, the environment chooses an arbitrary sequence of monotone submodular functions { f 1 , . . . , f T } , and the goal is to minimize regret against an approximation of the reward of the best set in hindsight ([17], [19], [38], [41]). [37] showed O ( k √ Tn log n ) R (1 -e -1 ) -regret is possible with partially transparent feedback(where after each round, f ( S ( i ) ) for

all i is revealed instead of only f ( S ( k ) ) ) and O ( kn 1 / 3 T 2 / 3 ) R (1 -e -1 ) -regret for the bandit-feedback setting. [27] proposed a generalized algorithm with ˜ O ( kn 2 / 3 T 2 / 3 ) R (1 -e -1 ) -regret with full bandit feedback, and showed all explore-then-commit greedy algorithms have Ω( T 2 / 3 ) regret, when applied to our setting. Without the monotone assumption, [27] gets O ( nT 2 / 3 ) R (1 / 2) -regret with bandit feedback. The upper-bound results in the adversarial setting doesn't naturally lead to results in the stochastic setting as the function is submodular and monotone only in expectation in the stochastic setting.

Continuous Submodular There are several works on online maximization of the continuous extensions of submodular set functions to a compact subspace such as Lovász and multilinear extensions([3], [14]). With a stronger assumption of DR-submodularity, it's possible to achieve higher approximation ratio guarantees and lower regret bounds ([7], [8], [33]). [41] uses multilinear extension to achieve O ( T 2 / 3 ) R (1 -e -1 ) -regret for adversarial submodular maximization with partition matroid constraint.

Low-degree polynomial In general reward functions without the submodular assumption, [12] showed if the reward function is a d -degree polynomial, Θ ( min( √ n d T, √ n k T ) ) regret is optimal.

2 LOWERBOUND

Theorem 2.1. For any n ≥ 4 , k ≤ ⌊ n/ 3 ⌋ , satisfying 512 k 7 n ≤ T ∈ N , let F denote the set of submodular functions that are non-decreasing and bounded by [0 , 1] for sets of size k or less, with f ( ∅ ) = 0 . Then

where the infimum is over all randomized algorithms and the supremum is over the functions in F , and i ∗ ∈ [ k ] is the largest value satisfying 16 n 2 k 6 ( n -k i ∗ ) 3 ≤ T .

The lowerbound is intuitively a mix of the greedy explore-then-commit algorithm for the first k -i ∗ arms, and then a standard MAB algorithm between all superarms of cardinality k that include those elements. For small T (i.e. T = O ( n 4 ) ) the regret would be Ω( k 1 / 3 n 1 / 3 T 2 / 3 ) , and for large T (i.e. T = Ω( n 3 k -2 ) ) the regret would be Ω( ( n k ) 1 / 2 T 1 / 2 ) . This lowerbound also immediately gives a lower bound for the adversarial setting where f i = f + N (0 , 1) is the function chosen by the environment at time i .

Proof Sketch We construct a hard instance so that at each cardinality a single set gives an elevated reward. Focusing on k = 2 for illustration, the instance would be the following:

For a set of alternative instances, we choose a set of size k and elevate its reward by 2∆ . We also elevate every prefix set of a permutation of this set by 2∆ so that the new set can be found by a where ∆ is the gap of the best set that we will tune based on T . Pulling any arm of cardinality less than 2 would incur Ω(1) regret, however, since there are only n such sets (compared to ( n 2 ) sets of size 2 ), pulling these simple arms give more information on the optimal set.

greedy algorithm. Again, for k = 2 , and any { ˆ i, ˆ j } ∈ [ n ] \{ 1 , 2 }

Note that, if ∆ < 1 16 for the k = 2 instance, All the functions are submodular, as f ( { a, b } ) -f ( { b } ) ≤ 1 4 +2∆ ≤ 1 / 2 -∆ ≤ f ( { a } ) -f ( { ϕ } ) for any a, b ∈ [ n ] .

For H 0 , if ϵ i < ∆ for all i ∈ [2] , then f H 0 ( S 2 , ϵ gr ) = 3 4 as the noisy greedy finds the best arm, and otherwise 1 T ϵ ≥ ∆ , so min ϵ ≥ 0 f H 0 ( S 2 , ϵ gr ) + 1 T ϵ = 3 4 . Similarly, min ϵ ≥ 0 f H ̂ i, ̂ j ( S 2 , ϵ gr ) + 1 T ϵ = 3 4 +∆ . So for these instances R gr = R ( S ∗ ) .

We show that if the KL divergence between an alternate instance and H 0 is small, then the algorithm cannot distinguish between the two environments and the maximum regret of the two would be Ω(∆ T ) . Let P ̂ i, ̂ j , E ̂ i, ̂ j be the probability and expectation under H ̂ i, ̂ j , respectively when executing some fixed algorithm with observations being corrupted by standard Gaussian noise. Then KL ( P 0 | P ̂ i, ̂ j ) = ∆ 2 2 ( E 0 [ T ̂ i ] + 4 E 0 [ T ̂ i, ̂ j ] ) for k = 2 , where T S is the number of pulls of set S , and

Lemma 2.2. There exists a pair i, j such that

Since ̂ i, ̂ j were arbitrary, the following Lemma shows that there exist a pair that are pulled for small number of times in expectation (see Lemma A.2 for general k ).

Proof. For a pair ( i, j ) , define Q ( i,j ) := E 0 [ T i ] + E 0 [ T i,j ] . Then the sum of this term for all pairs not equal to 1 , 2 would be

̸

̸

̸

Then by Pigeonhole principal there exist a pair i, j such that

Using the lemma, for some ( ̂ i, ̂ j ) , we have E 0 [ R gr ] + E ̂ i, ̂ j [ R gr ] ≥ 1 2 n ∑ i =1 E 0 [ T i ] + ∆ T 4 exp ( -2∆ 2 ( 2 n -2 ∑ i E 0 [ T i ] + T ( n -2 2 ) ) ) We choose an appropriate ∆ based on value of i ∗ .

If i ∗ = 1 , then for ∆ = T -1 / 3 n 1 / 3 , we have 2∆ 2 T ( n -2 2 ) ≤ 1 . So either the KL divergence is less than 2 , then the regret is lowerbounded by ∆ Te -2 = T 2 / 3 n 1 / 3 e -2 , or for KL divergence to be larger than 2 we would have ∑ i E 0 [ T i ] ≥ 1 4 T 2 / 3 n 1 / 3 , which from the above equation shows the regret is Ω( T 2 / 3 n 1 / 3 ) . This can be extended for expected value of pulls of each cardinality lower than i ∗ +1 for general k .

This shows that the expected regret is ˜ Ω(min i ( i 1 / 3 n 1 / 3 T 2 / 3 + √( n k -i ) T )) . The instance of general k , and the detailed proof is in appendix A.1. □

If i ∗ = 2 , then it can be shown that the term 1 2 ∑ n i =1 E 0 [ T i ] + ∆ T 4 exp ( -2∆ 2 ( 2 n -2 ∑ i E 0 [ T i ] + ( T -∑ n i =1 E 0 [ T i ]) / ( n -2 2 )) ) with ∆ = √ ( n -2 2 ) /T minimizes when ∑ n i =1 E 0 [ T i ] = 0 i.e. zero single arm sets being pulled in expectation, so the regret would be T 1 / 2 ( n -2 2 ) 1 / 2 exp( -2) .

We define an algorithm to be in non-adaptive greedy error-threshold class against R gr regret, if it selects ϵ ′ 1 , . . . , ϵ ′ k at the start only dependent on parameters T, n, k before any arm pulls, and minimizes regret against f ( S k, ϵ ′ gr ) + 1 T ϵ ′ . All the algorithms from previous work in the literature fall within this restricted class, and with this extra assumption we can prove a stronger lower bound.

Theorem 2.3. For any n ≥ 4 , k ≤ ⌊ n/ 3 ⌋ , satisfying 512 k 9 n ≤ T ∈ N , let F denote the set of submodular functions that are non-decreasing and bounded by [0 , 1] for sets of size k or less, with f ( ∅ ) = 0 . Then

where the infimum is over all randomized algorithms with non-adaptive greedy error threshold selection, and the supremum is over the functions in F , and i ∗ ∈ [ k ] is the largest value satisfying 16 n 2 k 6 ( n -k i ∗ ) 3 ≤ T .

3 UCB UPPER BOUND

Algorithm 1 SUB-UCB algorithm for set bandits with cardinality constraints

A natural approach to minimizing regret is to take an Explore-Then-Commit strategy motivated by the greedy algorithm. Such an algorithm would be the following - proceed in k rounds. Set S 0 = ∅ .

In round i pull each set in the collection { S i -1 ∪{ a } : a ∈ [ n ] \ S i -1 } , m times. Use these samples to update our estimate ˆ f of f on these sets, and set S ( i ) ← arg max a ∈ [ n ] \ S i -1 ˆ f ( S i -1 ∪{ a } ) . This approach has been pursued by existing works [28], and with an appropriate choice of m results in O ( kn 1 / 3 T 2 / 3 ) regret.

The disadvantage of this approach is that it can not achieve the correct trade-off between √ n k T and kn 1 / 3 T 2 / 3 exhibited by the lower bound. Motivated by the statement of the lower bound, our algorithm SUB-UCB attempts to interpolate between these different regret regimes. The critical quantity is i ∗ . For the first k -i ∗ cardinalities, our algorithm plays a UCB style strategy which more or less follows the ETC strategy described in the previous paragraph. After that, it defaults to a UCB algorithm on all subsets containing S k -i ∗ , a total of ( n -k + i ∗ i ∗ ) possible arms. Theorem 3.1. For any l ≤ k , SUB-UCB guarantees

when m = T 2 / 3 n -2 / 3 log T 1 / 3 .

Proof Sketch We show that for ϵ := 2 √ 2 log(2 knT 2 ) /m , the greedy part of SUB-UCB with high probability adds an ϵ -optimal arm in each step. Defining event G to be | ˆ µ S -f ( S ) | ≤ √ 2 T S log(2 knT 2 ) for all iterations, we prove that this event is true with a probability of at least 1 -1 T .

On Event G , We show that an ϵ -good arm is selected at each step of the greedy algorithm for ϵ = 2 √ 2 log(2 knT 2 ) m . Let a be a sub-optimal arm with expected reward value more than 2 √ 2 log(2 knT 2 ) m from the best arm in the i -th step i.e. ∆ S ( i ) ,a := max a ′ f ( S ( i ) ∪ { a ′ } ) -f ( S ( i ) ∪ { a } ) ≥ 2 √ 2 log(2 knT 2 ) m . Then if arm a is added in i -th step, we have U a ( t ) ≥ U a ∗ ( t ) ≥ f ( S ( i )) ∪{ a ∗ } , and therefore, so ˆ µ S ( i ) ∪{ a } -f ( S ( i ) ∪{ a } ) > √ 2 log(2 knT 2 ) m . This is a contradiction with event G , so on event G such an arm cannot be selected. Lastly, we expand the regret of two stages. As UCB in the second part of the algorithm has the regret of 65 √ T ( n -k k -l ) log T + 32 15 ( n -k k -l ) against the best arm containing S ( l ) (see [24]), it is an upper bound for the regret against the greedy solution were the first l steps select an ϵ -good arm, and the last k -l steps select the best arm, so on event G the regret can be written against a set in S k, ϵ where

Therefore, the expected regret E [ R gr ] on event G can be written as

for any choice of m and l . So for m = T 2 / 3 n -2 / 3 log 1 / 3 (2 knT 2 ) the above term becomes ˜ O ( lT 2 / 3 n 1 / 3 + √ T ( n k -l ) ) . The detailed proof is in Appendix B □

4 EXPERIMENTS

For the experiments we compare SUB-UCB ( l ) for different greedy stop levels l , SUB-UCB ( k -i ∗ ) which selects the best stop level based on the regret analysis, the ETCG (explore-then-commit greedy) algorithm from [28], and UCB on all size k arms. Each arm pull has a 1 -Gaussian noise, with 50 trials for each setting. The expected reward functions are the following.

Functions:

The Unique greedy path hard instance i.e.

̸

This function is inspired by the hard instance in the proof of our lower-bound. Note that this particular parameterization is submodular when k ≤ 7 , not for general k .

Weighted set cover function i.e. f C ( S ) = ∑ C ∈C w ( C ) 1 { S ∩ C = ∅} for a partition C of [ n ] and weight function w on the partition. For n = 15 and k = 4 , we use the partitions of size 5 , 5 , 4 , 1 with weights of 1 / 10 , 1 / 10 , 2 / 10 , 6 / 10 respectively.

Results: As illustrated in figure 1, we observe that our algorithm with the level selection of k -i ∗ outperforms both ETCG and naive UCB on all size k arms, as it combines the advantages of greedy approach for small T s and UCB on many super arms for large T . For smaller T s compared to ( n k ) , both SUB-UCB and ETCG outperform normal UCB as it doesn't have enough budget to find optimal sets of size k , so it gets linear regret(as the other two get O ( T 2 / 3 ) ). However, as T becomes larger the reverse happens as ( n k ) T 1 / 2 becomes smaller than T 2 / 3 , but SUB-UCB adopts to T and continues to outperform the two until it converges with naive UCB for very large T .

O-

SUB-UCB(O)

105

O1

SUB-UCB(1)

40000

O1

SUB-UCB(2)

O1

SUB-UCB(3)

O1

SUB-UCB(4)

30000

SUB-UCB(5)

total regret

O

T=10000

104

O1

T=100000

20000

O

1O1

T=10000000

1O1

T=100000000

10000

103

103

104

105

106

107

108

109

0

2

Sub-UCB(I)

3

T

SUB-UCB(ours)

UCB on all size k

10°

total regret

104

10

10

104

10

10~

T

In figure 2, we compare the performance of SUB-UCB for different choices of greedy stop cardinality, and observe that the best choice gradually decreases from k to 0 as T gets larger, and k -i ∗ is a practical selection of the best stop cardinality before running the algorithm. Note that the defined stop level was chosen to minimize the worst-case bound on the regret, and if the gaps between arms on a particular instance are larger than the worst case, this stop level could be conservative. So k -i ∗ is near the optimal stop level, and not the exact one as seen in these figures. Also, the empirical standard derivation is much smaller than O ( T 1 / 2 ) due to the regret symmetry of non-optimal sets at each cardinality, and it's not visible in the plots.

5 CONCLUSION

In this paper we showed that min L ( L 1 / 3 T 2 / 3 n 1 / 3 + √( n k -L ) T ) , ignoring logarithmic factors, is a lower bound on the regret against robust greedy solutions of stochastic submodular functions, and a stronger lower bound if the algorithm class is slightly restricted. We also matched this bound with an algorithm. This work is the first minimax lower bound for submodular bandits, and beyond closing the k 2 / 3 gap between the general lowerbound and upperbound, it remains open to prove similar minimax optimal bounds in settings with different types of constraint such as matroid, or in general, any offline-to-online greedy procedure that is robust to local noise (e.g. Non-monotonic submodular maximization where the greedy approach gets a 1 / 2 -approximation of the function, or DR-submodular optimization for the continuous setting which also has a (1 -e -1 ) -approximation).

References

Mridul Agarwal, Vaneet Aggarwal, Christopher J. Quinn, and Abhishek Umrawal. DART: aDaptive Accept RejecT for non-linear top-K subset identification, November 2020. arXiv:2011.07687 [cs, stat].

Mridul Agarwal, Vaneet Aggarwal, Christopher J. Quinn, and Abhishek K. Umrawal. Stochastic Top-$K$ Subset Bandits with Linear Space and Non-Linear Feedback, October 2021. arXiv:1811.11925 [cs, stat].

Francis Bach. Submodular functions: from discrete to continuous domains. Math. Program. , 175(12):419459, May 2019.

Maria-Florina Balcan and Nicholas J A Harvey. Learning Submodular Functions.

Eric Balkanski and Yaron Singer. The adaptive complexity of maximizing a submodular function. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1138-1151, Los Angeles CA USA, June 2018. ACM.

Andrew An Bian, Joachim M. Buhmann, Andreas Krause, and Sebastian Tschiatschek. Guarantees for Greedy Maximization of Non-submodular Functions with Applications, May 2019. arXiv:1703.02100 [cs, math].

Andrew An Bian, Kfir Y. Levy, Andreas Krause, and Joachim M. Buhmann. Continuous drsubmodular maximization: structure and algorithms. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 486496, Red Hook, NY, USA, 2017. Curran Associates Inc.

Andrew An Bian, Baharan Mirzasoleiman, Joachim Buhmann, and Andreas Krause. Guaranteed Non-convex Optimization: Submodular Maximization over Continuous Domains. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 111-120. PMLR, 20-22 Apr 2017.

Lin Chen, Hamed Hassani, and Amin Karbasi. Online continuous submodular maximization. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1896-1905. PMLR, 09-11 Apr 2018.

Wei Chen, Wei Hu, Fu Li, Jian Li, Yu Liu, and Pinyan Lu. Combinatorial Multi-Armed Bandit with General Reward Functions. In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Wei Chen, Yajun Wang, Yang Yuan, and Qinshi Wang. Combinatorial Multi-Armed Bandit and Its Extension to Probabilistically Triggered Arms, March 2016. arXiv:1407.8339 [cs].

Xi Chen, Yanjun Han, and Yining Wang. Adversarial Combinatorial Bandits with General Non-linear Reward Functions, January 2021. arXiv:2101.01301 [cs, stat].

Uriel Feige. A threshold of ln n for approximating set cover. J. ACM , 45(4):634652, jul 1998.

Moran Feldman and Amin Karbasi. Continuous submodular maximization: Beyond drsubmodularity. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1404-1416. Curran Associates, Inc., 2020.

Fares Fourati, Vaneet Aggarwal, Christopher Quinn, and Mohamed-Slim Alouini. Randomized greedy learning for non-monotone stochastic submodular maximization under full-bandit feedback. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 7455-7471. PMLR, 25-27 Apr 2023.

Michel X. Goemans, Nicholas J. A. Harvey, Satoru Iwata, and Vahab Mirrokni. Approximating Submodular Functions Everywhere. In Proceedings of the Twentieth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 535-544. Society for Industrial and Applied Mathematics, January 2009.

Daniel Golovin, Andreas Krause, and Matthew Streeter. Online submodular maximization under a matroid constraint with application to learning assignments, 2014.

Botao Hao, Tor Lattimore, and Mengdi Wang. High-Dimensional Sparse Linear Bandits, September 2021. arXiv:2011.04020 [cs, math, stat].

Nicholas Harvey, Christopher Liaw, and Tasuku Soma. Improved Algorithms for Online Submodular Maximization via First-order Regret Bounds. In Advances in Neural Information Processing Systems , volume 33, pages 123-133. Curran Associates, Inc., 2020.

Daniel N Hill, Houssam Nassif, Yi Liu, Anand Iyer, and SVN Vishwanathan. An efficient bandit algorithm for realtime multivariate optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1813-1821, 2017.

Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the Complexity of Best Arm Identification in Multi-Armed Bandit Models, November 2016. arXiv:1407.4443 [cs, stat].

Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning , 49(2):209-232, 2002.

Andreas Krause and Daniel Golovin. Submodular Function Maximization. In Lucas Bordeaux, Youssef Hamadi, and Pushmeet Kohli, editors, Tractability , pages 71-104. Cambridge University Press, 1 edition, February 2014.

Tor Lattimore and Csaba Szepesvari. Bandit algorithms. 2017.

Tatsuya Matsuoka, Shinji Ito, and Naoto Ohsaka. Tracking Regret Bounds for Online Submodular Optimization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , pages 3421-3429. PMLR, March 2021. ISSN: 2640-3498.

George L Nemhauser and Laurence A Wolsey. Submodular set functions, matroids and the greedy algorithm: Tight worst-case bounds and some generalizations of the rado-edmonds theorem. Combinatorica , 3(3-4):257-268, 1978.

Rad Niazadeh, Negin Golrezaei, Joshua Wang, Fransisca Susan, and Ashwinkumar Badanidiyuru. Online Learning via Offline Greedy Algorithms: Applications in Market Design and Optimization, February 2023. arXiv:2102.11050 [cs, math, stat].

Guanyu Nie, Mridul Agarwal, Abhishek Kumar Umrawal, Vaneet Aggarwal, and Christopher John Quinn. An Explore-then-Commit Algorithm for Submodular Maximization Under Full-bandit Feedback. 2022.

Guanyu Nie, Yididiya Y Nadew, Yanhui Zhu, Vaneet Aggarwal, and Christopher John Quinn. A framework for adapting offline algorithms to solve combinatorial multi-armed bandit problems with bandit feedback. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

Stephen U. Pasteris, Alberto Rumi, Fabio Vitale, and Nicolò Cesa-Bianchi. Sum-max submodular bandits. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 2323-2331. PMLR, 02-04 May 2024.

Mohammad Pedramfar and Vaneet Aggarwal. Stochastic Submodular Bandits with Delayed Composite Anonymous Bandit Feedback, March 2023. arXiv:2303.13604 [cs].

Tim Roughgarden and Joshua R. Wang. An optimal learning algorithm for online unconstrained submodular maximization. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory , volume 75 of Proceedings of Machine Learning Research , pages 1307-1325. PMLR, 06-09 Jul 2018.

Omid Sadeghi, Prasanna Raut, and Maryam Fazel. Improved Regret Bounds for Online Submodular Maximization, June 2021. arXiv:2106.07836 [cs, math, stat].

Max Simchowitz, Kevin Jamieson, and Benjamin Recht. Best-of-K Bandits, March 2016. arXiv:1603.02752 [cs, stat].

Adish Singla, Sebastian Tschiatschek, and Andreas Krause. Noisy submodular maximization via adaptive sampling with applications to crowdsourced image collection summarization. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , AAAI'16, page 20372043. AAAI Press, 2016.

Matthew Streeter and Daniel Golovin. An Online Algorithm for Maximizing Submodular Functions:. Technical report, Defense Technical Information Center, Fort Belvoir, V A, December 2007.

Matthew Streeter and Daniel Golovin. An Online Algorithm for Maximizing Submodular Functions. In Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc., 2008.

Matthew Streeter, Daniel Golovin, and Andreas Krause. Online learning of assignments. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems , volume 22. Curran Associates, Inc., 2009.

Maxim Sviridenko, Jan Vondrák, and Justin Ward. Optimal approximation for submodular and supermodular optimization with bounded curvature, December 2014. arXiv:1311.4728 [cs].

Zoya Svitkina and Lisa Fleischer. Submodular approximation: sampling-based algorithms and lower bounds, May 2010. arXiv:0805.1071 [cs].

Zongqi Wan, Jialin Zhang, Wei Chen, Xiaoming Sun, and Zhijie Zhang. Bandit multi-linear DR-submodular maximization and its applications on adversarial submodular bandits. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 35491-35524. PMLR, 23-29 Jul 2023.

Zheng Wen, Branislav Kveton, Michal Valko, and Sharan Vaswani. Online influence maximization under independent cascade model with semi-bandit feedback. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 30263036, Red Hook, NY, USA, 2017. Curran Associates Inc.

Mingrui Zhang, Lin Chen, Hamed Hassani, and Amin Karbasi. Online continuous submodular maximization: From full-information to bandit feedback. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Qixin Zhang, Zengde Deng, Zaiyi Chen, Haoyuan Hu, and Yu Yang. Stochastic continuous submodular maximization: Boosting via non-oblivious function. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 26116-26134. PMLR, 17-23 Jul 2022.

Junlong Zhu, Qingtao Wu, Mingchuan Zhang, Ruijuan Zheng, and Keqin Li. Projection-free decentralized online learning for submodular maximization over time-varying networks. Journal of Machine Learning Research , 22(51):1-42, 2021.

A Lowerbound proofs

A.1 Proof of Theorem 2.1

For any { x 1 , x 2 , . . . , x k } ∈ ( [ n ] \{ 1 ,...,k } k ) , define instance H 0 , H ( x 1 ,...,x k ) , H ( x i +1 ,...,x k ) with reward functions as follows:

where H n = ∑ n k =1 1 k is the n -th harmonic number.

Lemma A.1. If ∆ ≤ (1 / 8 k 2 ) then H 0 and H ( x 1 ,...,x k ) are submodular.

Proof. for any S ⊊ T ⊂ [ n ] where | T | < k (the function is only defined on sets of cardinality at most k ) and x / ∈ T we have to show f ( S + x ) -f ( S ) ≥ f ( T + x ) -f ( T ) .

For H 0 if ϵ i < ∆ at each step i of the greedy arm selection, then S k, ϵ gr = { 1 , . . . , k } , otherwise f H 0 ( S k, ϵ gr ) + 1 T ϵ ≥ H 2 k -H k +∆ -∆ = H 2 k -H k = f H 0 ( { 1 , . . . , k } ) . So min ϵ f H 0 ( S k, ϵ gr ) + 1 T ϵ = f H 0 ( { 1 , . . . , k } ) . This means that we can compute our regret against f H 0 ( { 1 , . . . , k } ) . Similarly, min ϵ f H ( x 1 ,...,x k ) ( S k,ϵ gr ) + 1 T ϵ = H 2 k -H k +∆= f H ( x 1 ,...,x k ) ( { x 1 , . . . , x k } ) showing that we can compute our regret against { x 1 , · · · , x k } .

Let E 0 and E ( x 1 ,...,x k ) denote the probability law under H 0 and H ( x 1 ,...,x k ) , respectively. For any S ⊂ [ n ] let T S denote the random variable describing the number of time the set S is played by a policy π . Define T i := ∑ S ⊂ [ n ]: | S | = i T S .

Then by the definition of H 0 we have

Lemma A.2. For any i ≤ k here exist a sequence ( x i , . . . , x k ) , where

Proof. For i ≤ k and a sequence ( x i , ..., x k ) , define Q ( x i ,...,x k ) := ∑ k j = i E 0 [ T { 1 ,...,i -1 ,x i ,...,x j } ] . Then we have

̸

Then by Pigeonhole principle, the exists a sequence ( x , . . . , x ) such that

Lemma A.3. For H 0 and H ( x 1 ,...,x k ) defined above, we have

where P 0 ( S ) = N ( f H 0 ( S ) , 1) and P { x i ,...,x k } ( S ) = N ( f H ( x i ,...,x k ) ( S ) , 1) are the reward distributions of arm S in H 0 and H ( x i ,...,X k ) respectively.

Using two above lemmas, we have,

̸

̸

For 1 ≤ i ≤ k -i ∗ + 1 , n 2 / 3 T 1 / 3 ( n -k k -i +1 ) ≤ 1 by definition of i ∗ ; so either the maximum regret is larger than 1 4 T 2 / 3 ( k -i ∗ ) 1 / 3 n 1 / 3 exp( -8) , which proves the theorem, or 1 n -k E 0 [ T i ] + 2 ( n -k )( n -k -1) E 0 [ T i +1 ] + 4 ( n -k )( n -k -1) ∑ k -1 j = i +2 k -j 2 k E 0 [ T j ] ≥ 3(1 / ∆ 2 ) . If the third term is larger than 1 / ∆ 2 , then ∑ k -1 j = i +2 k -j 2 k E 0 [ T j ] ≥ 1 16 n ( k -i ∗ ) 2 / 3 T 2 / 3 n 1 / 3 which proves the lowerbound as n ( k -i ∗ ) 2 / 3 ≥ ( k -i ∗ ) 1 / 3 . Therefore, the only remaining case is that either the first or second term is ≥ 1 / ∆ 2 . This means that for 1 ≤ i ≤ k -i ∗ + 1 , either E 0 [ T i ] ≥ 1 4 ( k -i ∗ ) -2 / 3 T 2 / 3 n 1 / 3 or E 0 [ T i +1 ] ≥ 1 8 ( n -k -1)( k -i ∗ ) -2 / 3 T 2 / 3 n 1 / 3 ≥ 1 4 ( k -i ∗ ) -2 / 3 T 2 / 3 n 1 / 3 . Therefore, for at least half of the 1 ≤ i ≤ k -i ∗ +1 , E 0 [ T i ] ≥ 1 4 ( k -i ∗ ) -2 / 3 T 2 / 3 n 1 / 3 , and

Note that since T ≥ 512 k 7 n , we have ∆ ≤ ( kn/T ) 1 / 3 ≤ 1 8 k 2 , so the functions with this selection of ∆ are submodular.

̸

̸

̸

We now lower bound the regret in a different way. Let λ := ∑ k -1 j = k -i ∗ +1 k -i 2 k E 0 [ T i ] T , then λ ≤ 1 , and using lemma A.2 we have that there exists a selection of ( x i , . . . , x k ) such that,

̸

̸

The last inequality holds as log( 4 T 1 / 2 ( n -k k -i +1 ) 3 / 2 ( n -k ) T ) ≤ 0 , and the function relative to λ is convex, λ = 0 minimizes in the last inequality. Combining the two parts of the proof we have

̸

A.2 Proof of Theorem 2.3

We generalize the lowerbound distance of Theorem 2.1 by having the gap ∆ i in cardinality i . For any { x 1 , x 2 , . . . , x k } ∈ ( [ n ] \{ 1 ,...,k } k ) , define instance H 0 , H ( x 1 ,...,x k ) , H ( x i +1 ,...,x k ) with reward functions as follows:

̸

The KL divergance between reward distribution of two instances is similarly:

Lemma A.4. For any i ≤ k here exist a sequence ( x i , . . . , x k ) , where

Proof. For i ≤ k and a sequence ( x i , ..., x k ) , define Q ( x i ,...,x k ) := ∑ k j = i ∆ 2 j E 0 [ T { 1 ,...,i -1 ,x i ,...,x j } ] . Then we have

̸

Then by Pigeonhole principle, the exists a sequence ( x i , . . . , x k ) such that

Wenowassign ∆ i for lower cardinalities based on the value of ∆ k . If 1 T ϵ ′ ≤ 2∆ k , For i ≤ k -1 , we assign ∆ i = ϵ ′ i , so a greedy procedure with ϵ ′ will retrieve the best set, hence f H 0 ( S k, ϵ ′ gr ) + 1 T ϵ ′ ≥ f H 0 ( { 1 , . . . , k } ) and f H ( x i ,...,x k ) ( S k, ϵ ′ gr )+ 1 T ϵ ′ ≥ f H ( x i ,...,x k ) ( { 1 , . . . , i -1 , x i . . . , x k } ) . Otherwise, since the gap of any set of size k and the best set is at most 2∆ k for both H 0 and H ( x i ,...,x k ) , f H 0 ( S k, ϵ ′ gr ) + 1 T ϵ ′ ≥ H 2 k -H k -∆ k +2∆ k = f H 0 ( { 1 , . . . , k } ) and f H ( x i ,...,x k ) ( S k, ϵ ′ gr ) + 1 T ϵ ′ ≥ H 2 k -H k -∆ k +2∆ k ≥ f H ( x i ,...,x k ) ( { 1 , . . . , i -1 , x i . . . , x k } ) ; so for i ≤ k -1 , and we assign ∆ i = ∆ k k . Therefore, in both cases R gr ≥ R ( S ∗ ) , and we give a lower bound for R ( S ∗ ) .

For the first part of the lower bound, we'll assign ∆ k = ( k -i ∗ )( n T ) 1 / 3 . Now similarly to proof of Theorem 2.1, we have

̸

For 1 ≤ i ≤ k -i ∗ + 1 , ( k -i ∗ ) 2 n 2 / 3 T 1 / 3 ( n -k k -i +1 ) ≤ k 2 n 2 / 3 T 1 / 3 ( n -k k -i +1 ) ≤ 1 by definition of i ∗ ; so either the maximum regret is larger than 1 4 T 2 / 3 ( k -i ∗ ) n 1 / 3 exp( -10) , which proves the theorem, or

If the forth term is larger than 1 , then

So ∑ k -1 j = i +3 k -j 2 k E 0 [ T j ] ≥ n 3 96 1 ∆ 2 k ≥ 1 96 ( k -i ∗ ) n 1 / 3 T 2 / 3 which proves the lower bound.

Therefore, the only remaining case is that at least one of the first three terms is ≥ 1 . This means that for 1 ≤ i ≤ k -i ∗ +1 , either E 0 [ T i ] ≥ n 2∆ 2 i , or E 0 [ T i +1 ] ≥ n 4∆ 2 i +1 ( n -k -1) ≥ n 4∆ 2 i +1 , or E 0 [ T i +2 ] ≥ n 12∆ 2 i +2 ( n -k -1)( n -k -2) ≥ n 12∆ 2 i +2 .

Therefore, for at least 1 / 3 of the 1 ≤ i ≤ k -i ∗ +1 , E 0 [ T i ] ≥ n 12∆ 2 i . Let I be all cardinalities in which this inequality holds(so | I | ≥ k -i ∗ 3 ); since ∑ k -1 i =1 ∆ i ≤ 2∆ k , using Lemma C.1, we have

For the second part of the lower bound, using ∆ i ≤ 2∆ k , we have

̸

, and the rest of the argument follows the proof of 2.1.

B Proof of Theorem 3.1

Proof. We use the notation l = k -i ∗ to match our lowerbound, however as k -i ∗ is arbitrary, it can be used for any other choice of l as well. Define the event G := ⋂ k i =1 ⋂ a ∈ [ n ] \ S ( i -1) ⋂ T t =1 g i,a,t where

Let E i be the event that the arm selected at the i -th step of the algorithm is within 2 √ 2 log(2 knT 2 ) m of the best possible arm at that step, i.e.

We prove that on event G , ∪ i ∈ [ k -i ∗ ] E i is true.

Let a be a sub optimal arm with value more than 2 √ 2 log(2 knT 2 ) m from the best arm in the i -th step. That is, if a ∗ := arg max a ′ f ( S ( i ) ∪ { a ′ } ) and ∆ S ( i ) ,a := f ( S ( i ) ∪ { a ∗ } ) -f ( S ( i ) ∪ { a } ) , then assume that ∆ S ( i ) ,a ≥ 2 √ 2 log(2 knT 2 ) m . Then on event G and arm a being added in i -th step, which implies

But this implies that

which is a contradiction of event G . Thus, on event G such an arm cannot be selected.

As UCB in the second part of the algorithm has the regret of 65 √ T ( n -k k -l ) log T + 32 15 ( n -k k -l ) against S ( k ) which is the best size k arm containing S ( k -i ∗ ) (see [24]), on event G , it is an upper bound for the regret against the greedy solution were the first k -i ∗ steps select an ϵ -good arm, and the last i ∗ steps select the best arm, so on event G the regret can be written against a set in S k, ϵ where

Therefore, we upper bound the regret relative to f ( S k ) + 2( k -i ∗ ) 2 log(2 knT 2 ) m , as by Lemma 1.1 it's greater than 1 c (1 -e -c ) f ( S ∗ ) . Let T i be the set of times where we pulled a set of cardinality i From the while loop condition in the algorithm, we have | T i | ≤ a/ ∈ S ( i -1) min 1 ∆ 2 S ( i 1) ,a , m ≤

C Auxiliary Lemmas

Lemma C.1. For any sequence of numbers a 1 , . . . , a n bounded between (0 , 1] , If ∑ i a i ≤ C ≤ 1 , then

Proof. If there exists j, k ∈ [ n ] such that a j < a k , then for a new sequence a ′ i = { a i i / ∈ { j, k } a j + a k 2 i ∈ { j, k } we have

Therefore, the infimum value of ∑ a -2 i over all such sequences is when all elements are equal, and

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We've stated all our results and the assumptions for them in the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: For our theoretical results, the assumptions are clearly stated, and our results are within these assumptions. We also state the experimental limitation of our algorithm in the experiments section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the assumptions for each theorem are stated, a proof sketch is provided in the main text, and full proofs are in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes all the algorithms that are used in experiments and all the function definitions that are used as inputs are clearly written.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: This paper is a theoretical work, and the experiments apply the proposed algorithm in main text on synthetic functions which are fully defined in the main text. There are no datasets used, and no code beyond the main algorithm implementation.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We explain the functions used as inputs, and the synthetic noise applied to them. There are no other setting details as this is mainly a theory paper and the experiments are trivial.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The empirical SDs are included in the plots, although hardly visible because of the logarithmic scale.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: There are no experiments that need anything more than a laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes this paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is mainly theoretical, and there are no direct social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper doesn't pose any such risks as it is theoretical.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No assets are used in this work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are released in this work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: he paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider stochastic bandit submodular maximization. For this problem, there are two known regret upper bounds: a $\\sqrt{T}$ regret upper bound with large coefficients obtained by naively considering the problem as the multi-armed bandit problem, and a $T^{2/3}$ regret upper bound with a relatively small coefficient obtained by the well-known greedy maximization procedure. This paper clarifies that the minimax regret of stochastic bandit submodular maximization can be roughly characterized by the minimum of these two bounds. Additionally, the paper demonstrates that this minimax regret can be achieved by a UCB-based algorithm.\n\nSTRENGTHS:\nIn online submodular maximization with bandit feedback, the focus has often been on achieving a $T^{2/3}$ regret guarantee with reasonable coefficients, and focusing on the $\\sqrt{T}$ regret upper bound obtained by naively applying MAB is new. To my knowledge, this paper presents the first lower bound for bandit submodular maximization, which is an important contribution to the community. Instead of using the commonly employed approximation regret in bandit submodular maximization, it uses regret defined based on the comparator determined by the greedy algorithm, which is somewhat appealing. Although the paper gives an impression of not being thoroughly proofread (see Weakness), it is overall easy to read.\n\nWEAKNESSES:\nOne of the major contributions of this paper is the lower bound, but its explanation could be improved. \nWhile an intuitive explanation is provided from Lines 168 to 172, it is unclear why regret minimization algorithms (such as ETC and the \"standard\" MAB algorithm) appear in the discussion of the lower bound. \n\nThe authors are expected to provide a more detailed explanation of existing algorithms in Section 3. The current explanation lacks clarity on how $\\hat{f}$ is computed.\n\nThe regret definition involves a comparison with $S^{k,\\mathbf{0}}$. Although the authors state that comparing with $S^{k,\\mathbf{0}}$ may be impossible in a noisy setting, can the authors present a lower bound for this statement?\n\nIn Section 1.2, the authors mention the use of the Lovasz extension for online submodular maximization. Isn't this technique used for submodular minimization?\n\nAdditionally, the paper overall gives the impression of not being thoroughly proofread. To highlight some minor points:\nL120-122: [22] in Theorem 4.1, [24] in Theorem 1, .... -> Theorem 4.1 in [22], Theorem 1 in [24], ....,\nL160: Showed -> showed, L232: abbreviation ETC is not defined, Sec3: There are $S^{i}$ and $S^{(i)}$, which are confusing, L268: UBC -> UCB.\n\nQUESTIONS:\nThe reviewer expects the authors to address the questions mentioned above.\n\nLIMITATIONS:\nnot appicable", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate lower bounds for regret for combinatorial multi-armed bandit (CMAB) problems under submodular rewards and bandit feedback.   For stochastic environments, there has been an open question about the gap between $O(\\sqrt{T})$ dependence typically seen in MAB problems (and submodular CMAB with extra feedback) and the $O(T^{2/3})$ dependence typically achieved using explore-then-commit (ETC) algorithms for bandit feedback in past works.  The authors propose a related but different notion of regret then used in previous works and show this regret does indeed have a $\\Omega(T^{2/3})$ dependence.  Their formula generalizes the ``small’’ horizon $T$ regime studied in those past works and identify a minimax bound that interpolates $\\Omega(T^{2/3})$ for small $T$ and $\\Omega(\\sqrt{T})$ for large $T$ (i.e. essentially large enough that separate super-arms can be treated as arms in a standard MAB algorithm).  The authors include experiments on toy functions to illustrate the interpolation.\n\nSTRENGTHS:\n### Major\n- This paper is among the first to tackle lower bounds for combinatorial (cardinality constrained) MAB with monotone submodular rewards and bandit feedback in stochastic environments.  \n    - To do so the authors argue that a notion of regret based on nesting of near-greedily selected sets, which the authors point out was implicit in the proofs of several prior papers (all of which adapted the same classic greedy approximation algorithm from the offline setting), should be used instead of the $1-1/e$-regret that has been used in the past.  \n    -  For this alternative (greedy) regret, the authors prove a $\\Omega(T^{2/3})$ lower bound (for the so called “small $T$” regime), strongly suggesting that there is indeed a hardness gap between CMAB problems with bandit feedback and other studied classes (standard MAB, linear CMAB, even submodular CMAB with semi-bandit feedback).  \n  - The authors also propose an algorithm with matching upper bound regret.  The algorithm basically adapts the greedy algorithm (like previous ETC algorithms had) for some of the horizon and then switches to UCB-style over nesting super-sets)\n\n### Minor\n- The lower bounds and algorithm proposed is not only for the so called “small” $T$ regime but for large enough $T$ that one can sample super-arms as regular arms in a standard MAB algorithm to get $\\sqrt{T}$ dependence.    The lower bound and their matching algorithm identify the trade-off (basically to what cardinality $i^*$ does one follow the greedy before switching to treating all (nesting) cardinality $k$ superarms as standard MAB arms).\n- The authors include experiments with two classes of toy functions to illustrate the trade-offs between ETC style algorithms (designed in prior works for “small” $T$) and UCB style (known to get $\\sqrt{T}$ regret bound for very large $T$).\n\nWEAKNESSES:\n### Major\n- [21] proved $\\Omega(T^{2/3})$ lower bounds for the adversarial setting under a slightly different feedback model (though is relevant for any explore-then-commit strategies) and for a general class of problems (including submodular CMAB as a special case) which is not discussed.\n\n\n### Minor\n- The experiment section does not clearly specify $n$ or $k$ or $i^*$ for the problem instances considered.  In the main text, for $n$, there is only a mention in line 263 only for the weighted set cover “For $n=15$, we use…” but that makes it sound like you are using multiple values of $n$.  Since $i^*$ is described as a “critical quantity” (line 231), its value should be transparent in the experiments. Only Figure 1’s caption, which is for the submodular cover, clearly mentions $n$ and $k$ (without mention of $i^*$’s value).\n- I think more discussion on what ``small’’ entails for some potential applications would be valuable.  In line 169 the example of $T= O(n^4)$ is given.  In the experiments even for a small $n$ and $k$ for toy experiments it is around $T$ is in the millions to billions that there is a tradeoff, though would such large $T$ be plausible for problems mentioned in the introduction as motivation?  Would the rewards remain i.i.d. for each super-arm over such a large horizon?\n\n\n### Very Minor\n- for the summary 127-133, this is pointing out that for instances of a special sub-class which have a strictly better $\\alpha=1$ worst-case approximation guarantee, the $\\alpha$-regret of the larger class with provably harder instances behaves weirdly (even giving negative regret values).  That is an important observation, but $\\alpha$-regret bounds are for a class of problems, arising from the worst cases, and there are harder cases for cardinality-constrained submodular optimization than modular functions, where the $(1-1/e)$ approximation coefficient comes from.  \n- line 80 missing parentheses\n- line 268 “UBC”\n- line 287 ‘were’--> ‘where’\n- In the experiments, 1-Gaussian noise seems strange given the functions are bounded in [0,1].\n- Figure 2 legend use power of 10 notation, or add commas\n- I think the intro may flow better if you open with problems that are submodular and mention they are important, then say people have been trying to solve them but it is unclear the extent that current methods are the best that could be had\n\nQUESTIONS:\n### Major\n1. The greedy algorithm from Nemhauser and Wolsey is classic and was the most widely adapted for online settings, but there are other approximation algorithms for monotone cardinality-constrained submodular maximization that behave differently in subset selection, such as the threshold greedy https://theory.stanford.edu/~jvondrak/data/submod-fast.pdf.  From a quick look it is not clear to me that the sets would satisfy a nesting like that in line 102 (though I didn’t check carefully, maybe it would in a different order than elements get added).  So while the $R_{gr}$ lower bound is over any online algorithm, I would wonder if it is possible it might be too strict, in the sense that a different regret notion based on some structure underlying a different algorithm (like a ``$R_{thresh-gr}$’’ regret) could be defined, still with $ R_{1-1/e} \\leq R_{thresh-gr}$ perhaps permitting $\\sqrt{T}$ upper bound even in the regime where $T$ is small relative to $n$ choose $k$?\n\n2. How would this work generalize to other problems, even similar but harder problems like monotone submodular maximization with knapsack constraints or intersections of matroid and knapsack constraints, or non-monotone problems?  There are known approximation coefficients, so the $\\alpha$ regret could be easily defined accordingly (of course with limitations on interpretability you point out) but would some analog of Lemma 1.1 need to be identified for each of those problem classes and then (3) defined, and in some sense would we need to choose a reference algorithm if there were more than one available that achieved the same approximation guarantee?\n\n### Very Minor \n3. Lemma 1.1 what is the definition of $S_{gr}^{k,\\epsilon}$?  $S_{gr}^{f}$ was defined earlier wrt exact value oracle for $f$.  But $S_{gr}^{k,\\epsilon}$ is ambiguous.  If in Lemma 1.1 any set $S \\in \\mathcal{S}^{k,\\epsilon}$ satisfies then I’d suggest the dummy variable $S$ for clarity.  \n4. what is the size of $i$ for which it is advantageous to be less than $k$?  if in the “small T” regime, that is interesting.\n5. Theorem 2.1 requires $k\\leq \\lfloor{n/3}\\rfloor$ -- is there a simple reason for that?\n6. In the experiments, there are no error bars on the figures – was each marker for a single run?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of stochastic submodular bandits. The main contributions are twofold: it provides a new lower bound for the problem and proposes a novel UCB-type algorithm whose regret matches this lower bound.\n\nSTRENGTHS:\n- The paper introduces notation clearly and provides a sufficiently thorough comparison with related work.\n- The lower bound proof is new. While it adheres to the standard framework for proving regret lower bounds in stochastic bandits, the challenging aspect lies in constructing the problem instance. The problem instance constructed is original.\n- The proposed algorithm showcases an interesting insight on the hardness of the problem, even though it combines well-known approaches for online submodular maximization.\n\nWEAKNESSES:\n- Sub-UCB is a modified version of the algorithm in Nie et al. (2022). While this algorithm has a lower regret upper bound, the algorithm itself is an incremental change compared to prior work.\n- The lower bound is demonstrated to hold for the minimax expected regret $R_{gr}$, which differs from the $\\alpha$-regret commonly used in the literature. While the former is shown to be an upper bound for the latter, it can potentially be much larger. Consequently, a lower bound for $R_{gr}$ does not necessarily imply a corresponding lower bound for $R_\\alpha$. Therefore, I believe it is not sound to argue for the minimax optimality of the problem at hand based on this specific change in the regret notion adopted and analyzed.\n- It seems that the lower bound result needs to be conditioned on a harsh prerequisite that T needs to be very large. For example, when $k=10$, T needs to be at least $10^11$ for the lower bound to hold.\n\nQUESTIONS:\nI make the following suggestions:\n- It would be better to formally prove the constructed problem instances (both $k=2$ and general k cases) are submodular functions (possibly in a lemma). It is not obvious to me for general $k$ thy those functions are submodular.\n- Also, the minimax optimal version of the UCB or Thompson Sampling can be used in the second phase to fully match the lower bound.\n\nLIMITATIONS:\nSee Weakness and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the stochastic submodular bandit problem. In this problem, the decision-maker needs to select a subset with size at most k from a known ground set, and then the decision-maker gains a stochastic reward associated with the subset. The expectation of the reward is a submodular function. This problem is a natural extension of combinatorial linear bandit which relaxed the linearity assumption of the reward function. \n\nThis paper mainly focuses on the lower bound of the submodular bandit problem, which is a recognized open question. Different from other literature of submodular bandit, this paper defines a new notion of regret which is called robust greedy regret. Roughly speaking, the new regret compares the algorithm gained reward with the reward gained by the output subset of an offline “approximated” greedy algorithm. The authors prove a lower bound of their defined notion of regret. This lower bound changes from $kn^{1/3} T^{2/3}$ to $n^k T^{1/2}$ as the size of T, and this results implies that $T^{2/3}$-type regret is inevitable when T is relatively small(polynomial to k and n). \nThis paper also proposed an algorithm that nearly match the lower bound. All their results are in the sense of the new defined “robust greedy regret” rather than commonly used γ-regret.\n\nSTRENGTHS:\nThe major contribution of this paper is the lower bound. Even though the first algorithm for (adversarial) submodular bandit has been proposed for 17 years, we do not have any regret lower bound now. Given this fact, I think a lower bound result on this problem should be extensively encouraged. \n\nBack to the previous algorithms for submodular bandit (both stochastic setting and adversarial setting), all these algorithms only achieve $O(T^{2/3})$ regret. The main reason is that these algorithms must clearly distinguish whether the current round is an exploration round or an exploitation round. The technique in this paper somewhat explains the intuition behind that. That is, the small size subset will bring more information but cost large regret, and the full-rank subset incurs low regret but bring less information. This naturally divide the subset with different size to exploration action and exploitation action. In this sense, I think this paper do give some intuition on the hardness of submodular bandit and this difficulty also suggests that the submodular bandit is completely different in nature from the linear bandit.\n\nOverall, this paper provides me important knowledge and I believe it will also bring new knowledge to the people who are working on submodular bandit.\n\nWEAKNESSES:\nThe main weakness of this paper is that the regret lower bound is in terms of the new defined notion of regret—“robust greedy regret”. However, almost all papers about submodular bandit or more generally combinatorial bandit are considering the notion of a scaled “$\\gamma$-regret”. As the authors state, robust greedy regret is always larger than gamma regret, so a lower bound of robust greedy regret does not imply anything about the lower bound of $\\gamma$-regret. Also, the major difficulty of proving the lower bound of $\\gamma$-regret is “the immediate regret can be negative”, which does not exist in traditional 1-regret. However, by defining the notion of robust greedy regret, the authors can avoid this difficulty. The robust greedy regret lower bound is actually a traditional 1-regret lower bound in the hardcase authors have constructed. So I think this result is still quite far away from figuring out the hardness of $\\gamma$-regret.\n\nAs the author states, $\\gamma$-regret may not be a good metric of submodular bandit problem or other combinatorial bandit problem. But the robust greedy result depends too much on the form of offline algorithm. Also, not all algorithms for submodular bandit are based on the greedy algorithm. For example, the algorithms in [1][2] are totally different from the offline greedy algorithm and they are also not based on the “offline algorithm to online” style. So it’s even hard to define a robust ”xxx” regret. These papers are also missed in the related literature. Thus, I think it’s inappropriate to state “$\\gamma$-regret is not appropriate for this problem”. At least, $\\gamma$-regret provides a way to evaluate all-type algorithm rather than restrict the algorithm in the category of “offline to online conversion”.\n\nPlease note, even if I proposed this weakness. I still appreciate the author's contributions to the lower bound results and I think the contribution of this paper outweighs the weaknesses.\n\n[1] Zongqi Wan, Jialin Zhang, Wei Chen, Xiaoming Sun, Zhijie Zhang, \"Bandit Multi-linear DR-Submodular Maximization and Its Applications on Adversarial Submodular Bandits\", ICML 2023\n\n[2] Stephen Pasteris, Alberto Rumi, Fabio Vitale, Nicolò Cesa-Bianchi, “Sum-max Submodular Bandits”, AISTATS 2024.\n\nQUESTIONS:\nSome suggestions:\n1. In page 19, the proof of lemma C.1 should be added. Although I have checked that the claim is correct, a proof is necessary for a complete paper or the authors can give the reference of the proof.\n2. In abstract, the authors use the notation L. But in the main text, they use the notation i instead of L. \n3. Line 80, there should be a “(” before “min”.\n4. It’s better to remove the claim in line 132: “$R_{gr}$ is the more appropriate measure of performance … not $R_{\\alpha}$”. Or use more rigorous language to limit the scope of the claim to those algorithms that use offline to online conversion.\n5. Some related literature on submodular bandit has been missed, for example, [1][2]. Also, the current version of the literature review of online continuous submodular maximization looks very weird, the authors missed all the important papers in this field. For example, [3,4,5] and more. It is better to survey more in this field. \n\n[1]Zongqi Wan, Jialin Zhang, Wei Chen, Xiaoming Sun, Zhijie Zhang, \"Bandit Multi-linear DR-Submodular Maximization and Its Applications on Adversarial Submodular Bandits\", ICML 2023\n\n[2]Stephen Pasteris, Alberto Rumi, Fabio Vitale, Nicolò Cesa-Bianchi, “Sum-max Submodular Bandits”, AISTATS 2024.\n\n[3]Zhang, M., Chen, L., Hassani, H., and Karbasi, A. “Online continuous submodular maximization: from full information to bandit feedback.” NeurIPS 2019\n\n[4]Chen, L., Hassani, H., and Karbasi, A. “Online continuous submodular maximization”, AISTATS 2018\n\n[5] Zhang, Q., Deng, Z., Chen, Z., Hu, H., and Yang, Y. “Stochastic continuous submodular maximization: Boosting via non-oblivious function”. ICML 2022\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

comments: SUMMARY:
The authors of this paper study the submodular maximization problem with bandit feedback. By adopting regret as the metric, the authors prove the minimax bound of the regret for their problem. A UCB-based algorithm is devised to tackle the submodular optimization problem and the authors prove that this algorithm's regret almost matches the minimax bound. In sum, this paper provides some new and solid theoretical results for submodular optimization.

STRENGTHS:
S1. The authors prove the minimax regret bound of the submodular maximization problem with bandit feedback. 

S2. The authors also devise the algorithm Sub-UCB that can match the minimax bound up to logarithmic factors.

S3. The theoretical results in this paper seem correct.

WEAKNESSES:
W1. There are some typos in the paper. For example, in the abstract "we prove, we prove that".

W2. The experiment is not interesting as it is a pure numerical simulation. What about applications of submodular optimization such as influence maximization where the objective can only be obtained via sampling and is uncertain?

W3. Instead of (1-1/e)-regret, the authors adopt regret as the metric for the reason that (1-1/e)-regret may be loose. Although the greedy solution's approximation is 1-1/e in theory, in practice the approximation ratio can often be close to 1. For example, for the modular objective case mentioned by the authors, greedy-based algorithm is actually optimal. Therefore, the justification of using regret rather than (1-1/e)-regret is not that strong.

QUESTIONS:
W2

LIMITATIONS:
W1, W2, W3
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15