Title: No-Regret M${}^{\natural}$-Concave Function Maximization: Stochastic Bandit Algorithms and NP-Hardness of Adversarial Full-Information Setting
Venue: NeurIPS 2024
Accepted: yes
Keywords: online learning

discrete convex analysis

combinatorial bandit
Abstract: M${}^{\natural}$-concave functions, a.k.a. gross substitute valuation functions, play a fundamental role in many fields, including discrete mathematics and economics. In practice, perfect knowledge of M${}^{\natural}$-concave functions is often unavailable a priori, and we can optimize them only interactively based on some feedback. Motivated by such situations, we study online M${}^{\natural}$-concave function maximization problems, which are interactive versions of the problem studied by Murota and Shioura (1999). For the stochastic bandit setting, we present $O(T^{-1/2})$-simple regret and $O(T^{2/3})$-regret algorithms under $T$ times access to unbiased noisy value oracles of M${}^{\natural}$-concave functions.  A key to proving these results is the robustness of the greedy algorithm to local errors in M${}^{\natural}$-concave function maximization, which is one of our main technical results. While we obtain those positive results for the stochastic setting, another main result of our work is an impossibility in the adversarial setting. We prove that, even with full-information feedback, no algorithms that run in polynomial time per round can achieve $O(T^{1-c})$ regret for any constant $c > 0$ unless $\mathsf{P} = \mathsf{NP}$.  Our proof is based on a reduction from the matroid intersection problem for three matroids, which would be a novel idea in the context of online learning.

No-Regret M ♮ -Concave Function Maximization: Stochastic Bandit Algorithms and NP-Hardness of Adversarial Full-Information Setting

Abstract

M ♮ -concave functions, a.k.a. gross substitute valuation functions, play a fundamental role in many fields, including discrete mathematics and economics. In practice, perfect knowledge of M ♮ -concave functions is often unavailable a priori, and we can optimize them only interactively based on some feedback. Motivated by such situations, we study online M ♮ -concave function maximization problems, which are interactive versions of the problem studied by Murota and Shioura (1999). For the stochastic bandit setting, we present O ( T -1 / 2 ) -simple regret and O ( T 2 / 3 ) -regret algorithms under T times access to unbiased noisy value oracles of M ♮ -concave functions. A key to proving these results is the robustness of the greedy algorithm to local errors in M ♮ -concave function maximization, which is one of our main technical results. While we obtain those positive results for the stochastic setting, another main result of our work is an impossibility in the adversarial setting. We prove that, even with full-information feedback, no algorithms that run in polynomial time per round can achieve O ( T 1 -c ) regret for any constant c > 0 unless P = NP . Our proof is based on a reduction from the matroid intersection problem for three matroids, which would be a novel idea in the context of online learning.

1 Introduction

M ♮ -concave functions form a fundamental function class in discrete convex analysis [32], and various combinatorial optimization problems are written as M ♮ -concave function maximization. In economics, M ♮ -concave functions (restricted to the unit-hypercube) are known as gross substitute valuations [19, 13, 25]; in operations research, M ♮ -concave functions are often used in modeling resource allocation problems [46, 30]. Furthermore, M ♮ -concave functions form a theoretically interesting special case of (DR-)submodular functions that the greedy algorithm can exactly maximize (see, Murota and Shioura [33], Murota [32, Note 6.21], and Soma [48, Remark 3.3.1]), while it is impossible for the submodular case [34, 10] and the greedy algorithm can find only approximately optimal solutions [35]. Due to the wide-ranging applications and theoretical importance, efficient methods for maximizing M ♮ -concave functions have been extensively studied [33, 46, 30, 20, 39].

When it comes to maximizing M ♮ -concave functions in practice, we hardly have perfect knowledge of objective functions in advance. For example, it is difficult to know the exact utility an agent gains from some items, which is often modeled by a gross substitute valuation function. Similar issues are also prevalent in submodular function maximization, and researchers have addressed them by developing no-approximate-regret algorithms in various settings, including stochastic/adversarial environments

and full-information/bandit feedback [49, 14, 43, 54, 16, 36, 37, 38, 51, 55, 11, 40]. On the other hand, no-regret algorithms for M ♮ -concave function maximization have not been well studied, despite the aforementioned importance and practical relevance. Since the greedy algorithm can exactly solve M ♮ -concave function maximization, an interesting question is whether we can develop no-regret algorithms-in the standard sense without approximation -for M ♮ -concave function maximization.

1.1 Our contribution

This paper studies online M ♮ -concave function maximization for the stochastic bandit and adversarial full-information settings. Below are details of our results.

In Section 4, we study the stochastic bandit setting, where we can only observe values of an underlying M ♮ -concave function perturbed by sub-Gaussian noise. We first consider the stochastic optimization setting and provide an O ( T -1 / 2 ) -simple regret algorithm (Theorem 4.2), where T is the number of times we can access the noisy value oracle. We then convert it into an O ( T 2 / 3 ) -cumulative regret algorithm (Theorem 4.3), where T is the number of rounds, using the explore-then-commit technique. En route to developing these algorithms, we show that the greedy algorithm for M ♮ -concave function maximization is robust to local errors (Theorem 3.1), which is one of our main technical contributions and is proved differently from related results in submodular and M ♮ -concave function maximization.

In Section 5, we establish the NP -hardness of no-regret learning for the adversarial full-information setting. Specifically, Theorem 5.2 shows that unless P = NP , no algorithms that run in polynomial time in each round can achieve poly( N ) · T 1 -c regret for any constant c > 0 , where poly( N ) stands for any polynomial of N , the per-round problem size. Our proof is based on the fact that maximizing the sum of three M ♮ -concave functions is at least as hard as the matroid intersection problem for three matroids, which is known to be NP -hard. 2 We carefully construct a concrete online M ♮ -concave function maximization instance that enables reduction from this NP -hard problem. Our high-level idea, namely, connecting sequential decision-making and finding a common base of three matroids, might be useful for proving hardness results in other online combinatorial optimization problems.

1.2 Related work

There is a large stream of research on no-regret submodular function maximization. Our stochastic bandit algorithms are inspired by a line of work on explore-then-commit algorithms for stochastic bandit problems [37, 38, 11] and by a robustness analysis for extending the offline greedy algorithm to the online setting [36]. However, unlike existing results for the submodular case, the guarantees of our algorithms in Section 4 involve no approximation factors. Moreover, while robustness properties similar to Theorem 3.1 are widely recognized in the submodular case, our proof for the M ♮ -concave case substantially differs from them. See Appendix A for a detailed discussion.

Combinatorial bandits with linear reward functions have been widely studied [6, 9, 8, 41], and many studies have also considered non-linear functions [7, 21, 15, 29]. However, the case of M ♮ -concave functions has not been well studied. Zhang et al. [53] studied stochastic minimization of L ♮ -convex functions, which form another important class in discrete convex analysis [32] but fundamentally differ from M ♮ -convex functions. Apart from online learning, a body of work has studied maximizing valuation functions approximately from samples to do with imperfect information [2, 3, 4].

Regarding hardness results in online learning, most arguments are typically information-theoretic. For instance, the minimax regret of hopeless games in partial monitoring is Ω( T ) [24, Section 37.2]. By contrast, we establish the NP -hardness of the adversarial full-information online M ♮ -concave function maximization, even though the offline M ♮ -concave function maximization is solvable in polynomial time. Such a situation is rare in online learning. One exception is the case studied by Bampis et al. [5]. They showed that no polynomial-time algorithm can achieve sub-linear approximate regret for some online min-max discrete optimization problems unless NP = RP , even though their offline counterparts are solvable in polynomial time. Despite the similarity in the situations, the problem class and proof techniques are completely different. Indeed, while their proof is based on the NP -hardness of determining the minimum size of a feasible solution, it can be done in polynomial time for M ♮ -concave function maximization [47, Corollary 4.2]. They also proved the NP -hardness of

the multi-instance setting, which is similar to the maximization of the sum of M ♮ -concave functions. However, they did not relate the hardness of multi-instance problems to that of no-regret learning.

2 Preliminaries

Let V = { 1 , . . . , N } be a ground set of size N . Let 0 be the all-zero vector. For i ∈ V , let e i ∈ R V denote the i th standard vector, i.e., the i th element is 1 and the others are 0 ; let e 0 = 0 for convenience. For x ∈ R V and S ⊆ V , let x ( S ) = ∑ i ∈ S x i . Slightly abusing notation, let x ( i ) = x ( { i } ) = x i . For a function f : Z V → R ∪ {-∞} on the integer lattice Z V , its effective domain is defined as dom f := { x ∈ Z V : f ( x ) > -∞} . A function f is called proper if dom f = ∅ . We say a proper function f : Z V → R ∪{-∞} is M ♮ -concave if for every x, y ∈ dom f and i ∈ V with x ( i ) > y ( i ) , there exists j ∈ V ∪ { 0 } with x ( j ) < y ( j ) or j = 0 such that the following inequality holds:

̸

Similarly, we say f : Z V → R ∪ { + ∞} is M ♮ -convex if -f is M ♮ -concave. If x ( V ) ≤ y ( V ) , M ♮ -concave functions satisfy more detailed conditions, as follows.

Proposition 2.1 (Corollary of Murota and Shioura [33, Theorem 4.2]) . Let f : Z V → R ∪{-∞} be an M ♮ -concave function. Then, the following conditions hold for every x, y ∈ dom f : (a) if x ( V ) < y ( V ) , ∃ j ∈ V with x ( j ) < y ( j ) , f ( x ) + f ( y ) ≤ f ( x + e j ) + f ( y -e j ) holds. (b) if x ( V ) ≤ y ( V ) , ∀ i ∈ V with x ( i ) > y ( i ) , ∃ j ∈ V with x ( j ) < y ( j ) , (1) holds.

̸

Let [ a, b ] = { x ∈ Z V : a ( i ) ≤ x ( i ) ≤ b ( i ) } be an integer interval of a, b ∈ ( Z ∪ {±∞} ) V and f be M ♮ -concave. If dom f ∩ [ a, b ] = ∅ , restricting dom f to [ a, b ] preserves the M ♮ -concavity [32, Proposition 6.14]. The sum of M ♮ -concave functions is not necessarily M ♮ -concave [32, Note 6.16]. In this paper, we do not assume monotonicity, i.e., x ≤ y (element-wise) does not imply f ( x ) ≤ f ( y ) .

2.1 Examples of M ♮ -concave functions

Maximum-flow on bipartite graphs. Let ( V, W ; E ) be a bipartite graph, where the set V of N left-hand-side vertices is a ground set. Each edge ij ∈ E is associated with a weight w ij ∈ R . Given sources x ∈ Z V ≥ 0 allocated to the vertices in V , let f ( x ) be the maximum-flow value, i.e.,

This function f is M ♮ -concave; indeed, more general functions specified by convex-cost flow problems on networks are M ♮ -concave [32, Theorem 9.27]. If we restrict the domain to { 0 , 1 } V and regard V as a set of items, W as a set of agents, and w ij ≥ 0 as the utility of matching an item i with an agent j , the resulting set function f : { 0 , 1 } V → R ≥ 0 coincides with the OXS valuation function known in combinatorial auctions [45, 25], which is a special case of the following gross substitute valuation.

Gross substitute valuation. In economics, an agent's valuation (a non-negative monotone set function of items) is said to be gross substitute (GS) if, whenever the prices of some items increase while the prices of the other items remain the same, the agent keeps demanding the same-priced items that were demanded before the price change [19, 25]. M ♮ -concave functions can be viewed as an extension of GS valuations to the integer lattice [32, Section 6.8]. Indeed, the class of M ♮ -concave functions restricted to { 0 , 1 } V is equivalent to the class of GS valuations [13].

Resource allocation. M ♮ -concave functions also arise in resource allocation problems [46, 30], which are extensively studied in the operations research community. For example, given n univariate concave functions f i : Z → R ∪ {-∞} and a positive integer K , a function f defined by f ( x ) = ∑ n i =1 f i ( x ( i )) if x ≥ 0 and x ( V ) ≤ K and f ( x ) = -∞ otherwise is M ♮ -concave. More general examples of M ♮ -concave functions used in resource allocation are given in, e.g., Moriguchi et al. [30].

More examples can be found in Murota and Shioura [33, Section 2] and Murota [32, Section 6.3]. As shown above, M ♮ -concave functions are ubiquitous in various fields. However, those are often difficult to know perfectly in advance: we may neither know all edge weights in maximum-flow problems, exact valuations of agents, nor f i s' values at all points in resource allocation. Such situations motivate us to study how to maximize them interactively by selecting solutions and observing some feedback.

Algorithm 1 Greedy-style procedure with possibly erroneous local updates

1: x 0 = 0

2: for k = 1 , . . . , K :

3: Select i k ∈ V ∪ { 0 }

4: x k ← x k -1 + e i k

2.2 Basic setup

Similar to bandit convex optimization [23], we consider a learner who interacts with a sequence of M ♮ -concave functions, f 1 , . . . , f T , over T rounds. To avoid incurring f t ( x ) = -∞ , we assume that dom f 2 , . . . , dom f T are identical to dom f 1 . We also assume 0 ∈ dom f 1 and dom f 1 ⊆ Z V ≥ 0 , which are reasonable in all the examples in Section 2.1. We consider a constrained setting where the learner's action x ∈ dom f 1 must satisfy x ( V ) ≤ K . If dom f 1 ⊆ { 0 , 1 } V , this is equivalent to the cardinality constraint common in set function maximization. Let X := { x ∈ dom f 1 : x ( V ) ≤ K } denote the set of feasible actions, which the learner is told in advance. (More precisely, a poly( N ) -time membership oracle of X is given.) Additional problem settings specific to stochastic bandit and adversarial full-information cases are provided in Sections 4 and 5, respectively.

3 Robustness of greedy M ♮ -concave function maximization to local errors

This section studies a greedy-style procedure with possibly erroneous local updates for M ♮ -concave function maximization, which will be useful for developing stochastic bandit algorithms in Section 4. Let f : Z V → R ∪ {-∞} be an M ♮ -concave function such that 0 ∈ dom f ⊆ Z V ≥ 0 , which we want to maximize under x ( V ) ≤ K . Let x ∗ ∈ arg max { f ( x ) : x ∈ dom f, x ( V ) ≤ K } be an optimal solution. We consider the procedure in Algorithm 1. If f is known a priori and i 1 , . . . , i K are selected as in the comment in Step 3, it coincides with the standard greedy algorithm for M ♮ -concave function maximization and returns an optimal solution [33]. However, when f is unknown, we may select different i 1 , . . . , i K than those selected by the exact greedy algorithm. Given any x ∈ Z V and update direction i ∈ V ∪ { 0 } , we define the local error of i at x as

which quantifies how much direction i deviates from the choice of the exact greedy algorithm when x is given. The following result states that local errors affect the eventual suboptimality only additively, ensuring that Algorithm 1 applied to M ♮ -concave function maximization is robust to local errors.

Proof. The claim is vacuously true if err( i k | x k -1 ) = + ∞ occurs for some k ≤ K . Below, we focus on the case with finite local errors. For k = 0 , 1 , . . . , K , we define

where y ≥ x k is read element-wise. That is, Y k ⊆ X consists of feasible points that can be reached from x k by the remaining K -k updates (see Figure 1). Note that x ∗ ∈ Y 0 and Y K = { x K } hold.

To prove the theorem, we will show that the following inequality holds for any k ∈ { 1 , . . . , K } :

Take y k -1 ∈ arg max y ∈Y k -1 f ( y ) and y k ∈ arg max y ∈Y k f ( y ) . If f ( y k ) ≥ f ( y k -1 ) , we are done since err( i k | x k -1 ) ≥ 0 . Thus, we assume f ( y k ) < f ( y k -1 ) , which implies y k -1 ∈ Y k -1 \ Y k . Then, we can prove the following helper claim by using the M ♮ -concavity of f .

Assuming the helper claim, we can easily obtain (3). Specifically, (i) f ( y k -1 + e i k -e j ) ≤ f ( y k ) holds due to y k -1 + e i k -e j ∈ Y k and the choice of y k , and (ii) err( i k | x k -1 ) ≥ f ( x k -1 + e j ) -

▷ Standard greedy selects i k ∈ arg max i ∈ V ∪{ 0 } f ( x k -1 + e i ) .

i

1

i

2

x

k

-

1

Y

k

-

1

i

1

i

2

x

k

Y

k

i

1

i

2

Y

k

x

k

f ( x k -1 + e i k ) = f ( x k -e i k + e j ) -f ( x k ) holds due to the definition of the local error (2). Combining these with (4) and rearranging terms imply (3) as follows:

Given (3), the theorem follows from a simple induction on k = 1 , . . . , K . For each k , we will prove

The case of k = 1 follows from (3) since x ∗ ∈ Y 0 . If it is true for k -1 , (3) and the induction hypothesis imply max y ∈Y k f ( y ) ≥ max y ∈Y k -1 f ( y ) -err( i k | x k -1 ) ≥ f ( x ∗ ) -∑ k -1 k ′ =1 err( i k ′ | x k ′ -1 ) -err( i k | x k -1 ) , thus obtaining (5). Since Y K = { x K } holds, setting k = K in (5) yields Theorem 3.1.

The rest of the proof is dedicated to proving the helper claim, which we do by examining the following three cases. The middle (right) image in Figure 1 illustrates case 1 (cases 2 and 3).

Case 1: i k = 0 . Due to x k -1 = x k , y k -1 ∈ Y k -1 \Y k implies y k -1 ( V ) = K -( k -1)+ x k -1 ( V ) . Thus, x k ( V ) = x k -1 ( V ) = y k -1 ( V ) -( K -k +1) < y k -1 ( V ) holds. From Proposition 2.1 (a), there exists j ∈ V with x k ( j ) < y k -1 ( j ) that satisfies (4). Also, y k -1 ≥ x k -1 = x k , x k ( j ) < y k -1 ( j ) , and ( y k -1 -e j )( V ) = K -k + x k -1 ( V ) = K -k + x k ( V ) imply y k -1 -e j ∈ Y k .

̸

Case 2: i k = 0 and x k ( V ) ≤ y k -1 ( V ) . In this case, y k -1 ∈ Y k -1 \ Y k implies y k -1 ≥ x k -1 and y k -1 ≱ x k = x k -1 + e i k , hence x k ( i k ) > y k -1 ( i k ) . From Proposition 2.1 (b), there exists j ∈ V with x k ( j ) < y k -1 ( j ) that satisfies (4). Since y k -1 ≥ x k -1 = x k -e i k and x k ( j ) < y k -1 ( j ) , we have y k -1 + e i k -e j ≥ x k . Also, we have ( y k -1 + e i k -e j )( V ) = y k -1 ( V ) ≤ K -( k -1) + x k -1 ( V ) = K -k + x k ( V ) . Therefore, we have y k -1 + e i k -e j ∈ Y k .

̸

Case 3: i k = 0 and x k ( V ) > y k -1 ( V ) . Since y k -1 ∈ Y k -1 , we have y k -1 ≥ x k -1 . We also have y k -1 ( V ) < x k ( V ) = x k -1 ( V ) + 1 . These imply y k -1 = x k -1 . Therefore, (4) with j = 0 holds by equality, and y k -1 + e i k = x k ∈ Y k also holds.

The robustness property in Theorem 3.1 plays a crucial role in developing stochastic bandit algorithms in Section 4. Furthermore, the robustness would be beneficial beyond the application to stochastic bandits since M ♮ -concave functions often involve uncertainty in practice, as discussed in Section 2.1. Note that Theorem 3.1 has not been known even in the field of discrete convex analysis and that the above proof substantially differs from the original proof for the greedy algorithm without local errors for M ♮ -concave function maximization [33]. Indeed, the original proof does not consider a set like Y k , which is crucial in our proof. It is also worth noting that Theorem 3.1 automatically implies the original result on the errorless case by setting err( i k | x k -1 ) = 0 for all k . We also emphasize that while Theorem 3.1 resembles robustness properties known in the submodular case [49, 14, 36, 37, 38, 11], ours is different from them in that it involves no approximation factors and requires careful inspection of the solution space, as in Figure 1. See Appendix A for a detailed discussion.

4 Stochastic bandit algorithms

This section presents no-regret algorithms for the following stochastic bandit setting.

Problem setting. For t = 1 , . . . , T , the learner selects x t ∈ X and observes f t ( x t ) = f ∗ ( x t ) + ε t , where f ∗ : Z V → [0 , 1] ∪ {-∞} is an unknown M ♮ -concave function and ( ε t ) T t =1 is a sequence of i.i.d. 1 -sub-Gaussian noises, i.e., E [exp( λε t )] ≤ exp( λ 2 / 2) for λ ∈ R . 3 Let x ∗ ∈ arg max x ∈X f ∗ ( x ) denote the offline best action. In Theorem 4.2, we will discuss the simple-regret minimization setting, where the learner selects x T +1 ∈ X after the T th round to minimize the expected simple regret:

Here, the expectation is about the learner's randomness, which may originate from noisy observations and possible randomness in their strategy. This is a stochastic bandit optimization setting, where the learner aims to find the best action without caring about the cumulative regret over the T rounds. On the other hand, Theorem 4.3 is about the standard regret minimization setting, where the learner aims to minimize the expected cumulative regret (or the pseudo-regret, strictly speaking):

In this section, we assume that T is large enough to satisfy T ≥ K ( N +2) to simplify the discussion.

Pure-exploration algorithm. Below, we will use a UCB-type algorithm for pure exploration in the standard stochastic multi-armed bandit problem as a building block. The algorithm is based on MOSS (Minimax Optimal Strategy in the Stochastic case) and is known to achieve an O ( T -1 / 2 ) simple regret as follows. For completeness, we provide the proof and the pseudo-code in Appendix B.

Proposition 4.1 (Lattimore and Szepesvári [24, Corollary 33.3]) . Consider a stochastic multi-armed bandit instance with M arms and T ′ rounds, where T ′ ≥ M . Assume that the reward of the i th arm in the t th round, denoted by Y t i , satisfies the following conditions: µ i := E [ Y t i ] ∈ [0 , 1] and Y t i -µ i is 1 -sub-Gaussian. Then, there is an algorithm that, after pulling arms T ′ times, randomly returns i ∈ { 1 , . . . , M } with µ ∗ -E [ µ i ] = O ( √ M/T ′ ) , where µ ∗ := max { µ 1 , . . . , µ M } .

Given this fact and our robustness result in Theorem 3.1, it is not difficult to develop an O ( T -1 / 2 ) -simple regret algorithm; we select i k in Algorithm 1 with the algorithm in Proposition 4.1 consuming ⌊ T/K ⌋ rounds and bound the simple regret by using Theorem 3.1, as detailed below. Also, given the O ( T -1 / 2 ) -simple regret algorithm, an O ( T 2 / 3 ) -regret algorithm follows from the explore-thencommit technique, as described subsequently. Therefore, we think of these no-regret algorithms as byproducts and the robustness result in Theorem 3.1 as our main technical contribution on the positive side. Nevertheless, we believe those algorithms are beneficial since no-regret maximization of M ♮ -concave functions have not been well studied, despite their ubiquity as discussed in Section 2.1.

The following theorem presents our result regarding an O ( T -1 / 2 ) -simple regret algorithm.

Theorem 4.2. There is an algorithm that makes up to T queries to the noisy value oracle of f ∗ and outputs x T +1 such that sReg T = O ( K 3 / 2 √ N/T ) .

Proof. Based on Algorithm 1, we consider a randomized algorithm consisting of K phases. Fixing a realization of x k -1 , we discuss the k th phase. We consider the following multi-armed bandit instance with at most N +1 arms and ⌊ T/K ⌋ rounds. The arm set is { i ∈ V ∪ { 0 } : x k -1 + e i ∈ X } , i.e., the collection of all feasible update directions; note that the learner can construct this arm set since X is told in advance. In each round t ∈ { 1 , . . . , ⌊ T/K ⌋} , the reward of an arm i ∈ V ∪ { 0 } is given by Y t i = f ∗ ( x k -1 + e i ) + ε t , where ε t is the 1 -sub-Gaussian noise. Let µ i = E [ Y t i ] = f ∗ ( x k -1 + e i ) ∈ [0 , 1] denote the mean reward of the arm i and µ ∗ = max i ∈ V ∪{ 0 } µ i the optimal mean reward. If we apply the algorithm in Proposition 4.1 to this bandit instance, it randomly returns i k ∈ V ∪{ 0 } such that E [ err( i k | x k -1 ) | x k -1 ] = µ ∗ -E [ µ i k | x k -1 ] = O ( √ KN/T ) , consuming ⌊ T/K ⌋ queries.

Consider sequentially selecting i k as above and setting x k = x k -1 + e i k , thus obtaining x 1 , . . . , x K . For any realization of i 1 , . . . , i K , Theorem 3.1 guarantees f ∗ ( x K ) ≥ f ∗ ( x ∗ ) -∑ K k =1 err( i k | x k -1 ) . By taking the expectations of both sides and using the law of total expectation, we obtain

Thus, x T +1 = x K achieves the desired bound. The number of total queries is K ⌊ T/K ⌋ ≤ T .

We then convert the O ( T -1 / 2 ) -simple regret algorithm into an O ( T 2 / 3 ) -regret algorithm by using the explore-then-commit technique as follows.

Theorem 4.3. There is an algorithm that achieves Reg T = O ( KN 1 / 3 T 2 / 3 ) .

Proof. Let ˜ T ≤ T be the number of exploration rounds, which we will tune later. If we use the algorithm of Theorem 4.2 allowing ˜ T queries, we can find x ˜ T +1 ∈ X with sReg ˜ T = O ( K 3 / 2 √ N/ ˜ T ) . If we commit to x ˜ T +1 in the remaining T -˜ T rounds, the total expected regret is

5 NP-hardness of adversarial full-information setting

This section discusses the NP -hardness of the following adversarial full-information setting.

Problem setting. An oblivious adversary chooses an arbitrary sequence of M ♮ -concave functions, f 1 , . . . , f T , where f t : Z V → [0 , 1] ∪ {-∞} for t = 1 , . . . , T , in secret from the learner. Then, for t = 1 , . . . , T , the learner selects x t ∈ X and observes f t , i.e., full-information feedback. More precisely, we suppose that the learner gets free access to a poly( N ) -time value oracle of f t by observing f t since M ♮ -concave functions may not have polynomial-size representations in general. The learner aims to minimize the expected cumulative regret:

where the expectation is about the learner's randomness. To simplify the subsequent discussion, we focus on the case where the constraint is specified by K = N and f 1 , . . . , f T are defined on { 0 , 1 } V ; therefore, the set of feasible actions is X = { x ∈ dom f 1 : x ( V ) ≤ K } = { 0 , 1 } V .

For this setting, there is a simple no-regret algorithm that takes exponential time per round. Specifically, regarding each x ∈ X as an expert, we use the standard multiplicative weight update algorithm to select x 1 , . . . , x T [26, 12]. Since the number of experts is |X| = 2 N , this attains an expected regret bound of O ( √ T log |X| ) ≲ poly( N ) √ T , while taking prohibitively long poly( N ) |X| ≳ 2 N time per round. An interesting question is whether a similar regret bound is achievable in polynomial time per round. Thus, we focus on the polynomial-time randomized learner , as with Bampis et al. [5].

Definition 5.1 (Polynomial-time randomized learner) . Wesay an algorithm for computing x 1 , . . . , x T is a polynomial-time randomized learner if, given poly( N ) -time value oracles of revealed functions, it runs in poly( N,T ) time in each round, regardless of realizations of the algorithm's randomness. 4

Note that the per-round time complexity can depend polynomially on T . Thus, the algorithm can use past actions, x 1 , . . . , x t -1 , as inputs for computing x t , as long as the per-round time complexity is polynomial in the input size. The following theorem is our main result on the negative side.

Theorem 5.2. In the adversarial full-information setting, for any constant c > 0 , no polynomial-time randomized learner can achieve a regret bound of poly( N ) · T 1 -c in expectation unless P = NP . 5

5.1 Proof of Theorem 5.2

As preparation for proving the theorem, we first show that it suffices to prove the hardness for any polynomial-time deterministic learner and some distribution on input sequences of functions, which follows from celebrated Yao's principle [52]. We include the proof in Appendix C for completeness.

Proposition 5.3 (Yao [52]) . Let A be a finite set of all possible deterministic learning algorithms that run in polynomial time per round and F 1: T a finite set of sequences of M ♮ -concave functions, f 1 , . . . , f T . Let Reg T ( a, f 1: T ) be the cumulative regret a deterministic learner a ∈ A achieves on a sequence f 1: T = ( f 1 , . . . , f T ) ∈ F 1: T . Then, for any polynomial-time randomized learner A and any distribution q on F 1: T , it holds that

Note that the left-hand side is nothing but the expected cumulative regret (6) of a polynomial-time randomized learner A on the worst-case input f 1: T . Thus, to prove the theorem, it suffices to show that the right-hand side, i.e., the expected regret of the best polynomial-time deterministic learner on some input distribution q , cannot be as small as poly( N ) · T 1 -c unless P = NP . To this end, we will construct a finite set F 1: T of sequences of M ♮ -concave functions and a distribution on it.

The fundamental idea behind the subsequent construction of M ♮ -concave functions is that the matroid intersection problem for three matroids (the 3-matroid intersection problem, for short) is NP -hard.

3-matroid intersection problem. A matroid M over V is defined by a non-empty set family B ⊆ 2 V such that for B 1 , B 2 ∈ B and i ∈ B 1 \ B 2 , there exists j ∈ B 2 \ B 1 such that B 1 \{ i }∪{ j } ∈ B . Elements in B are called bases . We suppose that, given a matroid, we can test whether a given S ⊆ V is a base in poly( N ) time. (This is equivalent to the standard poly( N ) -time independence testing.) The 3-matroid intersection problem asks to determine whether three given matroids M 1 , M 2 , M 3 over a common ground set V have a common base B ∈ B 1 ∩ B 2 ∩ B 3 or not.

Proposition 5.4 (cf. Schrijver [44, Chapter 41]) . The 3-matroid intersection problem is NP -hard.

We construct M ♮ -concave functions that appropriately encode the 3-matroid intersection problem. Below, for any B ⊆ V , let 1 B ∈ { 0 , 1 } V denote a vector such that 1 B ( i ) = 1 if and only if i ∈ B .

Lemma 5.5. Let M be a matroid over V and B ⊆ 2 V its base family. There is a function f : { 0 , 1 } V → [0 , 1] such that (i) f ( x ) = 1 if and only if x = 1 B for some B ∈ B and f ( x ) ≤ 1 -1 /N otherwise, (ii) f is M ♮ -concave, and (iii) f ( x ) can be computed in poly( N ) time at every x ∈ { 0 , 1 } V .

Proof. Let ∥·∥ 1 denote the ℓ 1 -norm. We construct the function f : { 0 , 1 } V → [0 , 1] as follows:

Since 0 ≤ ∥ y ∥ 1 ≤ N for y ∈ [ -1 , +1] V , f ( x ) takes values in [0 , 1] . Moreover, min B ∈B ∥ x -1 B ∥ 1 is zero if x = 1 B for some B ∈ B and at least 1 otherwise, establishing (i). Below, we show that f is (ii) M ♮ -concave and (iii) computable in poly( N ) time.

We prove that τ ( x ) := min B ∈B ∥ x -1 B ∥ 1 is M ♮ -convex, which implies the M ♮ -concavity of f . Let δ B : Z V →{ 0 , + ∞} be the indicator function of B , i.e., δ B ( x ) = 0 if x = 1 B for some B ∈ B and + ∞ otherwise. Observe that τ is the integer infimal convolution of ∥·∥ 1 and δ B . Here, the integer infimal convolution of two functions f 1 , f 2 : Z V → R ∪{ + ∞} is a function of x ∈ Z V defined as ( f 1 □ Z f 2 )( x ) := min { f 1 ( x -y ) + f 2 ( y ) : y ∈ Z V } , and the M ♮ -concavity is preserved under this operation [32, Theorem 6.15]. Thus, the M ♮ -convexity of τ ( x ) = ( ∥·∥ 1 □ Z δ B )( x ) follows from the M ♮ -convexity of the ℓ 1 -norm ∥·∥ 1 [32, Section 6.3] and the indicator function δ B [32, Section 4.1].

Next, we show that τ ( x ) is computable in poly( N ) time for every x ∈ { 0 , 1 } V , which implies the poly( N ) -time computability of f ( x ) . As described above, τ is the integer infimal convolution of

∥·∥ 1 and δ B , i.e., τ ( x ) = min {∥ x -y ∥ 1 + δ B ( y ) : y ∈ Z V } . Since the function y ↦→∥ x -y ∥ 1 is M ♮ -convex [32, Theorem 6.15], τ ( x ) is the minimum value of the sum of the two M ♮ -convex functions. While the sum of two M ♮ -convex functions f 1 , f 2 : Z V → Z ≥ 0 ∪{ + ∞} is no longer M ♮ -convex in general, we can minimize it via reduction to the M-convex submodular flow problem [32, Note 9.30]. Wecan solve this by querying f 1 and f 2 values poly( N, log L, log M ) times, where L is the minimum of the ℓ ∞ -diameter of dom f 1 and dom f 2 and M is an upper bound on function values [18, 17]. In our case of f 1 ( y ) = ∥ x -y ∥ 1 and f 2 ( y ) = δ B ( y ) , we have L = 1 and M ≤ N , and we can compute f 1 ( y ) and f 2 ( y ) values in poly( N ) time (where the latter is due to the poly( N ) -time membership testing for B ). Therefore, τ ( x ) is computable in poly( N ) time, and so is f ( x ) .

Now, we are ready to prove Theorem 5.2.

Proof of Theorem 5.2. Let M 1 , M 2 , M 3 be three matroids over V and f 1 , f 2 , f 3 functions defined as in Lemma 5.5, respectively. Let F 1: T be a finite set such that each f t ( t = 1 , . . . , T ) is either f 1 , f 2 , or f 3 . Let q be a distribution on F 1: T that sets each f t to f 1 , f 2 , or f 3 with equal probability.

Suppose for contradiction that some polynomial-time deterministic learner achieves poly( N ) · T 1 -c regret in expectation for the above distribution q . Let T be the smallest integer such that the regret bound satisfies poly( N ) · T 1 -c < T 3 N ⇔ T > (3 N poly( N )) 1 /c . Note that T is polynomial in N since c > 0 is a constant. We consider the following procedure.

Run the polynomial-time deterministic learner on the distribution q and obtain x t for t = 1 , . . . , T . If some x t satisfies f 1 ( x t ) = f 2 ( x t ) = f 3 ( x t ) = 1 , output 'Yes' and otherwise 'No.'

̸

If M 1 , M 2 , M 3 have a common base B ∈ B 1 ∩B 2 ∩B 3 , we have f 1 ( 1 B ) = f 2 ( 1 B ) = f 3 ( 1 B ) = 1 . On the other hand, if x t = 1 B for every B ∈ B 1 ∩B 2 ∩B 3 , E [ f t ( x t )] ≤ 1 -1 3 N holds from Lemma 5.5 and the fact that f t is drawn uniformly from { f 1 , f 2 , f 3 } . Thus, to achieve the poly( N ) · T 1 -c regret for T > (3 N poly( N )) 1 /c , the learner must return x t corresponding to some common base at least once among T rounds. Consequently, the above procedure outputs 'Yes.' If M 1 , M 2 , M 3 have no common base, none of x 1 , . . . , x T can be a common base, and hence the procedure outputs 'No.' Therefore, the above procedure returns a correct answer to the 3-matroid intersection problem. Recall that T is polynomial in N . Since the learner runs in T · poly( N,T ) time and we can check f 1 ( x t ) = f 2 ( x t ) = f 3 ( x t ) = 1 for t = 1 , . . . , T in T · poly( N ) time, the procedure runs in poly( N ) time. This contradicts the NP -hardness of the 3-matroid intersection problem (Proposition 5.4) unless P = NP . Therefore, no polynomial-time deterministic learner can achieve poly( N ) · T 1 -c regret in expectation. Finally, this regret lower bound applies to any polynomial-time randomized learner on the worst-case input due to Yao's principle (Proposition 5.3), completing the proof.

Remark 5.6. One might think that the hardness simply follows from the fact that no-regret learning in terms of (6) is too demanding. However, similar criteria are naturally met in other problems: there are efficient no-regret algorithms for online convex optimization and no-approximate-regret algorithms for online submodular function maximization. What makes online M ♮ -concave function maximization NP -hard is its connection to the 3-matroid intersection problem, as detailed in the proof. Consequently, even though offline M ♮ -concave function maximization is solvable in polynomial time, no polynomial-time randomized learner can achieve vanishing regret in the adversarial online setting.

6 Conclusion and discussion

This paper has studied no-regret M ♮ -concave function maximization. For the stochastic bandit setting, we have developed O ( K 3 / 2 √ N/T ) -simple regret and O ( KN 1 / 3 T 2 / 3 ) -regret algorithms. A crucial ingredient is the robustness of the greedy algorithm to local errors, which we have first established for the M ♮ -concave case. For the adversarial full-information setting, we have proved the NP -hardness of no-regret learning through a reduction from the 3-matroid intersection problem.

Our stochastic bandit algorithms are limited to the sub-Gaussian noise model, while our hardness result for the adversarial setting comes from a somewhat pessimistic analysis. Overcoming these limitations by exploring intermediate regimes between the two settings, such as stochastic bandits with adversarial corruptions [28], will be an exciting future direction from the perspective of beyond the worst-case analysis [42]. We also expect that our stochastic bandit algorithms have room for

improvement, considering existing regret lower bounds for stochastic combinatorial (semi-)bandits with linear reward functions. For topK combinatorial bandits, there is a sample-complexity lower bound of Ω( N/ε 2 ) for any ( ε, δ ) -PAC algorithm [41]. Since our O ( K 3 / 2 √ N/T ) -simple regret bound implies that we can achieve an ε -error in expectation with O ( K 3 N/ε 2 ) samples, our bound seems tight when K = O (1) , while the K factors would be improvable. Regarding the cumulative regret bound, there is an Ω( √ KNT ) lower bound for stochastic combinatorial semi-bandits [22]. Filling the gap between our O ( KN 1 / 3 T 2 / 3 ) upper bound and the lower bound is an open problem. (Since we have assumed T = Ω( KN ) in Section 4, our upper bound does not contradict the lower bound.) We believe that our upper bound is essentially tight considering a recent minimax regret bound by Tajdini et al. [50] for bandit submodular maximization, which we discuss in detail below. Regarding the adversarial setting, it will be interesting to explore no-approximate-regret algorithms. If M ♮ -concave functions are restricted to { 0 , 1 } V , the resulting problem is a special case of online submodular function maximization and hence vanishing 1 / 2 -approximate regret is already possible [43, 16, 36]. We may be able to improve the approximation factor by using the M ♮ -concavity.

Discussion on the tightness of the O ( KN 1 / 3 T 2 / 3 ) bound. As mentioned above, obtaining a tight regret bound for stochastic bandit M ♮ -concave maximization is left open. Nevertheless, we conjecture that our O ( KN 1 / 3 T 2 / 3 ) bound in Theorem 4.3 is tight unless we admit exponential factors in K . The rationale behind this conjecture lies in a recent result by Tajdini et al. [50]. They studied stochastic bandit monotone submodular maximization with a ground set of size N and a cardinality constraint of K , and they showed that there is a lower bound of

on robust greedy regret , which compares the learner's actual reward with the output of the greedy algorithm, denoted by S gr , applied to the underlying true submodular function. Here, i ≤ K is the largest positive integer with 16 N 2 K 6 ( N -K i ) 3 ≤ T ; see Tajdini et al. [50, Theorem 2.3] for details. 6 This lower bound suggests that the O ( KN 1 / 3 T 2 / 3 ) regret for stochastic bandit submodular maximization, which can also be achieved by the explore-then-commit strategy, is inevitable in general. We can interpret the √ ( N -K i ) T term as the regret achieved by regrading all ( N -K i ) subsets as arms and using a UCB-type algorithm. Thus, the lower bound consists of the two regret terms achieved by the explore-then-commit and the UCB applied to exponentially many arms.

Currently, we have observed that the proof of the lower bound by Tajdini et al. [50] does not directly apply to our stochastic bandit M ♮ -concave maximization problem. Specifically, the function used in their proof for obtaining the lower bound is submodular but not M ♮ -concave. Nevertheless, the problem setting of Tajdini et al. [50] and our problem in Section 4, with the domain restricted to { 0 , 1 } V , have notable connections:

Since the greedy algorithm applied to the unknown true M ♮ -concave function f ∗ can find an optimal solution x ∗ , we have x ∗ = S gr . Therefore, the notion of robust greedy regret in Tajdini et al. [50] essentially coincides with the standard regret in our case.

Both the O ( KN 1 / 3 T 2 / 3 ) and O (√ ( N -K i ) T ) regret bounds discussed above can also be achieved by the explore-then-commit and UCB strategies, respectively, in our M ♮ -concave case, where the former is exactly what our Theorem 4.3 states.

Considering these facts, we expect that we can construct a hard instance of stochastic bandit M ♮ -concave maximization similar to Tajdini et al. [50] to establish the same regret lower bound. Therefore, we conjecture that our O ( KN 1 / 3 T 2 / 3 ) regret bound in Theorem 4.3 is tight in K , N , and T , if we want to avoid the exponential factor, which generally scales as N K , regardless of the value of T .

References

S. Arora and B. Barak. Computational Complexity: A Modern Approach . Cambridge University Press, 2009 (cited on page 7).

M. F. Balcan, F. Constantin, S. Iwata, and L. Wang. Learning valuation functions. In Proceedings of the 25th Annual Conference on Learning Theory , volume 23, pages 4.1-4.24. PMLR, 2012 (cited on page 2).

E. Balkanski, A. Rubinstein, and Y. Singer. The power of optimization from samples. In Advances in Neural Information Processing Systems , volume 29, pages 4017-4025. Curran Associates, Inc., 2016 (cited on page 2).

E. Balkanski, A. Rubinstein, and Y. Singer. The limitations of optimization from samples. Journal of the ACM , 69(3):1-33, 2022 (cited on page 2).

E. Bampis, D. Christou, B. Escoffier, and K. T. NGuyen. Online learning for min-max discrete problems. Theoretical Computer Science , 930:209-217, 2022 (cited on pages 2, 7).

N. Cesa-Bianchi and G. Lugosi. Combinatorial bandits. Journal of Computer and System Sciences , 78(5):1404-1422, 2012 (cited on page 2).

W. Chen, Y. Wang, and Y. Yuan. Combinatorial multi-armed bandit: general framework and applications. In Proceedings of the 30th International Conference on Machine Learning , volume 28, pages 151-159. PMLR, 2013 (cited on page 2).

A. Cohen, T. Hazan, and T. Koren. Tight bounds for bandit combinatorial optimization. In Proceedings of the 30th Annual Conference on Learning Theory , volume 65, pages 629-642. PMLR, 2017 (cited on page 2).

R. Combes, M. S. Talebi, A. Proutiere, and M. Lelarge. Combinatorial bandits revisited. In Advances in Neural Information Processing Systems , volume 28, pages 2116-2124. Curran Associates, Inc., 2015 (cited on page 2).

U. Feige. A threshold of ln n for approximating set cover. Journal of the ACM , 45(4):634-652, 1998 (cited on page 1).

F. Fourati, C. J. Quinn, M.-S. Alouini, and V. Aggarwal. Combinatorial stochastic-greedy bandit. Proceedings of the 38th AAAI Conference on Artificial Intelligence , 38(11):1205212060, 2024 (cited on pages 2, 5, 14).

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997 (cited on page 7).

S. Fujishige and Z. Yang. A note on Kelso and Crawford's gross substitutes condition. Mathematics of Operations Research , 28(3):463-469, 2003 (cited on pages 1, 3).

D. Golovin, A. Krause, and M. Streeter. Online submodular maximization under a matroid constraint with application to learning assignments. arXiv:1407.1082 , 2014 (cited on pages 2, 5, 14).

Y. Han, Y. Wang, and X. Chen. Adversarial combinatorial bandits with general non-linear reward functions. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 4030-4039. PMLR, 2021 (cited on page 2).

N. Harvey, C. Liaw, and T. Soma. Improved algorithms for online submodular maximization via first-order regret bounds. In Advances in Neural Information Processing Systems , volume 33, pages 123-133. Curran Associates, Inc., 2020 (cited on pages 2, 10).

S. Iwata, S. Moriguchi, and K. Murota. A capacity scaling algorithm for M-convex submodular flow. Mathematical Programming , 103(1):181-202, 2005 (cited on page 9).

S. Iwata and M. Shigeno. Conjugate scaling algorithm for Fenchel-type duality in discrete convex optimization. SIAM Journal on Optimization , 13(1):204-211, 2002 (cited on page 9).

A. S. Kelso and V. P. Crawford. Job matching, coalition formation, and gross substitutes. Econometrica , 50(6):1483-1504, 1982 (cited on pages 1, 3).

R. Kupfer, S. Qian, E. Balkanski, and Y. Singer. The adaptive complexity of maximizing a gross substitutes valuation. In Advances in Neural Information Processing Systems , volume 33, pages 19817-19827. Curran Associates, Inc., 2020 (cited on page 1).

B. Kveton, Z. Wen, A. Ashkan, and C. Szepesvari. Combinatorial cascading bandits. In Advances in Neural Information Processing Systems , volume 28, pages 1450-1458. Curran Associates, Inc., 2015 (cited on page 2).

B. Kveton, Z. Wen, A. Ashkan, and C. Szepesvari. Tight regret bounds for stochastic combinatorial semi-bandits. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics , volume 38, pages 535-543. PMLR, 2015 (cited on page 10).

T. Lattimore. Bandit convex optimisation. arXiv:2402.06535 , 2024 (cited on page 4).

T. Lattimore and C. Szepesvári. Bandit Algorithms . Cambridge University Press, 2020 (cited on pages 2, 6, 14, 15).

B. Lehmann, D. Lehmann, and N. Nisan. Combinatorial auctions with decreasing marginal utilities. Games and Economic Behavior , 55(2):270-296, 2006 (cited on pages 1, 3).

N. Littlestone and M. K. Warmuth. The weighted majority algorithm. Information and Computation , 108(2):212-261, 1994 (cited on page 7).

L. H. Loomis. On a theorem of von Neumann. Proceedings of the National Academy of Sciences of the United States of America , 32(8):213-215, 1946 (cited on page 15).

T. Lykouris, V. Mirrokni, and R. Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 114-122. ACM, 2018 (cited on page 9).

N. Merlis and S. Mannor. Tight lower bounds for combinatorial multi-armed bandits. In Proceedings of the 33rd Annual Conference on Learning Theory , volume 125, pages 28302857. PMLR, 2020 (cited on page 2).

S. Moriguchi, A. Shioura, and N. Tsuchimura. M-convex function minimization by continuous relaxation approach: proximity theorem and algorithm. SIAM Journal on Optimization , 21(3):633-668, 2011 (cited on pages 1, 3).

R. Motwani and P. Raghavan. Randomized Algorithms . Cambridge University Press, 1995 (cited on page 15).

K. Murota. Discrete Convex Analysis , volume 10. SIAM, 2003 (cited on pages 1-3, 8, 9).

K. Murota and A. Shioura. M-convex function on generalized polymatroid. Mathematics of Operations Research , 24(1):95-105, 1999 (cited on pages 1, 3-5, 14).

G. L. Nemhauser and L. A. Wolsey. Best algorithms for approximating the maximum of a submodular set function. Mathematics of Operations Research , 3(3):177-188, 1978 (cited on page 1).

G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functions-I. Mathematical Programming , 14(1):265-294, 1978 (cited on page 1).

R. Niazadeh, N. Golrezaei, J. R. Wang, F. Susan, and A. Badanidiyuru. Online learning via offline greedy algorithms: applications in market design and optimization. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 737-738. ACM, 2021 (cited on pages 2, 5, 10, 14).

G. Nie, M. Agarwal, A. K. Umrawal, V. Aggarwal, and C. John Quinn. An explore-thencommit algorithm for submodular maximization under full-bandit feedback. In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence , volume 180, pages 1541-1551. PMLR, 2022 (cited on pages 2, 5, 14).

G. Nie, Y. Y. Nadew, Y. Zhu, V. Aggarwal, and C. J. Quinn. A framework for adapting offline algorithms to solve combinatorial multi-armed bandit problems with bandit feedback. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 26166-26198. PMLR, 2023 (cited on pages 2, 5, 14).

T. Oki and S. Sakaue. Faster discrete convex function minimization with predictions: the Mconvex case. In Advances in Neural Information Processing Systems , volume 36, pages 6857668588. Curran Associates, Inc., 2023 (cited on page 1).

S. Pasteris, A. Rumi, F. Vitale, and N. Cesa-Bianchi. Sum-max submodular bandits. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics , volume 238, pages 2323-2331. PMLR, 2024 (cited on page 2).

I. Rejwan and Y. Mansour. Topk combinatorial bandits with full-bandit feedback. In Proceedings of the 31st International Conference on Algorithmic Learning Theory , volume 117, pages 752-776. PMLR, 2020 (cited on pages 2, 10).

T. Roughgarden. Beyond the Worst-Case Analysis of Algorithms . Cambridge University Press, 2021 (cited on page 9).

T. Roughgarden and J. R. Wang. An optimal learning algorithm for online unconstrained submodular maximization. In Proceedings of the 31st Annual Conference on Learning Theory , volume 75, pages 1307-1325. PMLR, 2018 (cited on pages 2, 10).

A. Schrijver. Combinatorial Optimization , volume 24. Springer, 2003 (cited on page 8).

L. S. Shapley. Complements and substitutes in the opttmal assignment problem. Naval Research Logistics Quarterly , 9(1):45-48, 1962 (cited on page 3).

A. Shioura. Fast scaling algorithms for M-convex function minimization with application to the resource allocation problem. Discrete Applied Mathematics , 134(1):303-316, 2004 (cited on pages 1, 3).

A. Shioura. M-convex function minimization under L1-distance constraint and its application to dock reallocation in bike-sharing system. Mathematics of Operations Research , 47(2):15661611, 2022 (cited on page 2).

T. Soma. Submodular and Sparse Optimization Methods for Machine Learning and Communication . PhD thesis, The University of Tokyo, 2016 (cited on page 1).

M. Streeter and D. Golovin. An online algorithm for maximizing submodular functions. In Advances in Neural Information Processing Systems , volume 21, pages 1577-1584. Curran Associates, Inc., 2008 (cited on pages 2, 5, 14).

A. Tajdini, L. Jain, and K. Jamieson. Nearly minimax optimal submodular maximization with bandit feedback. arXiv:2310.18465 , 2023. To appear in NeurIPS 2024. (cited on pages 10, 11).

Z. Wan, J. Zhang, W. Chen, X. Sun, and Z. Zhang. Bandit multi-linear DR-submodular maximization and its applications on adversarial submodular bandits. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 35491-35524. PMLR, 2023 (cited on page 2).

A. C.-C. Yao. Probabilistic computations: toward a unified measure of complexity. In Proceedings of the 18th Annual Symposium on Foundations of Computer Science , pages 222-227. IEEE, 1977 (cited on pages 8, 15).

H. Zhang, Z. Zheng, and J. Lavaei. Stochastic L ♮ -convex function minimization. In Advances in Neural Information Processing Systems , volume 34, pages 13004-13018. Curran Associates, Inc., 2021 (cited on page 2).

M. Zhang, L. Chen, H. Hassani, and A. Karbasi. Online continuous submodular maximization: from full-information to bandit feedback. In Advances in Neural Information Processing Systems , volume 32, pages 9210-9221. Curran Associates, Inc., 2019 (cited on page 2).

Q. Zhang, Z. Deng, Z. Chen, K. Zhou, H. Hu, and Y. Yang. Online learning for non-monotone DR-submodular maximization: from full information to bandit feedback. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , volume 206, pages 3515-3537. PMLR, 2023 (cited on page 2).

A Differences of Theorem 3.1 from robustness results in the submodular case

The basic idea of analyzing the robustness is inspired by similar approaches used in online submodular function maximization [49, 14, 36, 37, 38, 11]. However, our Theorem 3.1 for the M ♮ -concave case is fundamentally different from those for the submodular case.

At a high level, an evident difference lies in the comparator in the guarantees. Specifically, we need to bound the suboptimality compared to the optimal value in the M ♮ -concave case, while the comparator is an approximate value in the submodular case.

At a more technical level, we need to work on the solution space in the M ♮ -concave case, while the proof for the submodular case follows from analyzing objective values directly. Let us overview the standard technique for the case of monotone submodular function maximization under the cardinality constraint, which is the most relevant to our case due to the similarity in the algorithmic procedures. In this case, a key argument is that in each iteration, the marginal increase in the objective value is lower bounded by a 1 /K fraction of that gained by adding an optimal solution, minus the local error. That is, regarding f : { 0 , 1 } V → R as a submodular set function, the submodularity implies f ( x k ) -f ( x k -1 ) ≥ 1 K ( f ( x k -1 ∨ x ∗ ) -f ( x k -1 )) -err( i k | x k -1 ) , where ∨ is the element-wise maximum. Consequently, by rearranging terms in the same way as the proof of the ( 1 -1 / e )-approximation, one can confirm that local errors accumulate only additively over K iterations. In this way, the robustness property directly follows from incorporating the effect of local errors into the inequality for deriving the ( 1 -1 / e )-approximation in the submodular case. By contrast, in our proof of Theorem 3.1 for the M ♮ -concave case, we need to look at the solution space to ensure that the local update by i k with small err( i k | x k -1 ) does not deviate much from Y k -1 , as highlighted in (3) (and this also differs from the original proof without errors [33]). After establishing this, we can obtain the theorem by induction by virtue of the non-trivial design of Y k ( k = 0 , . . . , K ), which satisfies x ∗ ∈ Y 0 and Y K = { x K } .

B MOSS for pure exploration in stochastic multi-armed bandit

We overview the MOSS-based pure-exploration algorithm used in Section 4. For more details, see Lattimore and Szepesvári [24, Chapters 9 and 33].

Let I { A } take 1 if A is true and 0 otherwise, and let log + ( x ) = log max { 1 , x } . Given a stochastic multi-armed bandit instance with M arms and T ′ rounds, we consider an algorithm that randomly selects arms A 1 , . . . , A T ′ ∈ { 1 , . . . , M } . For t = 1 , . . . , T ′ , let Y t be a random variable representing the learner's reward in the t th round, ˆ τ i ( t ) = ∑ t s =1 I { A s = i } the number of times the i th arm is selected up to round t , and ˆ µ i ( t ) = 1 ˆ τ i ( t ) ∑ t s =1 I { A s = i } Y s the empirical mean reward of the i th arm up to round t . Given these definitions, the MOSS algorithm can be described as in Algorithm 2.

Algorithm 2 MOSS

Input: Bandit instance with M arms and T rounds 1: Choose each arm i ∈ { 1 , . . . , M } during the first M rounds 2: for t = M +1 , . . . , T ′ : √ )

′

Let a 1 , . . . , a T ′ denote the realization of A 1 , . . . , A T ′ , respectively, after running the MOSS algorithm. Then, we set the final output to i ∈ { 1 , . . . , M } with probability 1 T ′ ∑ T ′ t =1 I { a t = i } . This procedure gives an O ( √ M/T ′ ) -simple regret algorithm, as stated in Proposition 4.1.

Proposition 4.1 (Lattimore and Szepesvári [24, Corollary 33.3]) . Consider a stochastic multi-armed bandit instance with M arms and T ′ rounds, where T ′ ≥ M . Assume that the reward of the i th arm in the t th round, denoted by Y t i , satisfies the following conditions: µ i := E [ Y t i ] ∈ [0 , 1] and Y t i -µ i is 1 -sub-Gaussian. Then, there is an algorithm that, after pulling arms T ′ times, randomly returns i ∈ { 1 , . . . , M } with µ ∗ -E [ µ i ] = O ( √ M/T ′ ) , where µ ∗ := max { µ 1 , . . . , µ M } .

Proof. Since the suboptimality of the i th arm, defined by µ ∗ -µ i , is at most 1 for all i ∈ { 1 , . . . , M } , the MOSS algorithm enjoys a cumulative regret bound of Reg T ′ := T ′ · µ ∗ -E [ ∑ T ′ t =1 µ A t ] ≤ 39 √ MT ′ + M (see Lattimore and Szepesvári [24, Theorem 9.1]). Consider setting the final output to i ∈ { 1 , . . . , M } with probability 1 T ′ ∑ T ′ t =1 I { a t = i } , where a t denote the realization of A t . Then, it holds that µ ∗ -E i ∼ p [ µ i ] = Reg T ′ /T ′ (see Lattimore and Szepesvári [24, Proposition 33.2]). The right-hand side is at most (39 √ MT ′ + M ) /T ′ ≤ 40 √ M/T ′ , completing the proof.

C Proof of Proposition 5.3

Proposition 5.3 (Yao [52]) . Let A be a finite set of all possible deterministic learning algorithms that run in polynomial time per round and F 1: T a finite set of sequences of M ♮ -concave functions, f 1 , . . . , f T . Let Reg T ( a, f 1: T ) be the cumulative regret a deterministic learner a ∈ A achieves on a sequence f 1: T = ( f 1 , . . . , f T ) ∈ F 1: T . Then, for any polynomial-time randomized learner A and any distribution q on F 1: T , it holds that

Proof. We use the same proof idea as that of Yao's principle (see, e.g., Motwani and Raghavan [31, Section 2.2]). First, note that any polynomial-time randomized learner can be viewed as a polynomialtime deterministic learner with access to a random tape. Thus, we can take A to be chosen according to some distribution p on the family, A , of all possible polynomial-time deterministic learners.

Consider an |A| × |F 1: T | matrix M , whose entry corresponding to row a ∈ A and column f 1: T ∈ F 1: T is Reg T ( a, f 1: T ) . For any polynomial-time randomized learner A and any distribution q on F 1: T , it holds that

where p ′ and q ′ denote probability vectors on A and F 1: T , respectively, and e f 1: T and e a denote the standard unit vectors of f 1: T and a , respectively. The equality is due to Loomis' theorem [27].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clarified our scope (no-regret M ♮ -concave function maximization) and contributions (stochastic bandit algorithms and NP -hardness of the adversarial setting).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 6 discusses limitations and possible directions for overcoming them.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are clearly stated in Section 2.2 and problem-setting paragraphs in Sections 4 and 5. All theoretical results are followed by complete proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: This paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper is dedicated to theoretical results. While there are potential societal consequences of our work, we feel none of them must be specifically discussed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers online learning variants of ${\\rm M}^\\natural$-concave function maximization.  These types of function generalize maximum flows on graphs (where the variable is the vector of source values on each node), gross substitute valuations in economics, and have applications in resource allocation.\n\nThis paper has two main results:\n\n - A learning strategy with $O(K N^{1/3} T^{2/3})$-regret in the stochastic setting (here we can query a function at a given input and observe its output perturbed by mean 0, 1-subgaussian noise and we have $T$ rounds to perform these queries).  This is obtained by showing that the greedy algorithm is \"robust to local errors\", and is the key technical step for this result.  The regret bound follows by an application of known results for pure-exploration + the explore-then-commit paradigm.\n\n - NP-hardness for the adversarial setting.  Unless P=NP, it is impossible for any polynomial time learner to achieve sub-linear regret in the adversarial setting.  This is done via a clever reduction from the 3-matroid-intersection problem.\n\nSTRENGTHS:\n- This paper considers online learning for a very general class of functions, which makes the results potentially widely applicable.\n - The \"robust to local errors\" Theorem for $\\rm{M}^\\natural$-concave functions may be of independent interest.\n - The proofs are well written and easy to follow, which is challenging to do for the very technical topic at hand.  \n - The NP-hardness result is interesting and shows a sharp difference between the stochastic and adversarial case for this problem.\n\nWEAKNESSES:\n- For the upper bound, the techniques are fairly straightforward once we have Theorem 3.1.  I am wondering if more can be done, e.g. regret that is sublinear in $K$, improving the bound for $T$ to $O(\\sqrt{T})$, or sublinear approximate regret in the adversarial setting.  These suggestions have been pointed out by the authors themselves, but they are relevant questions.\n- The stochastic online setting for this problem could be better motivated than the paragraph in lines 122-126.  More concrete examples would be helpful to clarify the applicability of these results beyond the nice mathematics.\n- No experimental evaluation.  While this paper makes a nice theoretical contribution, it could be helpful to demonstrate that the theory is applicable to a problem that prior methods were not suitable for.\n\nQUESTIONS:\nPlease clarify the relationship between ${\\rm M}^{\\natural}$ concave maximization and submodular function maximization.  I found it hard to understand the relationship between these as it pertains to the results in this paper and the discussion therein.  For example, the conclusion states that if the domain is restricted to $\\{0,1\\}^V$ then we get online submodular function maximization as a special case.  Here, the offline problem is NP-hard so I'm not sure how to interpret the fact that we can get sublinear-regret with respect to $\\mathbb{E}[f^*(x^*)]$ with a polynomial-time algorithm.\n\nSome very minor comments:\n- ${\\rm M}^\\natural$-convexity is used in section 5, but not defined in the paper\n\nLIMITATIONS:\nThe authors have adequately addressed and discussed limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider online $M^\\sharp$-concave optimization, similar to problems like online convex optimization and online DR-submodular optimization.  $M^\\sharp$-concave function classes include resource allocation, valuation, and flow problems, and unlike DR-submodular functions can be exactly optimized (at least under a cardinality constraint) by a simple greedy algorithm.  The authors show standard online adaptation of the greedy algorithm analogous to similar work for submodular functions, though prove the robustness of the offline algorithm to oracle value errors in a manner distinct from previous robustness analyses (such as for submodular functions).  The authors also show that unlike OCO and online DR-submodular optimization, the adversarial setting is fundamentally harder than the stochastic setting – one cannot (with poly-time per round complexity) achieve sublinear regret.\n\nSTRENGTHS:\n1. The authors consider an interesting class of functions for online optimization, which are related to, but much easier to optimize than, (DR-)submodular functions.  They show an (potentially quite) interesting dichotomy in hardness between the online stochastic setting and the online adversarial setting, unlike related classes like online convex optimization or online DR-submodular optimization.  For the online stochastic setting they get analogous results to prior work for (DR-)submodular, but show through a matroid-intersection construction that one cannot obtain sublinear regret.  \n\n2. The robustness analysis is distinct from that used in submodular bandit papers.  As described, the standard proof(s) for offline cardinality-constrained $M^\\sharp$-concave maximization does not have a structure that lends itself to accounting for the function value impact in sequential mistakes by the greedy algorithm.  Thus, the authors develop a new proof that clearly shows such additive accumulation, and subsequently permits almost direct adaptation to the online stochastic setting.  (one minor note - line 191 “ours is different from them in that it involves no approximation factors” that aspect I don’t see in and of itself as a meaningful distinction since the offline problem is not NP-hard; from my reading it is how the offline proof is structured as to whether it can be easily modified to account for value oracle errors that matters.) \n\n3.  I found the paper overall well written, with good organization, logical flow, discussions, etc.\n\nWEAKNESSES:\n1. The stochastic setting algorithms and analysis adapting a greedy algorithm from the offline setting (including an ETC method getting $T^{2/3}$) are straightforward (once a robustness result is in hand), though the authors do acknowledge that in the main paper.  \n\n2. This is a somewhat minor point – since the need for a novel robustness analysis arises due to the offline proof of the greedy algorithm’s not already being in a manner that can be easily adapted to account for local errors (which is unlike the proof of $1-1/e$ for the greedy approximation algorithm for cardinality constrained submodular maximization, as noted in Appendix A), I think it would have been helpful to include the proof(s) for constrained $M^\\sharp$-concave maximization (specialized to cardinality constraint) so that this need would be more apparent, that no standard proof has a sequential exchange argument that would already lend itself more readily to analyzing robustness and that the robustness result used in the offline setting with exact value oracles clearly constitutes a distinct proof.  \n\n3. minor point – it is not clear to what extent the positive result in the stochastic setting can be generalized beyond a simple cardinality constraint.  \n\n4. minor point – there is no implementation or experiments for toy flow/valuation/resource allocation problems.\n\nQUESTIONS:\n1. What is known about the hardness of offline optimization for maximizing sums of (possibly non-monotone) $M^\\sharp$-concave functions? \nFor the adversarial setting, the authors use an interesting construction based on matroid intersection to prove hardness for no-approximation sublinear regret (with poly-time per round computation).  But I wonder if it could have been reached in a more straightforward manner.  Namely, for the stochastic setting, the regret is based on a single $M^\\sharp$-concave function $f^*$ for which in the offline setting the greedy algorithm can find the optimal solution. In the adversarial setting, however, the regret is based on a sum of $M^\\sharp$-concave functions and in line 100 we are told that in general the class of $M^\\sharp$ functions is not closed under addition.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper studies the online version of M-concave function maximization. With the help of a new lemma that bounds local errors for greedy algorithms, the authors derive an efficient $T^{-1/2}$ simple regret algorithm together with a $T^{2/3}$ regret algorithm for stochastic settings. Moreover, the authors show an interesting computational hardness result for adversarial settings with full information.

STRENGTHS:
1. This paper studies a new online learning problem and proposes an efficient algorithm with sublinear regret for stochastic settings.
2. A new computational hardness result for adversarial settings with full information is given, which introduces a new understanding of online optimization.

WEAKNESSES:
1. The regret bound for stochastic setting is not tight.

QUESTIONS:
1. What is the main technical challenge in getting $\sqrt{K}$ regret for stochastic settings? Specifically, what is the difficulty when implementing the UCB-type exploration on the fly rather than doing an explore-then-commit algorithm?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the online optimization problem with $M^\\natural$-concave function, which has many real-world implications such as maximum-flow on bipartite graphs, gross substitute valuation, resource allocation and bandit problem. $M^\\natural$-concave function forms the fundamental basis of discrete concave analysis. In the stochastic bandit setting, they present algorithms with $O(T^-1/2)$ simple regret and $O(T^2/3)$ cumulative regret. These results leverage the robustness of the greedy algorithm to local errors, a significant technical contribution of the paper.\n\nAdditionally, the author also presents the result in the adversarial online learning setting, the paper proves that achieving sub-linear regret is NP-hard, even with full-information feedback, which establishes a distinct difference from the offline setting.\n\nSTRENGTHS:\nThe paper focuses on $M^\\natural$-concave functions, which is a crucial class in discrete convex analysis with wide applications, thus benefiting the analysis on a large variety community.\nThe author brings both stochastic bandit and adversarial full-information settings. \nEspecially for the adversarial setting, at least linear regret justification will guide us to avoid building sub-linear regret algorithms.\nThis theorem provides a profound understanding of the challenges and encourages us to think more potential alternative model assumptions and solutions for optimizing $M^\\natural$-concave functions in the online scenarios.\n\nWEAKNESSES:\nNo obvious weaknesses.\n\nQUESTIONS:\nI am more curious about the conclusion in the bandit example.\n\n1. In theorem 4.2, does $sReg_T$ is equivalent to the error in the Best-arm identification problem?\n\n2. Does the bit-O term in theorem 4.2 and 4.3 hide the logarithmic term?\n\n3. The theorem 4.3 claims that the proposed algorithm for stochastic bandit achieves cumulative regret as $O(K N^{1/2} T^{2/3}$. While according to [1], the tight lower bound of a consistent bandit algorithm is $\\Theta(K^{1/2}T^{1/2})$. What kind of factor do you think causes the gap between your upper and lower bound?\n\n[1] Auer, Peter, et al. \"The nonstochastic multiarmed bandit problem.\" *SIAM journal on computing* 32.1 (2002): 48-77.\n\nLIMITATIONS:\nPlease refer to Questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}