Title: A Unified Approach for Maximizing Continuous DR-submodular Functions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Stochastic optimization

submodular maximization

Frank-Wolfe algorithm
Abstract: This paper presents a unified approach for maximizing continuous DR-submodular functions that encompasses a range of settings and oracle access types. Our approach includes a Frank-Wolfe type offline algorithm for both monotone and non-monotone functions, with different restrictions on the general convex set. We consider settings where the oracle provides access to either the gradient of the function or only the function value, and where the oracle access is either deterministic or stochastic. We determine the number of required oracle accesses in all cases. Our approach gives new/improved results for nine out of the sixteen considered cases, avoids computationally expensive projections in three cases, with the proposed framework matching performance of state-of-the-art approaches in the remaining four cases. Notably, our approach for the stochastic function value-based oracle enables the first regret bounds with bandit feedback for stochastic DR-submodular functions.

A Unified Approach for Maximizing Continuous DR-submodular Functions

Abstract

This paper presents a unified approach for maximizing continuous DRsubmodular functions that encompasses a range of settings and oracle access types. Our approach includes a Frank-Wolfe type offline algorithm for both monotone and non-monotone functions, with different restrictions on the general convex set. We consider settings where the oracle provides access to either the gradient of the function or only the function value, and where the oracle access is either deterministic or stochastic. We determine the number of required oracle accesses in all cases. Our approach gives new/improved results for nine out of the sixteen considered cases, avoids computationally expensive projections in three cases, with the proposed framework matching performance of state-of-the-art approaches in the remaining four cases. Notably, our approach for the stochastic function valuebased oracle enables the first regret bounds with bandit feedback for stochastic DR-submodular functions.

1 Introduction

The problem of optimizing DR-submodular functions over a convex set has attracted considerable interest in both the machine learning and theoretical computer science communities [2, 5, 16, 26]. This is due to its many practical applications in modeling real-world problems, such as influence/revenue maximization, facility location, and non-convex/non-concave quadratic programming [3, 9, 18, 15, 20]. as well as more recently identified applications like serving heterogeneous learners under networking constraints [20] and joint optimization of routing and caching in networks [21].

Numerous studies investigated developing approximation algorithms for constrained DRsubmodular maximization, utilizing a variety of algorithms and proof analysis techniques. These studies have addressed both monotone and non-monotone functions and considered various types of constraints on the feasible region. The studies have also considered different types of oraclesgradient oracles and value oracles, where the oracles could be exact (deterministic) or stochastic. Lastly, for some of the aforementioned offline problem settings, some studies have also considered analogous online optimization problem settings as well, where performance is measured in regret over a horizon. This paper aims to unify the disparate offline problems under a single framework by providing a comprehensive algorithm and analysis approach that covers a broad range of setups. By providing a unified framework, this paper presents novel results for several cases where previous research was either limited or non-existent, both for offline optimization problems and extensions to related stochastic online optimization problems.

This paper presents a Frank-Wolfe based meta-algorithm for (offline) constrained DR-submodular maximization where we could only query within the constraint set, with sixteen variants for sixteen problem settings. The algorithm is designed to handle settings where (i) the function is monotone

This table compares the different results for the number of oracle calls (complexity) within the feasible set for DR-submodular maximization. Shaded rows indicate problem settings for which our work has the first guarantees or beats the SOTA . The rows marked with a blue star (*) correspond to cases where Algorithm 2 generalizes the corresponding algorithm and therefore has the same performance. The different columns enumerate properties of the function, the convex feasible region (downward-closed, includes the origin, or general), and the oracle, as well as the approximation ratios and oracle complexity (the number of queries needed to achieve the stated approximation ratio with at most ϵ > 0 additive error). We have h := min x ∈K ∥ x ∥ ∞ . (See Appendix B regarding [23] and [32]). † when the oracle can be queried for any points in [0 , 1] d (even outside the feasible region K ), the problem of optimizing monotone DR-submodular functions over a general convex set simplifies - [4] and [23] achieve the same ratios and complexity bounds as listed above for 0 ∈ K ; [8] can achieve an approximation ratio of 1 -1 /e with the O (1 /ϵ 3 ) and O (1 /ϵ 5 ) complexity for exact and stochastic value oracles respectively. ‡ [16] and [32] use gradient ascent, requiring potentially computationally expensive projections.

or non-monotone, (ii) the feasible region is a downward-closed (d.c.) set (extended to include 0 for monotone functions) or a general convex set, (iii) gradient or value oracle access is available, and (iv) the oracle is exact or stochastic. Table 1 enumerates the cases and corresponding results on oracle complexity (further details are provided in Appendix A). We derive the first oracle complexity guarantees for nine cases, derive the oracle complexity in three cases where previous result had a computationally expensive projection step [32, 16] (and we obtain matching complexity in one of these), and obtain matching guarantees in the remaining four cases. In addition to proving approximation ratios and oracle complexities for several (challenging) settings that are the first or improvements over the state of the art, the technical novelties of our approach include:

(i) A new construction procedure of a shrunk constraint set that allows us to work with lower dimensional feasible sets when given a value oracle, resulting in the first results on general lower dimensional feasible sets given a value oracle.

(ii) The first Frank-Wolfe type algorithm for analyzing monotone functions over a general convex set for any type of oracle, where only feasible points can be queried.

(iii) Shedding light on a previously unexplained gap in approximation guarantees for monotone DR-submodular maximization. Specifically, by considering the notion of query sets and assuming that the oracles can only be queries within the constraint set, we divide the class of

monotone submodular maximization into monotone submodular maximization over convex sets containing the origin and monotone submodular maximization over general convex sets. Moreover, we conjecture that the 1 / 2 approximation coefficient, which has been considered sub-optimal in the literature, is optimal when oracle queries can only be made within the constraint set. (See Appendix B for more details.)

Furthermore, we also consider online stochastic DR-submodular optimization with bandit feedback, where an agent sequentially picks actions (from a convex feasible region), receives stochastic rewards (in expectation a DR-submodular function) but no additional information, and seeks to maximize the expected cumulative reward. Performance is measured against the best action in expectation (or a near-optimal baseline when the offline problem is NP-hard but can be approximated to within α in polynomial time), the difference denoted as expected α -regret. For such problems, when only bandit feedback is available (it is typically a strong assumption that semi-bandit or fullinformation feedback is available), the agent must be able to learn from stochastic value oracle queries over the feasible actions action. By designing offline algorithms that only query feasible points, we made it possible to convert those offline algorithms into online algorithms. In fact, because of how we designed the offline algorithms, we are able to access them in a black-box fashion for online problems when only bandit feedback is available. Note that previous works on DR-submodular maximization with bandit feedback in monotone settings (e.g. [31], [25] and [30]) explicitly assume that the convex set contains the origin.

For each of the offline setups, we extend the offline algorithm (the respective variants for stochastic value oracle) and oracle query guarantees to provide algorithms and α -regret bounds in the bandit feedback scenario. Table 2 enumerates the problem settings and expected regret bounds with bandit and semi-bandit feedback. The key contributions of this work can be summarized as follows:

1. This paper proposes a unified approach for maximizing continuous DR-submodular functions in a range of settings with different oracle access types, feasible region properties, and function properties. A Frank-Wolfe based algorithm is introduced, which compared to SOTA methods for each of the sixteen settings, achieves the best-known approximation coefficients for each case while providing (i) the first guarantees in nine cases, (ii) reduced computational complexity by avoiding projections in three cases, and (iii) matching guarantees in remaining four cases.

2. In particular, this paper gives the first results on offline DR-submodular maximization (for both monotone and non-monotone functions) over general convex sets and even for downward-closed convex sets, when only a value oracle is available over the feasible set. Most prior works on offline DR-submodular maximization require access to a gradient oracle.

3. The results, summarized in Table 2, are presented with two feedback models-bandit feedback where only the (stochastic) reward value is available and semi-bandit feedback where a single stochastic sample of the gradient at the location is provided. This paper presents the first regret analysis with bandit feedback for stochastic DRsubmodular maximization for both monotone and nonmonotone functions. For semi-bandit feedback case, we provide the first result in one case, improve the state of the art result in two cases, and gives the result without computationally intensive projections in one case.

This table compares the different results for the expected α -regret for online stochastic DR-submodular maximization for the under bandit and semibandit feedback. Shaded rows indicate problem settings for which our work has the first guarantees or beats the SOTA . We have h := min x ∈K ∥ x ∥ ∞ . † the analysis in [6] has an error (see the supplementary material for details). ‡ [16] uses gradient ascent, requiring potentially computationally expensive projections.

Related Work: The key related works are summarized in Tables 1 and 2. We briefly discuss some key works here; see the supplementary materials for more discussion. For online DRsubmodular optimization with bandit feedback, there has been some prior works in the adversarial setup [31, 33, 25, 30] which are not included in Table 2 as we consider the stochastic setup. [31] considered monotone DR-submodular functions over downward-closed convex sets and achieved (1 -1 /e ) -regret of O ( T 8 / 9 ) in adversarial setting. This was later improved by [25] to O ( T 5 / 6 ) . [30] further improved the regret bound to O ( T 3 / 4 ) . However, it should be noted that they use a convex optimization subroutine at each iteration which could be even more computationally expensive than projection. [33] considered non-monotone DR-submodular functions over downward-closed convex sets and achieved 1 /e -regret of O ( T 8 / 9 ) in adversarial setting.

2 Background and Notation

We introduce some basic notions, concepts and assumptions which will be used throughout the paper. For any vector x ∈ R d , [ x ] i is the i -th entry of x . We consider the partial order on R d where x ≤ y if and only if [ x ] i ≤ [ y ] i for all 1 ≤ i ≤ d . For two vectors x , y ∈ R d , the join of x and y , denoted by x ∨ y and the meet of x and y , denoted by x ∧ y , are defined by

respectively. Clearly, we have x ∧ y ≤ x ≤ x ∨ y . We use ∥ · ∥ to denote the Euclidean norm, and ∥ · ∥ ∞ to denote the supremum norm. In the paper, we consider a bounded convex domain K and w.l.o.g. assume that K ⊆ [0 , 1] d . We say that K is down-closed (d.c.) if there is a point u ∈ K such that for all z ∈ K , we have { x | u ≤ x ≤ z } ⊆ K . The diameter D of the convex domain K is defined as D := sup x , y ∈K ∥ x -y ∥ . We use B r ( x ) to denote the open ball of radius r centered at x . More generally, for a subset X ⊆ R d , we define B r ( X ) := ⋃ x ∈ X B r ( x ) . If A is an affine subspace of R d , then we define B A r ( X ) := A ∩ B r ( X ) . For a function F : D → R and a set L , we use F | L to denote the restriction of F to the set D∩L . For a linear space L 0 ⊆ R d , we use P L 0 : R d →L 0 to denote the projection onto L 0 . We will use R d + to denote the set { x ∈ R d | x ≥ 0 } . For any set X ⊆ R d , the affine hull of X , denoted by aff( X ) , is defined to be the intersection of all affine subsets of R d that contain X . The relative interior of a set X is defined by

It is well known that for any non-empty convex set K , the set relint( K ) is always non-empty. We will always assume that the feasible set contains at least two points and therefore dim(aff( K )) ≥ 1 , otherwise the optimization problem is trivial and there is nothing to solve.

Aset function f : { 0 , 1 } d → R + is called submodular if for all x , y ∈ { 0 , 1 } d with x ≥ y , we have

Submodular functions can be generalized over continuous domains. A function F : [0 , 1] d → R + is called DR-submodular if for all vectors x , y ∈ [0 , 1] d with x ≤ y , any basis vector e i = (0 , · · · , 0 , 1 , 0 , · · · , 0) and any constant c > 0 such that x + c e i ∈ [0 , 1] d and y + c e i ∈ [0 , 1] d , it holds that

Note that if function F is differentiable then the diminishing-return (DR) property (3) is equivalent to ∇ F ( x ) ≥ ∇ F ( y ) for x ≤ y with x , y ∈ [0 , 1] d . A function F : D → R + is G -Lipschitz continuous if for all x , y ∈ D , ∥ F ( x ) -F ( y ) ∥ ≤ G ∥ x -y ∥ . Adifferentiable function F : D → R + is L -smooth if for all x , y ∈ D , ∥∇ F ( x ) -∇ F ( y ) ∥ ≤ L ∥ x -y ∥ .

A (possibly randomized) offline algorithm is said to be an α -approximation algorithm (for constant α ∈ (0 , 1] ) with ϵ ≥ 0 additive error for a class of maximization problems over non-negative functions if, for any problem instance max z ∈K F ( z ) , the algorithm output x that satisfies the following relation with the optimal solution z ∗

where the expectation is with respect to the (possible) randomness of the algorithm. Further, we assume an oracle that can query the value F ( x ) or the gradient ∇ F ( x ) . The number of calls to the oracle to achieve the error in (4) is called the evaluation complexity .

3 Offline Algorithms and Guarantees

In this section, we consider the problem of maximizing a DR-submodular function over a general convex set in sixteen different cases, enumerated in Table 1. After setting up the problem in Section 3.1, we then explain two key elements of our proposed algorithm when we only have access to a value oracle, (i) the Black Box Gradient Estimate (BBGE) procedure (Algorithm 1) to balance bias and variance in estimating gradients (Section 3.2) and (ii) the construction of a shrunken feasible region to avoid infeasible value oracle queries during the BBGE procedure (Section 3.3). Our main algorithm is proposed in Section 3.4 and analyzed in Section 3.5.

3.1 Problem Setup

We consider a general non-oblivious constrained stochastic optimization problem

where F is a DR-submodular function, and ˆ F : K× X → R is determined by z and the random variable x which is independently sampled according to x ∼ p ( x ; z ) . We say the oracle has variance σ 2 if sup z ∈K var x ∼ p ( x ; z ) [ ˆ F ( z , x )] = σ 2 . In particular, when σ = 0 , then we say we have access to an exact (deterministic) value oracle. We will use ˆ F ( z ) to denote the random variables ˆ F ( z , x ) where x is a random variable with distribution p (˙ ; z ) . Similarly, we say we have access to a stochastic gradient oracle if we can sample from function ˆ G : K× Y → R such that ∇ F ( z ) = E y ∼ q ( y ; z ) [ ˆ G ( z , y )] , and ˆ G is determined by z and the random variable y which is sampled according to y ∼ q ( y ; z ) . Note that oracles are only defined on the feasible set. We will use ˆ G ( z ) to denote the random variables ˆ G ( z , y ) where y is a random variable with distribution q (˙ ; z ) .

Assumption 1. We assume that F : [0 , 1] d → R is DR-submodular, first-order differentiable, nonnegative, G -Lipschitz for some G < ∞ , and L -smooth for some L < ∞ . We also assume the feasible region K is a closed convex domain in [0 , 1] d with at least two points. Moreover, we also assume that we either have access to a value oracle with variance σ 2 0 ≥ 0 or a gradient oracle with variance σ 2 1 ≥ 0 .

Remark 1 . The proposed algorithm does not need to know the values of L , G , σ 0 or σ 1 . However, these constants appear in the final expressions of the number of oracle calls and the regret bounds.

3.2 Black Box Gradient Estimate

Without access to a gradient oracle (i.e., first-order information), we estimate gradient information using samples from a value oracle. We will use a variation of the 'smoothing trick' technique [14, 17, 1, 27, 31, 8, 33], which involves averaging through spherical sampling around a given point. Definition 1 (Smoothing Trick) . For a function F : D → R defined on D ⊆ R d , its δ -smoothed version ˜ F δ is given as

where v is chosen uniformly at random from the dim(aff( D )) -dimensional ball B aff( D ) -x 1 (0) . Thus, the function value ˜ F δ ( x ) is obtained by 'averaging' F over a sliced ball of radius δ around x .

When the value of δ is clear from the context, we may drop the subscript and simply use ˜ F to denote the smoothed version of F . It can be easily seen that if F is DR-submodular, G -Lipschitz continuous, and L -smooth, then so is ˜ F and ∥ ˜ F ( x ) -F ( x ) ∥ ≤ δG , for any point in the domain of both functions. Moreover, if F is monotone, then so is ˜ F (Lemma 3). Therefore ˜ F δ is an approximation of the function F . A maximizer of ˜ F δ also maximizes F approximately.

Our definition of smoothing trick differs from the standard usage by accounting for the affine hull containing D . This will be of particular importance when the feasible region is of (affine) dimension less than d , such as when there are equality constraints. When aff( D ) = R d , our definition reduces to the standard definition of the smoothing trick. In this case, it is well-known that the gradient of the smoothed function ˜ F δ admits an unbiased one-point estimator [14, 17]. Using a two-point estimator instead of the one-point estimator results in smaller variance [1, 27]. In Algorithm 1, we adapt the two-point estimator to the general setting.

3.3 Construction of K δ

We want to run Algorithm 1 as a subroutine within the main algorithm to estimate the gradient. However, in order to run Algorithm 1, we need to be able to query the oracle within the set B aff( K ) δ ( x ) . Since the oracle can only be queried at points within the feasible set, we need to restrict our attention to a set K δ such that B aff( K ) δ ( K δ ) ⊆ K . On the other hand, we want the optimal point of F within K δ to be close to the optimal point of F within K . One way to ensure

Algorithm 1 Black Box Gradient Estimate (BBGE)

1: Input: Point z , sampling radius δ , constraint linear space L 0 , k = dim( L 0 ) , batch size B

2: Sample u 1 , · · · , u B i.i.d. from S d -1 ∩ L 0

4: g ← 1 B ∑ B i =1 k 2 δ [ ˆ F ( y + i ) -ˆ F ( y -i ) ] u i

3: For i = 1 to B , let y + i ← z + δ u i , y -i ← z -δ u i , and evaluate ˆ F ( y + i ) , ˆ F ( y -i )

5: Output g

that is to have K δ not be too small. More formally, we want that B aff( K ) δ ′ ( K δ ) ⊇ K , for some value of δ ′ ≥ δ that is not too large. The constraint boundary could have a complex geometry, and simply maintaining a δ sized margin away from the boundary can result in big gaps between the boundary of K and K δ . For example, in two dimensions, if K is polyhedral and has an acute angle, maintaining a δ margin away from both edges adjacent to the acute angle means the closest point in the K δ to the corner may be much more than δ . For this construction, we choose a c ∈ relint( K ) and a real number r > 0 such that B aff( K ) r ( c ) ⊆ K . For any δ < r , we define

Clearly if K is downward-closed, then so is K c ,r δ . Lemma 7 shows that for any such choice of c and r > 0 , we have δ ′ δ ≤ D r . See Appendix G for more details about the choice of c and r . We will drop the superscripts in the rest of the paper when there is no ambiguity.

Remark 2 . This construction is similar to the one carried out in [31] which was for d -dimensional downward-closed sets. Here we impose no restrictions on K beyond Assumption 1. A simpler construction of shrunken constraint set was proposed in [8]. However, as we discuss in Appendix D, they require to be able to query outside of the constraint set.

3.4 Generalized DR-Submodular Frank-Wolfe

The pseudocode of our proposed offline algorithm, Generalized DR-Submodular Frank-Wolfe , is shown in Algorithm 2. At a high-level, it follows the basic template of Frank-Wolfe type methods, where over the course of a pre-specified number of iterations, the gradient (or a surrogate thereof) is calculated, an optimization sub-routine with a linear objective is solved to find a feasible point whose difference (with respect to the current solution) has the largest inner product with respect to the gradient, and then the current solution is updated to move in the direction of that feasible point.

However, there are a number of important modifications to handle properties of the objective function, constraint set, and ora-

Algorithm 2 Generalized DR-Submodular FrankWolfe

1: Input: Constraint set K , iteration limit N ≥ 4 , sampling radius δ , gradient step-size { ρ n } N n =1

2: Construct K δ

3: Pick any z 1 ∈ argmin z ∈K δ ∥ z ∥ ∞

4: ¯ g 0 ← 0

5: for n = 1 to N do

6: g n ← estimate-grad( z n , δ, L 0 = aff( K ) -z 1 )

7: ¯ g n ← (1 -ρ n )¯ g n -1 + ρ n g n

8: v n ← optimal-direction(¯ g n , z n )

9: z n +1 ← update( z n , v n , ε )

10: end for

11: Output z N +1

cle type. For the oracle type, for instance, standard Frank-Wolfe methods assume access to a deterministic gradient oracle. Frank-Wolfe methods are known to be sensitive to errors in estimates of the gradient (e.g., see [16]). Thus, when only a stochastic gradient oracle or even more challenging, only a stochastic value oracle is available, the gradient estimators must be carefully designed to balance query complexity on the one hand and output error on the other. The Black Box Gradient Estimate (BBGE) sub-routine, presented in Algorithm 1, utilizes spherical sampling to produce an unbiased gradient estimate. This estimate is then combined with past estimates using momentum, as seen in [23], to control and reduce variance.

Our algorithm design is influenced by state-of-the-art methods that have been developed for specific settings. One of the most closely related works is [8], which also dealt with using value oracle access for optimizing monotone functions. They used momentum and spherical sampling techniques that are similar to the ones we used in our Algorithm 1. However, we modified the sampling procedure and the solution update step. In their work, [8] also considered a shrunken feasible region to avoid sampling close to the boundary. However, they assumed that the value oracle could be queried outside the feasible set (see Appendix D for details).

In Algorithm 3, we consider the following cases for the function and the feasible set.

(A) If F is monotone DR-submodular and 0 ∈ K , we choose

and ε = 1 /N . We start at a point near the origin and always move to points that are bigger with respect to the partial order on R d . In this case, since the function is monotone, the optimal direction is a maximal point with respect to the partial order. The choice of ε = 1 /N guarantees that after N steps, we arrive at a convex combination of points in the feasible set and therefore the final point is also in the feasible set. The fact that the origin is also in the feasible set shows that the intermediate points also belong to the feasible set.

and ε = 1 /N . This case is similar to (A). However, since F is not monotone, we need to choose the optimal direction more conservatively.

(C) If F is monotone DR-submodular and K is a general convex set, we choose

,

and ε = log( N ) / 2 N . In this case, if we update like in cases (A) and (B), we do not have any guarantees of ending up in the feasible set, so we choose the update function to be a convex combination. Unlike (B), we do not need to limit ourselves in choosing the optimal direction and we simply choose ε to obtain the best approximation coefficient.

(D) If F is non-monotone DR-submodular and K is a general convex set, we choose

optimal-direction(¯ g n , z n ) = argmax v ∈K δ ⟨ v , ¯ g n ⟩ , update( z n , v n , ε ) = (1 -ε ) z n + ε v n , and ε = log(2) /N . This case is similar to (C) and we choose ε to obtain the best approximation coefficient.

The choice of subroutine estimate-grad and ρ n depend on the oracle. If we have access to a gradient oracle ˆ G , we set estimate-grad( z , δ, L 0 ) to be the average of B evaluations of P L 0 ( ˆ G ( z )) . Otherwise, we run Algorithm 1 with input z , δ , L 0 . If we have access to a deterministic gradient oracle, then there is no need to use any momentum and we set ρ n = 1 . In other cases, we choose ρ n = 2 ( n +3) 2 / 3 .

3.5 Approximation Guarantees for the Proposed Offline Algorithm

Theorem 1. Suppose Assumption 1 holds. Let N ≥ 4 , B ≥ 1 and choose c ∈ K and r > 0 according to Section 3.3. If we have access to a gradient oracle, we choose δ = 0 , otherwise we choose δ ∈ (0 , r/ 2) . Then the following results hold for the output z N +1 of Algorithm 2.

(A) If F is monotone DR-submodular and 0 ∈ K , then

(B) If F is non-monotone DR-submodular and K is a downward closed set containing 0 , then

(C) If F is monotone DR-submodular and K is a general convex set, then

(D) If F is non-monotone DR-submodular and K is a general convex set, then

In all these cases, we have

C is a constant, k = dim( K ) , D = diam( K ) , and z ∗ is the global maximizer of F on K .

Theorem 1 characterizes the worst-case approximation ratio α and additive error bounds for different properties of the function and feasible region, where the additive error bounds depend on selected parameters N for the number of iterations, batch size B , and sampling radius δ .

The proof of Parts (A)-(D) is provided in Appendix I-L, respectively.

The proof of parts (A), (B) and (D), when we have access to an exact gradient oracle is similar to the proofs presented in [4, 3, 24], respectively. Part (C) is the first analysis of a Frank-Wolfe type algorithm over general convex sets when the oracle can only be queried within the feasible set. When we have access to a stochastic gradient oracle, directly using a gradient sample can result in arbitrary bad performance as shown in Appendix B of [16]. The momentum technique, first used in continuous submodular maximization in [23], is used when we have access to a stochastic gradient oracle. The control on the estimate of the gradient is deferred to Lemma 9. Since the momentum technique is robust to noise in the gradient, when we only have access to a value oracle, we can use Algorithm 1, similar to [8], to obtain an unbiased estimate of the gradient and complete the proof.

Theorem 2 converts those bounds to characterize the oracle complexity for a user-specified additive error tolerance ϵ based on oracle properties (deterministic/stochastic gradient/value). The 16 combinations of the problem settings listed in Table 1 are enumerated by four cases (A)-(D) in Theorem 1 of function and feasible region properties (resulting in different approximation ratios) and the four cases 1-4 enumerated in Theorem 2 below of oracle properties. For the oracle properties, we consider the four cases as (Case 1): deterministic gradient oracle, (Case 2): stochastic gradient oracle, (Case 3): deterministic value oracle, and (Case 4): stochastic value oracle.

Theorem 2. The number of oracle calls for different oracles to achieve an α -approximation error of smaller than ϵ using Algorithm 1 is

Moreover, in all of the cases above, if F is non-monotone or 0 ∈ K , we may replace ˜ O with O .

See Appendix M for proof.

4 Online DR-submodular optimization under bandit or semi-bandit feedback

In this section, we first describe the Black-box Explore-Then-Commit algorithm that uses the offline algorithm for exploration, and uses the solution of the offline algorithm for exploitation. This is followed by the regret analysis of the proposed algorithm. This is the first algorithm for stochastic continuous DR-submodular maximization under bandit feedback and obtains state-of-the-art for semi-bandit feedback.

4.1 Problem Setup

There are typically two settings considered in online optimization with bandit feedback. The first is the adversarial setting, where the environment chooses a sequence of functions F 1 , · · · , F N and in each iteration n , the agent chooses a point z n in the feasible set K , observes F n ( z n ) and receives the reward F n ( z n ) . The goal is to choose the sequence of actions that minimize the following notion of expected α -regret.

In other words, the agent's cumulative reward is being compared to α times the reward of the best constant action in hindsight. Note that, in this case, the randomness is over the actions of the policy.

The second is the stochastic setting, where the environment chooses a function F : K → R and a stochastic value oracle ˆ F . In each iteration n , the agent chooses a point z n in the feasible set K , receives the reward ( ˆ F ( z n )) n by querying the oracle at z n and observes this reward. Here the outer subscript n indicates that the result of querying the oracle at time n , since the oracle is stochastic. The goal is to choose the sequence of actions that minimize the following notion of expected α -regret.

Further, two feedback models are considered - bandit and semi-bandit feedback. In the bandit feedback setting, the agent only observes the value of the function F n at the point z n . In the semibandit setting, the agent has access to a gradient oracle instead of a value oracle and observes ˆ G ( z n ) at the point z n , where ˆ G is an unbiased estimator of ∇ F .

In unstructured multi-armed bandit problems, any regret bound for the adversarial setup could be translated into bounds for the stochastic setup. However, having a non-trivial correlation between the actions of different arms complicates the relation between the stochastic and adversarial settings. Even in linear bandits, the relation between adversarial linear bandits and stochastic linear bandits is not trivial. (e.g. see Section 29 in [19]) While it is intuitively reasonable to assume that the optimal regret bounds for the stochastic case are better than that of the adversarial case, such a result is not yet proven for DR-submodular functions. Thus, while the cases of bandit feedback has been studied in the adversarial setup, the results do not reduce to stochastic setup. We also note that in the cases where there are adversarial setup results, this paper finds that the results in the stochastic setup achieve improved regret bounds (See Table 3 in Supplementary for the comparison).

4.2 Algorithm for DR-submodular maximization with Bandit Feedback

The proposed algorithm is described in Algorithm 3. In Algorithm 3, if there is semi-bandit feedback in the form of a stochastic gradient sample for each action z n , we run the offline algorithm (Algorithm 2) with parameters from the proof of case 2 of Theorem 2 for T 0 = ⌈ T 3 / 4 ⌉ total queries. If only the stochastic reward for each action z n is available (bandit feedback), we run the offline algorithm (Algorithm 2) with parameters from the proof of case 4 of Theorem 2 for T 0 = ⌈ T 5 / 6 ⌉ total queries. Then, for the remaining time (exploitation phase), we run the last action in the exploration phase.

Algorithm 3 DR-Submodular ExploreThen-Commit

1: Input: Horizon T , inner time horizon T 0

2: Run Algorithm 2 for T 0 , with according to parameters described in Theorem 2.

3: for remaining time do

4: Repeat the last action of Algorithm 2. 5: end for

4.3 Regret Analysis for DR-submodular maximization with Bandit Feedback

In this section, we provide the regret analysis for the proposed algorithm. We note that by Theorem 2, Algorithm 2 requires a sample complexity of ˜ O (1 /ϵ 5 ) with a stochastic value oracle for

offline problems (any of (A)-(D) in Theorem 1). Thus, the parameters and the results with bandit feedback are the same for all the four setups (A)-(D). Likewise, when a stochastic gradient oracle is available, Algorithm 2 requires a sample complexity of ˜ O (1 /ϵ 3 ) . Based on these sample complexities, the overall regret of online DR-submodular maximization problem is given as follows.

Theorem 3. For an online constrained DR-submodular maximization problem over a horizon T , where the expected reward function F , feasible region type K , and approximation ratio α correspond to any of the four cases (A)-(D) in Theorem 1, Algorithm 3 achieves α -regret (13) that is upperbounded as:

Moreover, in either type of feedback, if F is non-monotone or 0 ∈ K , we may replace ˜ O with O .

See Appendix N for the proof.

5 Conclusion

This work provides a novel and unified approach for maximizing continuous DR-submodular functions across various assumptions on function, constraint set, and oracle access types. The proposed Frank-Wolfe based algorithm improves upon existing results for nine out of the sixteen cases considered, and presents new results for offline DR-submodular maximization with only a value oracle. Moreover, this work presents the first regret analysis with bandit feedback for stochastic DR-submodular maximization, covering both monotone and non-monotone functions. These contributions significantly advance the field of DR-submodular optimization (with multiple applications) and open up new avenues for future research in this area.

Limitations: While the number of function evaluations in the different setups considered in the paper are state of the art, lower bounds have not been investigated.

References

Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In Adam Tauman Kalai and Mehryar Mohri, editors, Proceedings of the 23rd Annual Conference on Learning Theory (COLT 2010) , pages 28-40, 2010.

Francis Bach. Submodular functions: from discrete to continuous domains. Mathematical Programming , 175:419-459, 2019.

An Bian, Kfir Levy, Andreas Krause, and Joachim M Buhmann. Continuous DR-submodular maximization: Structure and algorithms. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Andrew An Bian, Baharan Mirzasoleiman, Joachim Buhmann, and Andreas Krause. Guaranteed Non-convex Optimization: Submodular Maximization over Continuous Domains. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 111-120. PMLR, 20-22 Apr 2017.

Yatao Bian, Joachim Buhmann, and Andreas Krause. Optimal continuous DR-submodular maximization and applications to provable mean field inference. In International Conference on Machine Learning , pages 644-653. PMLR, 2019.

Lin Chen, Christopher Harshaw, Hamed Hassani, and Amin Karbasi. Projection-free online optimization with stochastic gradient: From convexity to submodularity. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 814-823. PMLR, 10-15 Jul 2018.

Lin Chen, Hamed Hassani, and Amin Karbasi. Online continuous submodular maximization. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1896-1905. PMLR, 09-11 Apr 2018.

Lin Chen, Mingrui Zhang, Hamed Hassani, and Amin Karbasi. Black box submodular maximization: Discrete and continuous settings. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 1058-1070. PMLR, 26-28 Aug 2020.

Josip Djolonga and Andreas Krause. From map to marginals: Variational inference in Bayesian submodular models. Advances in Neural Information Processing Systems , 27, 2014.

Donglei Du. Lyapunov function approach for approximation algorithm design and analysis: with applications in submodular maximization. arXiv preprint arXiv:2205.12442 , 2022.

Donglei Du, Zhicheng Liu, Chenchen Wu, Dachuan Xu, and Yang Zhou. An improved approximation algorithm for maximizing a DR-submodular function over a convex set. arXiv preprint arXiv:2203.14740 , 2022.

Christoph D¨ urr, Nguyen Kim Thang, Abhinav Srivastav, and L´ eo Tible. Non-monotone DR-submodular maximization: Approximation and regret guarantees. arXiv preprint arXiv:1905.09595 , 2019.

Maryam Fazel and Omid Sadeghi. Fast First-Order Methods for Monotone Strongly DRSubmodular Maximization , pages 169-179. ACDA23, 2023.

Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms , pages 385-394, 2005.

Shuyang Gu, Chuangen Gao, Jun Huang, and Weili Wu. Profit maximization in social networks and non-monotone DR-submodular maximization. Theoretical Computer Science , 957:113847, 2023.

Hamed Hassani, Mahdi Soltanolkotabi, and Amin Karbasi. Gradient methods for submodular maximization. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Shinji Ito and Ryohei Fujimaki. Large-scale price optimization via network flow. Advances in Neural Information Processing Systems , 29, 2016.

Tor Lattimore and Csaba Szepesv´ ari. Bandit algorithms . Cambridge University Press, 2020.

Yuanyuan Li, Yuezhou Liu, Lili Su, Edmund Yeh, and Stratis Ioannidis. Experimental design networks: A paradigm for serving heterogeneous learners under networking constraints. IEEE/ACM Transactions on Networking , 31(5):2236-2250, 2023.

Yuanyuan Li, Yuchao Zhang, Stratis Ioannidis, and Jon Crowcroft. Jointly optimal routing and caching with bounded link capacities. In ICC 2023 - IEEE International Conference on Communications , pages 1130-1136, 2023.

Aryan Mokhtari, Hamed Hassani, and Amin Karbasi. Conditional gradient method for stochastic submodular maximization: Closing the gap. In International Conference on Artificial Intelligence and Statistics , pages 1886-1895. PMLR, 2018.

Aryan Mokhtari, Hamed Hassani, and Amin Karbasi. Stochastic conditional gradient methods: From convex minimization to submodular maximization. The Journal of Machine Learning Research , 21(1):4232-4280, 2020.

Loay Mualem and Moran Feldman. Resolving the approximability of offline and online nonmonotone DR-submodular maximization over general convex sets. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 2542-2564. PMLR, 25-27 Apr 2023.

Rad Niazadeh, Negin Golrezaei, Joshua R. Wang, Fransisca Susan, and Ashwinkumar Badanidiyuru. Online learning via offline greedy algorithms: Applications in market design and optimization. In Proceedings of the 22nd ACM Conference on Economics and Computation , EC '21, page 737-738, New York, NY, USA, 2021. Association for Computing Machinery.

Rad Niazadeh, Tim Roughgarden, and Joshua R Wang. Optimal algorithms for continuous nonmonotone submodular and DR-submodular maximization. The Journal of Machine Learning Research , 21(1):4937-4967, 2020.

Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with twopoint feedback. Journal of Machine Learning Research , 18(52):1-11, 2017.

Nguyen Kim Thang and Abhinav Srivastav. Online non-monotone DR-submodular maximization. Proceedings of the AAAI Conference on Artificial Intelligence , 35(11):9868-9876, May 2021.

Jan Vondr´ ak. Symmetry and approximability of submodular maximization problems. SIAM Journal on Computing , 42(1):265-304, 2013.

Zongqi Wan, Jialin Zhang, Wei Chen, Xiaoming Sun, and Zhijie Zhang. Bandit multi-linear DR-submodular maximization and its applications on adversarial submodular bandits. In Proceedings of the 40th International Conference on Machine Learning , pages 35491-35524. PMLR, 2023.

Mingrui Zhang, Lin Chen, Hamed Hassani, and Amin Karbasi. Online continuous submodular maximization: From full-information to bandit feedback. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Qixin Zhang, Zengde Deng, Zaiyi Chen, Haoyuan Hu, and Yu Yang. Stochastic continuous submodular maximization: Boosting via non-oblivious function. In Proceedings of the 39th International Conference on Machine Learning , pages 26116-26134. PMLR, 2022.

Qixin Zhang, Zengde Deng, Zaiyi Chen, Kuangqi Zhou, Haoyuan Hu, and Yu Yang. Online learning for non-monotone DR-submodular maximization: From full information to bandit feedback. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 3515-3537. PMLR, 25-27 Apr 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present a framework for maximizing continuous DR-submodular functions over a range of settings and Oracle access types. To achieve this, they employ a variant of the Frank-Wolfe algorithm which yields either the first guarantees in some cases or comparable results to the SOTA in other cases. The paper is generally well-written, and I have reviewed and found most of the proofs to be accurate. However, I do have a few significant concerns that I would like to highlight as weaknesses.\n\nSTRENGTHS:\n**Major:**\n1. This paper presents the first regret analysis with bandit feedback for stochastic DR-submodular functions maximization.\n\n**Minor:**\n1. Reducing computational complexity by avoiding projections in two cases.\n2. Obtaining the first results on offline DR-submodular maximization over general convex sets and down-closed convex sets.\n\nWEAKNESSES:\n**Major concerns:**\n\n1. This paper lacks empirical evaluation, and it is very crucial since the main contribution of this work is on avoiding computationally expensive projections. \n2. Motivation. In my opinion, this work lacks a very important paragraph/subsection on the importance of the setting where the oracle provides access to only the function value rather than the gradient of the function since the main results in this work are on this setting. For example, provide some applications in this setting.\n\nQUESTIONS:\n1. Can you please describe the main novelty of this framework?\n2. Besides the Online Stochastic DR-submodular maximization with Bandit Feedback result, the other results of this paper seem to be a straightforward extension of prior works. The authors need to highlight the challenges faced when proving these more general results and specify how they managed to overcome the challenges.\n\nLIMITATIONS:\nYes. The authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies offline constrained DR-submodular maximization in 16 different settings (monotone/non-monotone, down-closed/general convex constraint, gradient/value oracle access, and exact/stochastic oracle) and provides a unified approach to solve all 16 cases with the same Frank-Wolfe algorithmic framework. Moreover, the authors extend their tools to study the online stochastic DR-submodular maximization problem under bandit and semi-bandit feedback models. The provided offline and online algorithms either match or improve the state-of-the-art results for each of the various settings.\n\nSTRENGTHS:\n- Constrained DR-submodular maximization has been studied in numerous works under different assumptions and settings and a number of algorithms have been proposed. In contrast, the algorithmic framework proposed in this paper is general and could be applied to any of the possible settings. \n- The paper is well-written and while the proofs are moved to the appendix, the main concepts and ideas are highlighted clearly in the paper.\n- Algorithm 1 (BBGE) for gradient estimation contains some novel ideas.\n\nWEAKNESSES:\n- While the unifying approach of the paper is interesting, most of the tools and techniques for gradient estimation and the Frank-Wolfe-type algorithm used here have been previously introduced and are not novel.\n- The main contributions of this work are the improved complexity (offline setting) and regret bounds (online setting) and these improvements could be highlighted via numerical examples, however, the paper lacks any experiments.\n\nQUESTIONS:\n- I noticed that the following paper is missing from your references. In this work, the authors propose a boosting gradient ascent method that improves the $\\frac{1}{2}$ approximation ratio of gradient ascent to $1-\\frac{1}{e}$ for the monotone setting. How do their techniques and results compare to yours?\n\nZhang, Q., Deng, Z., Chen, Z., Hu, H. and Yang, Y., 2022, June. Stochastic continuous submodular maximization: Boosting via non-oblivious function. In International Conference on Machine Learning (pp. 26116-26134). PMLR.\n\n- Is it possible to use your unified approach and provide an analysis for the setting with bounded curvature (similar to what the following paper has done)?\n\nFazel, Maryam, and Omid Sadeghi. \"Fast First-Order Methods for Monotone Strongly DR-Submodular Maximization.\" In SIAM Conference on Applied and Computational Discrete Algorithms (ACDA23), pp. 169-179. Society for Industrial and Applied Mathematics, 2023.\n\n-  In the online setting with bandit feedback, how many value oracle queries are necessary per iteration to run Algorithm 3?\n\n--------------------------------\nI've read the authors' rebuttal, thanks for addressing my questions and concerns.\n\nLIMITATIONS:\nThis is a theoretical work and a discussion of potential negative societal impact is not necessary.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies maximizing stochastic DR-submodular functions under 16 different settings, which depend on 1) whether the function is monotone or not, 2) the feasible region is a downward-closed or a general convex set, 3) gradient or value oracle access is available, and 4) or the oracle is exact or stochastic. The authors present a unified approach based on the Frank-Wolfe meta-algorithm, which 1) provides the first oracle complexity guarantees in 9 settings, 2) reduces the computational complexity by avoiding projections in two settings, and 3) matches guarantees in the remaining 5 settings. The paper also considers online versions of the problem with bandit feedback and semi-bandit feedback and presents online algorithms with improved regret bounds.\n\nSTRENGTHS:\n1. The paper provides a unified approach for stochastic DR-submodular maximization, which encompasses a range of settings and beats or matches the SOTA results.\n\n2. The paper presents the first regret analysis for online stochastic DR-submodular maximization with bandit feedback.\n\n3. Technically, a novel construction procedure of a shrunk constraint set is invented that allows us to work with lower dimensional feasible sets when given a value oracle.\n\n4. The paper is organized very well, making it read clearly although there are so many settings. Besides, no typo was found after I read it.\n\nWEAKNESSES:\n1. I notice that only in the case where the value oracle is available, new guarantees that beat the SOTA are provided. So I consider it as the main part of the paper's contributions. In this case, the paper uses many standard techniques like the smoothing trick, the two-point estimator (for estimating gradients), and the momentum technique (for stochastic oracle). And to me what makes the paper different is the construction of a shrunk constraint set. But without reading the appendix, I can not figure out why such a construction is introduced and how it can achieve the claimed guarantees. So, I can not determine whether the paper enjoys excellent technical novelty or just consists of a refined combination of known techniques.\n\nQUESTIONS:\n1. If possible, please provides more intuition in the main text about what difficulties the construction of a shrunk constraint set is used to solve and how it can solve these difficulties.\n\n2. Very recently, There is a paper titled \"Bandit Multi-linear DR-Submodular Maximization and Its Applications on Adversarial Submodular Bandits\" [arXiv: 2305.12402, ICML23], which presents a $\\tilde{O}(T^{3/4})$ regret for monotone submodular maximization with adversary bandit feedback. This partially beats your results for bandit submodular maximization. As the approaches are different and the arXiv paper was unavailable during the NeurIPS submission, I do not think it is a \"weakness\". But it is suggested to include the paper in the Related Work.\n\nLIMITATIONS:\nYes, as stated in the paper, any non-trivial lower bound for the problem would be exciting.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of maximizing a continuous DR-submodular function over a convex set, under various settings for the objective (monotone/non-monotone), constraint set (downward-closed/includes the origin/general set), and oracle access types (deterministic/stochastic gradient/value oracle) -- 16 settings in total. For any type of oracle, the oracle is only allowed to query within the feasible set. The authors propose a Frank-Wolfe meta-algorithm and its different variants for each of the considered settings which achieve the best known approximation guarantee in each case. These approximation guarantees are the first ones in nine of the considered settings, reduce the computational complexity in two cases, and match existing guarantees in the remaining five cases. The authors also extend these results to the online stochastic setting with bandit and semi-bandit feedback. They provide the first regret bounds for the bandit setting, and in the semi-bandit setting, they provide the first regret bound in one case, improve over exiting bounds in two cases, and improve the computational complexity in one case.\n\nSTRENGTHS:\n- The paper provide the first approximation guarantees for various settings of DR-submodular maximization, where the oracle is only allowed to query within the feasible set, and improve over existing results in few other settings.\n- The paper cover a wide range of settings under a unified algorithm and analysis.\n- The presentation of the main paper is very good, especially taking into consideration the amount of material covered.\n- Relation to related work is clear: the authors discuss how their results compare to existing ones, and which ideas/proofs are based on existing work.\n\nWEAKNESSES:\n- Potential error in Lemma 5 (see questions), which affects all results with the function value oracle. If incorrect, this is easily fixable but the algorithm would require knowing the dimension of the affine hull of the constraint set.\n \n- The presentation in the appendix can be improved to make it easier to read the proofs (see suggestions in questions).\n\nQUESTIONS:\n- In Lemma 5, shouldn't d be replaced by k=dim(A) inside the expectation? Applying Lemma 4 with restriction to the first k coordinates is equivalent to applying it with sampling from a sphere of dimension k no? \n\n- Why is the claim on lines 671-672 in Lemma 5 true? Please explain or provide a reference.\n\n- The notations $\\hat{G}(z)|\\_\\mathcal{L}$ and $\\tilde{F}|\\_\\mathcal{L}$ are not defined, which makes it hard to understand some of the proofs. Are these the projection of $\\hat{G}$ on $\\mathcal{L}$ and the restriction of the domain of $\\tilde{F}$ to $\\mathcal{L}$?\n\n- $\\mathcal{L}$ is defined as $\\mathrm{aff}(K)$ in Lemma 9 and the proof of Theorem 1, but in Algorithm 2 it is defined as $\\mathrm{aff}(K) - z_1$.\n\n- I recommend motivating why it is important to restrict the oracle to query only feasible points.\n\n- It would be good to also include known lower bounds for settings where they are available.\n\n- h is not defined in Table 1. \n\n- In the appendix, it is helpful to remind the reader of what different terms refer to or to have a table listing them.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}