Title: From Linear to Linearizable Optimization:  A Novel Framework with Applications to Stationary and Non-stationary DR-submodular Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: convex optimization

DR-submodular optimization
Abstract: This paper introduces the notion of upper-linearizable/quadratizable functions, a class that extends concavity and DR-submodularity in various settings, including monotone and non-monotone cases over different types of convex sets. A general meta-algorithm is devised to convert algorithms for linear/quadratic maximization into ones that optimize upper-linearizable/quadratizable functions, offering a unified approach to tackling concave and DR-submodular optimization problems. The paper extends these results to multiple feedback settings, facilitating conversions between semi-bandit/first-order feedback and bandit/zeroth-order feedback, as well as between first/zeroth-order feedback and semi-bandit/bandit feedback. Leveraging this framework, new algorithms are derived using existing results as base algorithms for convex optimization, improving upon state-of-the-art results in various cases. Dynamic and adaptive regret guarantees are obtained for DR-submodular maximization, marking the first algorithms to achieve such guarantees in these settings. Notably, the paper achieves these advancements with fewer assumptions compared to existing state-of-the-art results, underscoring its broad applicability and theoretical contributions to non-convex optimization.

From Linear to Linearizable Optimization: A Novel Framework with Applications to Stationary and Non-stationary DR-submodular Optimization

Abstract

This paper introduces the notion of upper-linearizable/quadratizable functions, a class that extends concavity and DR-submodularity in various settings, including monotone and non-monotone cases over different types of convex sets. A general meta-algorithm is devised to convert algorithms for linear/quadratic maximization into ones that optimize upper-linearizable/quadratizable functions, offering a unified approach to tackling concave and DR-submodular optimization problems. The paper extends these results to multiple feedback settings, facilitating conversions between semi-bandit/first-order feedback and bandit/zeroth-order feedback, as well as between first/zeroth-order feedback and semi-bandit/bandit feedback. Leveraging this framework, new algorithms are derived using existing results as base algorithms for convex optimization, improving upon state-of-the-art results in various cases. Dynamic and adaptive regret guarantees are obtained for DRsubmodular maximization, marking the first algorithms to achieve such guarantees in these settings. Notably, the paper achieves these advancements with fewer assumptions compared to existing state-of-the-art results, underscoring its broad applicability and theoretical contributions to non-convex optimization.

1 Introduction

Overview: The prominence of optimizing continuous adversarial γ -weakly up-concave functions (with DR-submodular and concave functions as special cases) has surged in recent years, marking a crucial subset within the realm of non-convex optimization challenges, particularly in the forefront of machine learning and statistics. This problem has numerous real-world applications, such as revenue maximization, mean-field inference, recommendation systems [4, 20, 30, 12, 24, 18, 26]. This problem is modeled as a repeated game between an optimizer and an adversary. In each round, the optimizer selects an action, and the adversary chooses a γ -weakly up-concave reward function. Depending on the scenario, the optimizer can then query this reward function either at any arbitrary point within the domain (called full information feedback) or specifically at the chosen action (called semi-bandit/bandit feedback), where the feedback can be noisy/deterministic. The performance metric of the algorithm is measured with multiple regret notions - static adversarial regret, dynamic regret, and adaptive regret. The algorithms for the problem are separated into the ones that use a projection operator to project the point to the closest point in the domain, and the projectionfree methods that replace the projection with an alternative such as Linear Optimization Oracles (LOO) or Separation Oracles (SO). This interactive framework introduces a range of significant challenges, influenced by the characteristics of the up-concave function (monotone/non-monotone), the constraints imposed, the nature of the queries, projection-free/projection-based algorithms, and the different regret definitions.

Vaneet Aggarwal

This table compares different static regret results for the online up-concave maximization. The logarithmic terms in regret are ignored. Here h := min z ∈K ∥ z ∥ ∞ . Our algorithm is projection-free and use a separation oracle. The rows marked with ‡ use gradient ascent, requiring potentially computationally expensive projections. The rows marked with ( ∗ ) denote results that could be considered special case of our framework. In particular, we obtain those results if we use Online Gradient Ascent instead of SO -OGA as the base algorithm in Corollary 7. Note that the result of [39], marked by ‡‡ , uses a convex optimization subroutine in each iteration, which could potentially be more expensive than projection and therefore not considered a projection-free result. It is also the only existing result, in all the tables, that outperforms ours. We note that stochastic results can be used in deterministic, and bandit/semi-bandit in full information and thus cases where our result is not added, our result improves state-of-the-art projection free result because of the result in less information setup. All results assume that functions are Lipschitz. Except for our results on monotone functions over general convex sets, all results also assume differentiability. All previous results assume that functions are DR-submodular, while we only require up-concavity. Results of [35] and [39] also assume functions are smooth, i.e., their gradients are Lipschitz.

In this paper, we present a comprehensive approach to solving adversarial up-concave optimization problems, encompassing different feedback types (including bandit, semi-bandit and fullinformation feedback), characteristics of the up-concave function and constraint region, projectionfree/projection-based algorithms, and regret definitions. While the problem has been studied in many special cases, the main contribution of this work is a framework that is based on a novel notion of the function class being upper-linearizable (or upper-quadratizable). We design a metaalgorithm that converts certain algorithms designed for online linear maximization to algorithms capable of handling upper-linearizable function classes. This allows us to reduce the problem of up-concave maximization in three different settings to online linear maximization and obtain corresponding regret bounds. In particular, our results include monotone γ -weakly up-concave functions over general convex set, monotone γ -weakly up-concave functions over convex sets containing the origin and non-monotone up-concave functions. While the above result is for first order feedback, we then derive multiple results that increase the applicability of the above results. We extend the applicability of FOTZO and STB algorithm introduced in [34] to our setting which allows us to convert algorithms for first-order/semi-bandit feedback into algorithms for zeroth-order/bandit feedback. We also design a meta-algorithm that allows us to convert algorithms that require full-information feedback into algorithms that only require semi-bandit/bandit feedback.

We demonstrate the usefulness of results through two applications as described in the following. In the first application, we use the SO -OGD Algorithm in [17] as the base algorithm for online linear optimization, which is a projection-free algorithm. Using this, we first obtain the adaptive regret (and

therefore also static regret) guarantees for the three setups of DR-submodular (or more generally, upconcave) optimization with semi-bandit feedback/first order feedback in the respective cases. Then, the meta-algorithms for conversion of first-order/semi-bandit to zeroth-order/bandit are used to get result with zeroth-order/bandit feedback. In the cases where the algorithms are full-information and not (semi-)bandit, we use another meta-algorithm to obtain algorithms in (semi-)bandit feedback setting. In the next application, we use the 'Improved Ader' algorithm of [43] which is a projection based algorithm providing dynamic regret guarantees for the convex optimization. Afterwards, the same approach as above are used to obtain the results in the three scenarios of up-concave optimization with first-order feedback.

Technical Novelty: The main technical novelties in this work are as follows.

We proposes a novel notion of linearizable/quadratizable functions and extend the metaalgorithm framework of [34] from convex functions to linearizable/quadratizable functions. This allows us to relates a large class of algorithms and regret guarantees for optimization of linear/quadratic functions to that for linearizable/quadratizable functions.

We show that the class of quadratizable function optimization is general, and includes not only concave, but up-concave optimization in several cases. For some of the cases, this proof uses a generalization of the idea of boosting ([46, 48]) which was proposed for DR-submodular maximization, as mentioned in Corollaries 2 and 3.

We design a new meta-algorithm, namely SFTT , that captures the idea of random permutations (sometimes referred to as blocking) as used in several papers such as [45, 47, 35]. While previous works used this idea in specific settings, our meta-algorithm is applicable in general settings.

Wenote the generality of the above results in this paper. Our results are general in the following three aspects:

a) In this work, we improve results for projection-free static regret guarantees for DRsubmodular optimization in all considered cases and obtain the first results for dynamic and adaptive regret. Moreover, these guarantees follow from existing algorithms for the linear optimization, using only the statement of the regret bounds and simple properties of the algorithms.

b) We consider 3 classes of DR-submodular functions in this work. However, to extend these results to another function class, all one needs to do is to (i) prove that the function class is quadratizable; and (ii) provide an unbiased estimator of g (as described in Equation 1).

c) We consider 2 different feedback types in offline setting (first/zero order) and 4 types of feedback in the online setting (first/zero order and full-information/trivial query). Converting results between different cases is obtained through meta-algorithms and guarantees for the meta-algorithms which only relies on high level properties of the base algorithms (See Theorems 5, 7, 6 and 8)

Key contributions: The key contributions in this work are summarized as follows.

We formulate the notion of upper-quadratizable/upper-linearizeble functions, which is a class that generalizes the notion of strong-concavity/concavity and also DR-submodularity in several settings. In particular, we demonstrate the the following function classes are upperquadratizable: (i) monotone γ -weakly µ -strongly DR-submodular functions with curvature c over general convex sets, (ii) monotone γ -weakly DR-submodular functions over convex sets containing the origin, and (iii) non-monotone DR-submodular optimization over general convex sets.

We provide a general meta-algorithm that converts algorithms for linear/quadratic maximization to algorithms that maximize upper-quadratizable functions. This results is a unified approach to maximize both concave functions and DR-submodular functions in several settings.

While the above provides results for semi-bandit feedback (for monotone DR-submodular optimization over general convex sets) and first-order feedback (for monotone DR-submodular optimization over convex sets containing the origin, and non-monotone DR-submodular optimization over general convex sets), the results could be extended to more general feedback settings. Four meta algorithms are provided that relate semi-bandit/first-order feedback to bandit/zeroth order feedback; that relate; first order to deterministic zeroth order; and that relate

first/zeroth order feedback to semi-bandit/bandit feedback. Together they allow us to obtain results in 5 feedback settings (first/zeroth order full-information and semi-bandit/bandit; and deterministic zeroth order). We also discuss a meta-algorithm to convert online results to offline guarantees.

The above framework is applied using different algorithms as the base algorithms for linear optimization. SO -OGD [17] is a projection-free algorithm using separation oracles that provides adaptive regret guarantees for online convex optimization. We use our framework to cover the 18 cases in Table 1. We improve the regret guarantees for the previous SOTA projection-free algorithms in all the cases. If we also allow comparisons with the algorithms that are not projection-free, we still improve the SOTA results in 12 cases and match the SOTA in 5 cases.

Using our framework, we convert online results using SO -OGD to offline results to obtain 6 projection free algorithms described in Table 2. We improve the regret guarantees for the previous SOTA projection-free algorithms in all the cases, except for deterministic first order feedback where existing results are already SOTA. If we also allow comparisons with the algorithms that are not projection-free, we still improve the SOTA results in 6 cases and match the SOTA in the remaining 3 cases.

Weuse our framework to convert the adaptive regret guarantees of SO -OGD to obtain projectionfree algorithms with adaptive regret bounds that cover all cases in Table 3. Our results are first algorithms with adaptive regret guarantee for online DR-submodular maximization.

'Improved Ader' [43] is an algorithm providing dynamic regret guarantees for online convex optimization. We use our framework to obtain 6 algorithms which provide the dynamic regret guarantees as shown in Table 3. Our results are first algorithms with dynamic regret guarantee for online DR-submodular maximization.

For monotone γ -weakly functions with bounded curvature over general convex sets, we improve the approximation ratio (See Lemma 1).

As mentioned in the descriptions of the tables, in all cases considered, whenever there is another existing result, we obtain our results using fewer assumptions than the existing SOTA.

2 Problem Setup and Definitions

For a set D ⊆ R d , we define its affine hull aff( D ) to be the set of α x +(1 -α ) y for all x , y in K and α ∈ R . The relative interior of D is defined as relint( D ) := { x ∈ D | ∃ r > 0 , B d r ( x ) ∩ aff( D ) ⊆ D} . For any u ∈ K T , we define the path length P T ( u ) := ∑ T -1 i =1 ∥ u i -u i +1 ∥ . Given µ ≥ 0 and 0 < γ ≤ 1 , we say a differentiable function f : K → R is µ -strongly γ -weakly up-concave if it is µ -strongly γ -weakly concave along positive directions. Specifically if, for all x ≤ y in K , we have

We say ˜ ∇ f : K → R d is a µ -strongly γ -weakly up-super-gradient of f if for all x ≤ y in K , the above holds with ˜ ∇ instead of ∇ . We say f is µ -strongly γ -weakly up-concave if it is continuous and it has a µ -strongly γ -weakly up-super-gradient. When it is clear from the context, we simply refer to ˜ ∇ f as an up-super-gradient for f . When γ = 1 and the above inequality holds for all x , y ∈ K , we say f is µ -strongly concave. A differentiable function f : K → R is called γ -weakly continuous DR-submodular if for all x ≤ y , we have ∇ f ( x ) ≥ γ ∇ f ( y ) . It follows that any γ -weakly continuous DR-submodular functions is γ -weakly up-concave. We refer to Appendix B for more details.

Given a continuous monotone function f : K → R , its curvature is defined as the smallest number c ∈ [0 , 1] such that f ( y + z ) -f ( y ) ≥ (1 -c )( f ( x + z ) -f ( x )) , for all x , y ∈ K and z ≥ 0 such that x + z , y + z ∈ K . We define the curvature of a function class F as the supremum of the curvature of functions in F .

Online optimization problems can be formalized as a repeated game between an agent and an adversary. The game lasts for T rounds on a convex domain K where T and K are known to both players. In t -th round, the agent chooses an action x t from an action set K ⊆ R d , then the adversary chooses a loss function f t ∈ F and a query oracle for the function f t . Then, for 1 ≤ i ≤ k t ,

This table compares the different results for the number of oracle calls (complexity) within the constraint set for up-concave maximization. We refer to [36] for a more comprehensive table that includes results for deterministic first order feedback. Here h := min z ∈K ∥ z ∥ ∞ and γ ′ := γ +1 /γ .

‡ [20], [46] and [48] use gradient ascent, requiring potentially computationally expensive projections.

All previous results assume that functions are differentiable, DR-submodular, Lipschitz and smooth (i.e., their gradients are Lipschitz). Result of [19] also requires the function Hessians to be Lipschitz. It also requires the density of the stochastic oracle to be known and the log of density to be 4 times differentiable with bounded 4th derivatives. We only require the functions to be up-concave, differentiable and Lipschitz, except for results on monotone functions over general convex sets where we do not need differentiability.

the agent chooses a points y t,i and receives the output of the query oracle. The precise definition of agent (Ω A , A action , A query ) is given in Appendix B, with the query oracle being any of stochastic/deterministic first/zeroth order or semi-bandit/bandit.

An adversary Adv is a set of realized adversaries B = ( B 1 , · · · , B T ) , where each B t maps ( x 1 , · · · , x t ) ∈ K T to ( f t , Q t ) where f t ∈ F and Q t is a query oracle for f t . Adversaries can be oblivious ( B t are constant and independent of ( x 1 , · · · , x t ) ), weakly adaptive ( B t are independent of x t ), or fully adaptive (no restrictions). We use Adv f i ( F ) to denote the set of all possible realized adversaries with deterministic i -th order oracles. If the oracle is instead stochastic and bounded by B , we use Adv f i ( F , B ) to denote such an adversary. Finally, we use Adv o i ( F ) and Adv o i ( F , B ) to denote all oblivious realized adversaries with i -th order deterministic and stochastic oracles, respectively. In order to handle different notions of regret with the same approach, for an agent A , adversary Adv , compact set U ⊆ K T , approximation coefficient 0 < α ≤ 1 and 1 ≤ a ≤ b ≤ T , we define regret as R A α, Adv ( U )[ a, b ] :=

sup B∈ Adv E [ α max u =( u 1 , ··· , u T ) ∈U ∑ b t = a f t ( u t ) -∑ b t = a f t ( x t ) ] , where the expectation in the definition of the regret is over the randomness of the algorithm and the query oracle. We use the notation R A α, B ( U )[ a, b ] := R A α, Adv ( U )[ a, b ] when Adv = {B} is a singleton. We may drop α when it is equal to 1. When α < 1 , we often assume that the functions are nonnegative. Static adversarial regret or simply adversarial regret corresponds to a = 1 , b = T and U = K T ⋆ := { ( x , · · · , x ) | x ∈ K} . When a = 1 , b = T and U contains only a single element then it is referred to as the dynamic regret [51, 43]. Adaptive regret is defined as max 1 ≤ a ≤ b ≤ T R A α, Adv ( K T ⋆ )[ a, b ] [23]. We drop a , b and U when the statement is independent of their value or their value is clear from the context.

3 Formulation of Upper-Quadratizable Functions and Regret Relation to that of Quadratic Functions

Let K ⊆ R d be a convex set, F be a function class over K . We say the function class F is upperquadratizable if there are maps g : F ×K → R d and h : K → K and constants µ ≥ 0 , 0 < α ≤ 1

This table includes different results for non-stationary up-concave maximization, while no prior results exist in this setup to the best of our knowledge. The results for adaptive regret are projection-free and use a separation oracle while results for dynamic regret use convex projection. Note that full-information algorithms with deterministic feedback require 2 queries per function while the ones with stochastic feedback only require one, at the cost of higher regret.

and β > 0 such that 2

As a special case, when µ = 0 , we say F is upper-linearizable . We use the notation F µ, g to denote the class of functions q ( y ) := ⟨ g ( f, x ) , y -x ⟩ -µ 2 ∥ y -x ∥ 2 : K → R , for all f ∈ F and x ∈ K . Similarly, for any B 1 > 0 , we use the notation Q µ [ B 1 ] to denote the class of functions q ( y ) := ⟨ o , y -x ⟩ -µ 2 ∥ y -x ∥ 2 : K → R , for all x ∈ K and o ∈ B d B 1 ( 0 ) . A similar notion of lower-quadratizable/linearizable may be similarly defined for minimization problems such as convex minimization. 3

Wesay an algorithm G is a first order query algorithm for g if, given a point x ∈ K and a first order query oracle for f , it returns (a possibly unbiased estimate of) g ( f, x ) . We say G is bounded by B 1 if the output of G is always within the ball B d B 1 ( 0 ) and we call it trivial if it simply returns the output of the query oracle at x .

Recall that an online agent A is composed of action function A action and query function A query . Informally, given an online algorithm A with semi-bandit feedback, we may think of A ′ := OMBQ ( A , G , h ) as the online algorithm with ( A ′ ) action ≈ h ( A action ) and ( A ′ ) query ≈ G . As a special case, when h = Id and G is trivial, we have A ′ = A .

Algorithm 1: Online Maximization By Quadratization - OMBQ ( A , G , h )

Input :

horizon

T

, semi-bandit

algorithm

A

, query algorithm

G

for

g

, the map

h

:

K → K

for

t

= 1

,

2

, . . . , T

do

Play h ( x t ) where x t is the action chosen by A

The adversary selects f t and a first order query oracle for f t

Run

G

with access to

x

t

and the

query oracle for

f

t

to calculate

o

t

Return

o

t

as the output of the query

oracle to

A

end

Theorem 1. Let A be algorithm for online optimization with semi-bandit feedback. Also let F be a function class over K that is quadratizable with µ ≥ 0 and maps g : F ×K → R d and h : K → K , let G be a query algorithm for g and let A ′ = OMBQ ( A , G , h ) . Then the following are true.

′

1.If G returns the exact value of g , then we have R A α, Adv f 1 ( F ) ≤ β R A 1 , Adv f 1 ( F µ, g ) . 2.On the other hand, if G returns an unbiased estimate of g and the output of G is bounded by B 1 , then we have R A ′ α, Adv o 1 ( F ,B 1 ) ≤ β R A 1 , Adv f 1 ( Q µ [ B 1 ]) .

As a special case, when f is concave, we may choose α = β = 1 , h = Id , and g ( f, x ) to be a super-gradient of f at x . In this case, Theorem 1 reduces to the concave version of Theorems 2 and 5 in [34].

4 Up-concave function optimization is upper-quadratizable function optimization

In this section, we study three classes of up-concave functions and show that they are upperquadratizable. We further use this property to obtain meta-algorithms that convert algorithms for quadratic optimization into algorithms for up-concave maximization.

4.1 Monotone up-concave optimization over general convex sets

For differentiable DR-submodular functions, the following lemma is proven for the case γ = 1 in Lemma 2 in [13] and for the case µ = 0 in [20] (See Inequality 7.5 in the arXiv version). We show the result for general µ -strongly γ -weakly up-concave function with curvature bounded by c , See Appendix D for proof.

Lemma 1. Let f : [0 , 1] d → R be a non-negative monotone µ -strongly γ -weakly up-concave function with curvature bounded by c . Then, for all x , y ∈ [0 , 1] d , we have

where ˜ ∇ f is an up-super-gradient for f .

Further, we show that any semi-bandit feedback online linear optimization algorithm for fully adaptive adversary is also an online up-concave optimization algorithm.

Theorem 2. Let K ⊆ [0 , 1] d be a convex set, let µ ≥ 0 , γ ∈ (0 , 1] , c ∈ [0 , 1] and let A be algorithm for online optimization with semi-bandit feedback. Also let F be an M 1 -Lipschitz function class over K where every f ∈ F may be extended to a monotone µ -strongly γ -weakly up-concave function curvature bounded by c defined over [0 , 1] d . Then, for any B 1 ≥ M 1 , we have

These results follows immediately from Theorem 1 and Lemma 1. Note that it is important to assume that every function in F may be extended to a non-negative up-concave function over [0 , 1] d for Lemma 1 to be applied.

Corollary 1.

The results of [20], [8] and [13] on monotone

continuous

DR-submodular maximiza-

tion over general convex sets may be thought of as special cases of Theorem 2 when A is the online gradient ascent algorithm.

4.2 Monotone up-concave optimization over convex sets containing the origin

The following lemma is proven for differentiable DR-submodular functions in Theorem 2 and Proposition 1 of [46]. The proof works for general up-concave functions as well. We include a proof in Appendix E for completeness.

Algorithm 2: Boosted Query oracle for Monotone up-concave functions over convex sets containing the origin BQM0

Input : First order query oracle, point x Sample z ∈ [0 , 1] according to Equation (2) Return the output of the query oracle at z ∗ x

Lemma 2. Let f : [0 , 1] d → R be a non-negative monotone γ -weakly up-concave differentiable function and let F : [0 , 1] d → R be the function defined by

Then F is differentiable and, if the random variable Z ∈ [0 , 1] is defined by the law

then we have E [ ∇ f ( Z ∗ x ) ] = ∇ F ( x ) . Moreover, we have (1 -e -γ ) f ( y ) -f ( x ) ≤ 1 -e -γ γ ⟨∇ F ( x ) , y -x ⟩ .

Theorem 3. Let K ⊆ [0 , 1] d be a convex set containing the origin, let γ ∈ (0 , 1] and let A be algorithm for online optimization with semi-bandit feedback. Also let F be a function class over K where every f ∈ F is the restriction of a monotone γ -weakly up-concave function defined over [0 , 1] d to the set K . Assume F is differentiable and M 1 -Lipschitz for some M 1 > 0 . Then, for any B 1 ≥ M 1 , we have

where A ′ = OMBQ ( A , BQM0 , Id) .

This result now follows immediately from Theorem 1 and Lemma 2.

Corollary 2. The result of [46] in the online setting (when there is no delay) may be seen as an application of Theorem 3 when A is chosen to be online gradient ascent.

4.3 Non-monotone up-concave optimization over general convex sets

The following lemma is proven for differentiable DR-submodular functions in Corollary 2, Theorem 4 and Proposition 2 of [48]. The arguments works for general up-concave functions as well. We include a proof in Appendix F for completeness.

Lemma 3. Let f : [0 , 1] d → R be a non-negative continuous up-concave differentiable function and let x ∈ K . Define F : [0 , 1] d → R as the function F ( x ) := ∫ 1 0 2 3 z (1 -z 2 ) 3 ( f ( z 2 ∗ ( x -x ) + x ) -f ( x ) ) dz . Then F is differentiable and, if the random variable Z ∈ [0 , 1] is defined by the law

then we have E [ ∇ f ( Z 2 ∗ ( x -x ) + x ) ] = ∇ F ( x ) . Moreover, we have

Theorem 4. Let K ⊆ [0 , 1] d be a convex set, u ∈ K , h := ∥ u ∥ ∞ and A be algorithm for online optimization with semi-bandit feedback. Also let F be a function class over K where every f ∈ F is the restriction of an up-concave function defined over [0 , 1] d to the set K . Assume F is differentiable and M 1 -Lipschitz for some M 1 > 0 . Then, for any B 1 ≥ M 1 and A ′ = OMBQ ( A , BQN , x ↦→ x t + x 2 ) ,

These results now follows immediately from Theorem 1 and Lemma 3.

Corollary 3. The result of [48] in the online setting without delay may be seen as an application of Theorem 4 when A is chosen to be online gradient ascent.

Algorithm 3: Boosted Query oracle for Non-monotone up-concave functions over general convex sets BQN

Input : First order query oracle, point x Sample z ∈ [0 , 1] according to Equation 3 Return the output of the query oracle at z 2 ∗ ( x -x ) + x

5 Meta algorithms for other feedback cases

In this section, we study several meta-algorithms that allow us to convert between different feedback types and also convert results from the online setting to the offline setting.

First order/semi-bandit to zeroth order/bandit feedback: In this section we discuss metaalgorithms that convert algorithms designed for first order feedback into algorithms that can handle zeroth order feedback. These algorithms and results are generalization of similar results in [34] to the case where α < 1 .

We choose a point c ∈ relint( K ) and a real number r > 0 such that aff( K ) ∩ B d r ( c ) ⊆ K . Then, for any shrinking parameter 0 ≤ δ < r , we define ˆ K δ := (1 -δ r ) K + δ r c . For a function f : K → R defined on a convex set K ⊆ R d , its δ -smoothed version ˆ f δ : ˆ K δ → R is given as

where L 0 = aff( K ) -x , for any x ∈ K , is the linear space that is a translation of the affine hull of K and v is sampled uniformly at random from the k = dim( L 0 ) -dimensional ball L 0 ∩ B d 1 ( 0 ) . Thus, the function value ˆ f δ ( x ) is obtained by 'averaging' f over a sliced ball of radius δ around x . For a function class F over K , we use ˆ F δ to denote { ˆ f δ | f ∈ F } . We will drop the subscript δ when there is no ambiguity (See Appendix G for the description of the algorithms and the proof.).

Theorem 5. Let F be an M 1 -Lipschitz function class over a convex set K and choose c and r as described above and let δ < r . Let U ⊆ K T be a compact set and let ˆ U = (1 -δ r ) U + δ r c . Assume A is an algorithm for online optimization with first order feedback. Then, if A ′ = FOTZO ( A ) where FOTZO is described by Algorithm 5 and 0 < α ≤ 1 , we have

On the other hand, if we assume that A is semi-bandit, then the same regret bounds hold with A ′ = STB ( A ) , where STB is described by Algorithm 6.

Theorem 6. Under the assumptions of Theorem 5, if A ′ = FOTZO -2P ( A ) where FOTZO -2P is described by Algorithm 7 and 0 < α ≤ 1 , we have

Full information to trivial query: In this section, we discuss a meta-algorithm that converts algorithms that require full-information feedback into algorithms that have a trivial query oracle. In particular, it converts algorithms that require first-order full-information feedback into semi-bandit algorithms and algorithms that require zeroth-order full-information feedback into bandit algorithms.

Here we assume that A query does not depend on the observations in the current round. If the number of queries k t is not constant for each time-step, we simply assume that A queries extra points and then discards them, so that we obtain an algorithm that queries exactly K points at each time-step, where K does not depend on t . We say a function class F is closed under convex combination if for any f 1 , · · · , f k ∈ F and any δ 1 , · · · , δ k ≥ 0 with ∑ i δ i = 1 , we have ∑ i δ i f i ∈ F .

Theorem 7. Let A be an online optimization algorithm with full-information feedback and with K queries at each time-step where A query does not depend on the observations in the current round and A ′ = SFTT ( A ) . Then, for any M 1 -Lipschitz function class F that is closed under convex combination and any B 1 ≥ M 1 , 0 < α ≤ 1 and 1 ≤ a ≤ b ≤ T , let a ′ = ⌊ ( a -1) /L ⌋ + 1 , b ′ = ⌈ b/L ⌉ , D = diam( K ) and let { T } and { T/L } denote the horizon of the adversary. Then, we have

This result (proof in Appendix H) is based on the idea of random permutations used in [45, 47, 35].

Online to Offline: An offline optimization problem can be though of as an instance of online optimization where the adversary picks the same function and query oracle at each round. Moreover, instead of regret, the performance of the algorithm is measured by sample complexity, i.e., the minimum number of queries required so that the expected error from the α -approximation of the optimal value is less than ϵ .

Conversions of online algorithms to offline are referred to online-to-batch techniques and are well-known in the literature (See [38]). A simple approach is to simply run the online algorithm and if the actions chosen by the algorithm are x 1 , · · · , x T , return x t for 1 ≤ t ≤ T with probability 1 /T . We use OTB to denote the meta-algorithm that uses this approach to convert online algorithms to offline algorithms. The following theorem is a corollary which we include for completion. (See Appendix I for the proof.)

Theorem 8. Let A be an online algorithm that queries no more than K = T θ times per timestep that obtains an α -regret bound of O ( T η ) over an oblivious adversary Adv . Then the sample complexity of OTB ( A ) over { ( f, Q f ) | (( f, Q f ) , · · · , ( f, Q f )) ∈ Adv } is O ( ϵ -1+ θ 1 -η ) .

6 Applications

Figure 6 captures the applications that are mentioned in Tables 1, 2 and 3. The exact statements are stated in Corollaries 7 and 8 in the Appendix. To obtain a result from the graph, let A be one of SO -OGA or IA and select a directed path that has the following properties: (i) The path starts at one of the three nodes on the left. (ii) The path must be at least of length 1 and the edges must be the same color. (iii) If A is IA , the path should not contain SFTT or OTB .

For example, if A = SO -OGA and the path starts at the middle node on the left, then passes through OMBQ , FOTZO , SFTT , we get SFTT ( FOTZO ( OMBQ ( SO -OGA , BQM0 , Id))) , which is a projection-free algorithm (using separation oracles) with bandit feedback for

monotone up-concave functions over convex sets that contain the origin. As mentioned in Table 3 and Corollary 7-(c), the adaptive regret of this algorithm is of order O ( T 4 / 5 ) . Note that the text written in the three nodes on the left correspond to the inputs of the meta-algorithm OMBQ . Also note that the color red corresponds to the setting where G is a trivial query algorithm which means that the output of OMBQ is semi-bandit.

7 Conclusions

In this work, we have presented a comprehensive framework for addressing optimization problems involving upper-quadratizable functions, encompassing both concave and DR-submodular functions across various settings and feedback types. Our contributions include the formulation of upperquadratizable functions as a generalized class, the development of meta-algorithms for algorithmic conversions, and the derivation of new algorithms with improved static/ dynamic/ adaptive regret guarantees. Exploring more subset of classes of upper-quadratizable functions where such a framework could be applied is an important future direction.

References

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations Research , 63(5):1227-1244, 2015.

An Bian, Kfir Levy, Andreas Krause, and Joachim M Buhmann. Continuous DR-submodular maximization: Structure and algorithms. In Advances in Neural Information Processing Systems , 2017.

Andrew An Bian, Baharan Mirzasoleiman, Joachim Buhmann, and Andreas Krause. Guaranteed Non-convex Optimization: Submodular Maximization over Continuous Domains. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , April 2017.

Yatao Bian, Joachim Buhmann, and Andreas Krause. Optimal continuous DR-submodular maximization and applications to provable mean field inference. In Proceedings of the 36th International Conference on Machine Learning , June 2019.

Gruia Calinescu, Chandra Chekuri, Martin P´ al, and Jan Vondr´ ak. Maximizing a monotone submodular function subject to a matroid constraint. SIAM Journal on Computing , 40(6):17401766, 2011.

Chandra Chekuri, T.S. Jayram, and Jan Vondrak. On multiplicative weight updates for concave and submodular function maximization. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science , pages 201-210, 2015.

Lin Chen, Christopher Harshaw, Hamed Hassani, and Amin Karbasi. Projection-free online optimization with stochastic gradient: From convexity to submodularity. In Proceedings of the 35th International Conference on Machine Learning , July 2018.

Lin Chen, Hamed Hassani, and Amin Karbasi. Online continuous submodular maximization. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , April 2018.

Lin Chen, Mingrui Zhang, and Amin Karbasi. Projection-free bandit convex optimization. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , pages 2047-2056. PMLR, 2019.

Shengminjie Chen, Donglei Du, Wenguo Yang, Dachuan Xu, and Suixiang Gao. Continuous non-monotone DR-submodular maximization with down-closed convex constraint. arXiv preprint arXiv:2307.09616 , July 2023.

Amit Daniely, Alon Gonen, and Shai Shalev-Shwartz. Strongly adaptive online learning. In Proceedings of the 32nd International Conference on Machine Learning , pages 1405-1411. PMLR, 2015.

Josip Djolonga and Andreas Krause. From map to marginals: Variational inference in Bayesian submodular models. Advances in Neural Information Processing Systems , 2014.

Maryam Fazel and Omid Sadeghi. Fast first-order methods for monotone strongly drsubmodular maximization. In SIAM Conference on Applied and Computational Discrete Algorithms (ACDA23) , 2023.

Yuval Filmus and Justin Ward. A tight combinatorial algorithm for submodular maximization subject to a matroid constraint. In 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science , pages 659-668, 2012.

Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms , pages 385-394, 2005.

Dan Garber and Ben Kretzu. Improved regret bounds for projection-free bandit convex optimization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages 2196-2206. PMLR, 2020.

Dan Garber and Ben Kretzu. New projection-free algorithms for online convex optimization with adaptive regret guarantees. In Proceedings of Thirty Fifth Conference on Learning Theory , pages 2326-2359. PMLR, 2022.

Shuyang Gu, Chuangen Gao, Jun Huang, and Weili Wu. Profit maximization in social networks and non-monotone DR-submodular maximization. Theoretical Computer Science , 957:113847, 2023.

Hamed Hassani, Amin Karbasi, Aryan Mokhtari, and Zebang Shen. Stochastic conditional gradient++: (non)convex minimization and continuous submodular maximization. SIAM Journal on Optimization , 30(4):3315-3344, 2020.

Hamed Hassani, Mahdi Soltanolkotabi, and Amin Karbasi. Gradient methods for submodular maximization. In Advances in Neural Information Processing Systems , 2017.

Elad Hazan and Satyen Kale. Projection-free online learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning , ICML'12, pages 1843-1850. Omnipress, 2012.

Elad Hazan and Edgar Minasyan. Faster projection-free online learning. In Proceedings of Thirty Third Conference on Learning Theory , pages 1877-1893. PMLR, 2020.

Elad Hazan and C. Seshadhri. Efficient learning algorithms for changing environments. In Proceedings of the 26th Annual International Conference on Machine Learning , ICML '09, pages 393-400. Association for Computing Machinery, 2009.

Shinji Ito and Ryohei Fujimaki. Large-scale price optimization via network flow. Advances in Neural Information Processing Systems , 2016.

Duksang Lee, Nam Ho-Nguyen, and Dabeen Lee. Non-smooth, h \ 'older-smooth, and robust submodular maximization. arXiv preprint arXiv:2210.06061 , 2023.

Yuanyuan Li, Yuezhou Liu, Lili Su, Edmund Yeh, and Stratis Ioannidis. Experimental design networks: A paradigm for serving heterogeneous learners under networking constraints. IEEE/ACM Transactions on Networking , 2023.

Yucheng Liao, Yuanyu Wan, Chang Yao, and Mingli Song. Improved Projection-free Online Continuous Submodular Maximization. arXiv preprint arXiv:2305.18442 , May 2023.

Zhou Lu, Nataly Brukhim, Paula Gradu, and Elad Hazan. Projection-free adaptive regret with membership oracles. In Proceedings of The 34th International Conference on Algorithmic Learning Theory , pages 1055-1073. PMLR, 2023.

Zakaria Mhammedi. Efficient projection-free online convex optimization with membership oracle. In Proceedings of Thirty Fifth Conference on Learning Theory , pages 5314-5390. PMLR, 2022.

Siddharth Mitra, Moran Feldman, and Amin Karbasi. Submodular+ concave. Advances in Neural Information Processing Systems , 2021.

Aryan Mokhtari, Hamed Hassani, and Amin Karbasi. Stochastic conditional gradient methods: From convex minimization to submodular maximization. The Journal of Machine Learning Research , 21(1):4232-4280, 2020.

Loay Mualem and Moran Feldman. Resolving the approximability of offline and online nonmonotone DR-submodular maximization over general convex sets. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , April 2023.

Rad Niazadeh, Negin Golrezaei, Joshua Wang, Fransisca Susan, and Ashwinkumar Badanidiyuru. Online learning via offline greedy algorithms: Applications in market design and optimization. Management Science , 69(7):3797-3817, July 2023.

Mohammad Pedramfar and Vaneet Aggarwal. A unified framework for analyzing metaalgorithms in online convex optimization. arXiv preprint arXiv:2402.08621 , 2024.

Mohammad Pedramfar, Yididiya Y. Nadew, Christopher John Quinn, and Vaneet Aggarwal. Unified projection-free algorithms for adversarial DR-submodular optimization. In The Twelfth International Conference on Learning Representations , 2024.

Mohammad Pedramfar, Christopher Quinn, and Vaneet Aggarwal. A unified approach for maximizing continuous γ -weakly DR-submodular functions. optimization-online preprint optimization-online:25915 , 2024.

Mohammad Pedramfar, Christopher John Quinn, and Vaneet Aggarwal. A unified approach for maximizing continuous DR-submodular functions. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2):107-194, 2012.

Zongqi Wan, Jialin Zhang, Wei Chen, Xiaoming Sun, and Zhijie Zhang. Bandit multi-linear dr-submodular maximization and its applications on adversarial submodular bandits. In International Conference on Machine Learning , 2023.

Yibo Wang, Wenhao Yang, Wei Jiang, Shiyin Lu, Bing Wang, Haihong Tang, Yuanyu Wan, and Lijun Zhang. Non-stationary projection-free online learning with dynamic and adaptive regret guarantees. Proceedings of the AAAI Conference on Artificial Intelligence , 38(14):1567115679, 2024.

Bryan Wilder. Equilibrium computation and robust optimization in zero sum games with submodular structure. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Jiahao Xie, Zebang Shen, Chao Zhang, Boyu Wang, and Hui Qian. Efficient projection-free online methods with stochastic recursive gradient. Proceedings of the AAAI Conference on Artificial Intelligence , 34(4):6446-6453, 2020.

Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Lijun Zhang, Tianbao Yang, jin, and Zhi-Hua Zhou. Dynamic regret of strongly adaptive methods. In Proceedings of the 35th International Conference on Machine Learning , pages 5882-5891. PMLR, 2018.

Mingrui Zhang, Lin Chen, Hamed Hassani, and Amin Karbasi. Online continuous submodular maximization: From full-information to bandit feedback. In Advances in Neural Information Processing Systems , volume 32, 2019.

Qixin Zhang, Zengde Deng, Zaiyi Chen, Haoyuan Hu, and Yu Yang. Stochastic continuous submodular maximization: Boosting via non-oblivious function. In Proceedings of the 39th International Conference on Machine Learning , 2022.

Qixin Zhang, Zengde Deng, Zaiyi Chen, Kuangqi Zhou, Haoyuan Hu, and Yu Yang. Online learning for non-monotone DR-submodular maximization: From full information to bandit feedback. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , April 2023.

Qixin Zhang, Zongqi Wan, Zengde Deng, Zaiyi Chen, Xiaoming Sun, Jialin Zhang, and Yu Yang. Boosting gradient ascent for continuous DR-submodular maximization. arXiv preprint arXiv:2401.08330 , 2024.

Peng Zhao, Guanghui Wang, Lijun Zhang, and Zhi-Hua Zhou. Bandit convex optimization in non-stationary environments. Journal of Machine Learning Research , 22(125):1-45, 2021.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems , volume 33, pages 12510-12520, 2020.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , pages 928936, 2003.

A Related works

DR-submodular maximization Two of the main methods for continuous DR-submodular maximization are Frank-Wolfe type methods and Boosting based methods . This division is based on how the approximation coefficient appears in the proof.

In Frank-Wolfe type algorithms, the approximation coefficient appears by specific choices of the Frank-Wolfe update rules. (See Lemma 8 in [35]) The specific choices of the update rules for different settings have been proposed in [3, 2, 32, 37, 10]. The momentum technique of [31] has been used to convert algorithms designed for deterministic feedback to stochastic feedback setting. [19] proposed a Frank-Wolfe variant with access to a stochastic gradient oracle with known distribution . Frank-Wolfe type algorithms been adapted to the online setting using Meta-Frank-Wolfe [8, 9] or using Blackwell approachablity [33]. Later [45] used a Meta-Frank-Wolfe with random permutation technique to obtain full-information results that only require a single query per function and also bandit results. This was extended to another settings by [47] and generalized to many different settings with improved regret bounds by [35].

Another approach, referred to as boosting, is to construct an alternative function such that maximization of this function results in approximate maximization of the original function. Given this definition, we may consider the result of [20, 8, 13] as the first boosting based results. However, in these cases (i.e., the case of monotone DR-submodular functions over general convex sets), the alternative function is identical to the original function. The term boosting in this context was first used in [46] for monotone functions over convex sets containing the origin, based on ideas presented in [14, 30]. This idea was used later in [39, 27] in bandit and projection-free full-information settings. Finally, in [48] a boosting based method was introduced for non-monotone functions over general convex sets.

Up-concave maximization Not all continuous DR-submodular functions are concave and not all concave functions are continuous DR-submodular. [30] considers functions that are the sum of a concave and a continuous DR-submodular function. It is well-known that continuous DRsubmodular functions are concave along positive directions [5, 3]. Based on this idea, [41] defined an up-concave function as a function that is concave along positive directions. Up-concave maximization has been considered in the offline setting before, e.g. [25], but not in online setting. In this work, we focus on up-concave maximization which is a generalization of DR-submodular maximization.

Projection-free optimization In the past decade, numerous projection-free online convex optimization algorithms have emerged to tackle the computational limitations of their projection-based counterparts [21, 7, 42, 9, 22, 16, 29, 17]. In the context of DR-submodular maximization, the Frank-Wolfe type methods discussed above are projection-free.

Non-stationary regret Dynamic regret was first analyzed in [51] for first order deterministic feedback. Later [43] obtained the lower bound and optimal algorithm in this setting. This was later expanded to bandit setting in [49]. Adaptive regret was first analyzed in [23] and the first optimal algorithm for projection-free adaptive regret was proposed in [17]. We refer to [23, 1, 11, 44, 43, 50, 49, 28, 40, 17] and references therein for more details.

Optimization by quadratization The framework discussed here for analyzing online algorithms is based on the convex optimization framework introduced in [34]. We extend the framework to allows us to work with α -regret.Moreover, [34] also demonstrates that algorithms that are designed for quadratic/linear optimization with fully adaptive adversary obtain a similar regret in the convex setting. In this paper we introduce the notion of quadratizable functions generalizes this idea beyond convex functions to all quadratizable functions. (see Theorem 1) This allows us to integrate the boosting method with our framework to obtain various meta-algorithms for continuous DR-submodular maximization.

B Problem Setup in Detail

In this section, we further expand on the description in Section 2.

A function class is a set of real-valued functions. Given a set D , a function class over D is a subset of all real-valued functions on D . A set K ⊆ R d is called a convex set if for all x , y ∈ K and α ∈ [0 , 1] , we have α x + (1 -α ) y ∈ K . For any u ∈ K T , we define the path length P T ( u ) := ∑ T -1 i =1 ∥ u i -u i +1 ∥ .

A real-valued differentiable function f is called concave if f ( y ) -f ( x ) ≤ f ′ ( x )( y -x ) , for all x, y ∈ Dom( f ) . More generally, given µ ≥ 0 and 0 < γ ≤ 1 , we say a real-valued differentiable function is µ -strongly γ -weakly concave if

for all x, y ∈ Dom( f ) .

We say a differentiable function f : K → R is µ -strongly γ -weakly up-concave if it is µ -strongly γ -weakly concave along positive directions. Specifically if, for all x ≤ y in K , we have

This notion could be generalized in the following manner. We say ˜ ∇ f : K → R d is a µ -strongly γ -weakly up-super-gradient of f if for all x ≤ y in K , we have

Then we say f is µ -strongly γ -weakly up-concave if it is continuous and it has a µ -strongly γ -weakly up-super-gradient. When it is clear from the context, we simply refer to ˜ ∇ f as an up-super-gradient for f . When γ = 1 and the above inequality holds for all x , y ∈ K , we say f is µ -strongly concave.

A differentiable function f : K → R is called continuous DR-submodular if for all x ≤ y , we have ∇ f ( x ) ≥ ∇ f ( y ) . More generally, we say f is γ -weakly continuous DR-submodular if for all x ≤ y , we have ∇ f ( x ) ≥ γ ∇ f ( y ) . It follows that any γ -weakly continuous DR-submodular functions is γ -weakly up-concave.

Given a continuous monotone function f : K → R , its curvature is defined as the smallest number c ∈ [0 , 1] such that

for all x , y ∈ K and z ≥ 0 such that x + z , y + z ∈ K . 4 We define the curvature of a function class F as the supremum of the curvature of functions in F .

Online optimization problems can be formalized as a repeated game between an agent and an adversary. The game lasts for T rounds on a convex domain K where T and K are known to both players. In t -th round, the agent chooses an action x t from an action set K ⊆ R d , then the adversary chooses a loss function f t ∈ F and a query oracle for the function f t . Then, for 1 ≤ i ≤ k t , the agent chooses a points y t,i and receives the output of the query oracle. Here k t denotes the total number of queries made by the agent at time-step t , which may or may not be known in advance.

To be more precise, an agent consists of a tuple (Ω A , A action , A query ) , where Ω A is a probability space that captures all the randomness of A . We assume that, before the first action, the agent samples ω ∈ Ω . The next element in the tuple, A action = ( A action 1 , · · · , A action T ) is a sequence of functions such that A t that maps the history Ω A ×K t -1 × ∏ t -1 s =1 ( K × O ) k s to x t ∈ K where we use O to denote range of the query oracle. The last element in the tuple, A query , is the query policy. For each 1 ≤ t ≤ T and 1 ≤ i ≤ k t , A query t,i : Ω A × K t × ∏ t -1 s =1 ( K × O ) k s × ( K × O ) i -1 is a function that, given previous actions and observations, either selects a point y i t ∈ K , i.e., query, or signals that the query policy at this time-step is terminated. We may drop ω as one of the inputs of the above functions when there is no ambiguity. We say the agent query function is trivial if

k t = 1 and y t, 1 = x t for all 1 ≤ t ≤ T . In this case, we simplify the notation and use the notation A = A action = ( A 1 , · · · , A T ) to denote the agent action functions and assume that the domain of A t is Ω A × ( K×O ) t -1 .

Aquery oracle is a function that provides the observation to the agent. Formally, a query oracle for a function f is a map Q defined on K such that for each x ∈ K , the Q ( x ) is a random variable taking value in the observation space O . The query oracle is called a stochastic value oracle or stochastic zeroth order oracle if O = R and f ( x ) = E [ Q ( x )] . Similarly, it is called a stochastic up-supergradient oracle or stochastic first order oracle if O = R d and E [ Q ( x )] is a up-super-gradient of f at x . In all cases, if the random variable takes a single value with probability one, we refer to it as a deterministic oracle. Note that, given a function, there is at most a single deterministic gradient oracle, but there may be many deterministic up-super-gradient oracles. We will use ∇ to denote the deterministic gradient oracle. We say an oracle is bounded by B if its output is always within the Euclidean ball of radius B centered at the origin. We say the agent takes semi-bandit feedback if the oracle is first-order and the agent query function is trivial. Similarly, it takes bandit feedback if the oracle is zeroth-order and the agent query function is trivial. 5 If the agent query function is non-trivial, then we say the agent requires full-information feedback .

An adversary Adv is a set such that each element B ∈ Adv , referred to as a realized adversary , is a sequence ( B 1 , · · · , B T ) of functions where each B t maps a tuple ( x 1 , · · · , x t ) ∈ K t to a tuple ( f t , Q t ) where f t ∈ F and Q t is a query oracle for f t . We say an adversary Adv is oblivious if for any realization B = ( B 1 , · · · , B T ) , all functions B t are constant, i.e., they are independent of ( x 1 , · · · , x t ) . In this case, a realized adversary may be simply represented by a sequence of functions ( f 1 , · · · , f T ) ∈ F T and a sequence of query oracles ( Q 1 , · · · , Q T ) for these functions. We say an adversary is a weakly adaptive adversary if each function B t described above does not depend on x t and therefore may be represented as a map defined on K t -1 . In this work we also consider adversaries that are fully adaptive , i.e., adversaries with no restriction. Clearly any oblivious adversary is a weakly adaptive adversary and any weakly adaptive adversary is a fully adaptive adversary. Given a function class F and i ∈ { 0 , 1 } , we use Adv f i ( F ) to denote the set of all possible realized adversaries with deterministic i -th order oracles. If the oracle is instead stochastic and bounded by B , we use Adv f i ( F , B ) to denote such an adversary. Finally, we use Adv o i ( F ) and Adv o i ( F , B ) to denote all oblivious realized adversaries with i -th order deterministic and stochastic oracles, respectively.

In order to handle different notions of regret with the same approach, for an agent A , adversary Adv , compact set U ⊆ K T , approximation coefficient 0 < α ≤ 1 and 1 ≤ a ≤ b ≤ T , we define regret as

where the expectation in the definition of the regret is over the randomness of the algorithm and the query oracle. We use the notation R A α, B ( U )[ a, b ] := R A α, Adv ( U )[ a, b ] when Adv = {B} is a singleton. We may drop α when it is equal to 1. When α < 1 , we often assume that the functions are non-negative.

Static adversarial regret or simply adversarial regret corresponds to a = 1 , b = T and U = K T ⋆ := { ( x , · · · , x ) | x ∈ K} . When a = 1 , b = T and U contains only a single element then it is referred to as the dynamic regret [51, 43]. Adaptive regret , is defined as max 1 ≤ a ≤ b ≤ T R A α, Adv ( K T ⋆ )[ a, b ] [23]. We drop a , b and U when the statement is independent of their value or their value is clear from the context.

C Proof of Theorem 1

The proof is similar to the proof of Theorems 2 and 5 in [34].

Proof.

Deterministic oracle:

We first consider the case where G is a deterministic query oracle for g . Let o t = g ( f t , x t ) denote the output of G at time-step t . For any realization B = ( B 1 , · · · , B T ) ∈ Adv f 1 ( F ) , we define B ′ t ( x 1 , · · · , x t ) to be the tuple ( q t , ∇ ) where

and B ′ = ( B ′ 1 , · · · , B ′ T ) . Note that each B ′ t is a deterministic function of x 1 , · · · , x t and therefore B ′ ∈ Adv f 1 ( F µ, g ) . Since the algorithm uses semi-bandit feedback, the sequence of random vectors ( x 1 , · · · , x T ) chosen by A is identical between the game with B and B ′ . Therefore, according to definition of quadratizable functions, for any y ∈ K , we have

Therefore, we have

Hence

Stochastic oracle:

Next we consider the case where G is a stochastic query oracle for g .

Let Ω Q = Ω Q 1 ×··· × Ω Q T capture all sources of randomness in the query oracles of Adv o 1 ( F , B 1 ) , i.e., for any choice of θ ∈ Ω Q , the query oracle is deterministic. Hence for any θ ∈ Ω Q and realized adversary B ∈ Adv o 1 ( F , B 1 ) , we may consider B θ as an object similar to an adversary with a deterministic oracle. However, note that B θ does not satisfy the unbiasedness condition of the oracle, i.e., the returned value of the oracle is not necessarily the gradient of the function at that point. Recall that B t maps a tuple ( x 1 , · · · , x t ) to a tuple of f t and a stochastic query oracle for f t . We will use E Ω Q to denote the expectation with respect to the randomness of query oracle and E Ω Q t [ · ] := E Ω Q [ ·| f t , x t ] to denote the expectation conditioned the action of the agent and the adversary. Similarly, let E Ω A denote the expectation with respect to the randomness of the agent. Let o t be the random variable denoting the output of G at time-step t and let

Similar to the deterministic case, for any realization B = ( f 1 , · · · , f T ) ∈ Adv o ( F ) and any θ ∈ Ω Q , we define B ′ θ,t ( x 1 , · · · , x t ) to be the pair ( q t , ∇ ) where

We also define B ′ θ := ( B ′ θ, 1 , · · · , B ′ θ,T ) . Note that a specific choice of θ is necessary to make sure that the function returned by B ′ θ,t is a deterministic function of x 1 , · · · , x t and not a random variable and therefore B ′ θ belongs to Adv f 1 ( Q µ [ B 1 ]) .

Since the algorithm uses (semi-)bandit feedback, given a specific value of θ , the sequence of random vectors ( x 1 , · · · , x T ) chosen by A is identical between the game with B θ and B ′ θ . Therefore, for any u t ∈ K , we have

where the first inequality follows from the fact that f t is up-quadratizable and ¯ o t = g ( f t , x t ) . Therefore we have

Since B is oblivious, the sequence ( f 1 , · · · , f T ) is not affected by the randomness of query oracles or the agent. Therefore we have

where the second inequality follows from Jensen's inequality. Hence we have

D Proof of Lemma 1

Proof. We have x ∨ y + x ∧ y = x + y . Therefore, following the definition of curvature, we have

Since f is non-negative, this implies that

On the other hand, according to the definition, we have

Therefore, using Inequality 5 and the fact that f ( x ∧ y ) ≥ 0 , we see that

where we used x ∨ y + x ∧ y = x + y and

in the last equality. The claim now follows from multiply both sides by γ 2 1+ cγ 2 .

E Proof of Lemma 2

Proof. Clearly we have F ( 0 ) = 0 . For any x = 0 , the integrand in the definition of F is a continuous non-negative function of z that is bounded by

̸

Therefore F is well-defined on [0 , 1] d . Moreover, we have 6

It follows that F is differentiable everywhere and E [ ∇ f ( Z ∗ x ) ] = ∇ F ( x ) . To prove the last claim, first we note that

On the other hand, using monotonicity and up-concavity of f , we have

where we used ∫ 1 0 e γ ( z -1) γdz = 1 -e -γ in the last equality. Therefore

F Proof of Lemma 3

We start by extending a useful lemma from the literature. Versions of the following lemma for DRsubmodular functions appeared in [2, 6, 32] and it was later extended to γ -weakly DR-submodular functions in [36]. Here we further extend it to γ -weakly up-concave functions. The proof is similar, but we include it for completeness.

Lemma 4. For any two vectors x , y ∈ [0 , 1] d and any continuously differentiable non-negative γ -weakly up-concave function f we have

Proof of Lemma 4. If ∥ x ∥ ∞ = 0 , then x is the zero vector, and the lemma is trivially true. On the other hand, if x ∨ y = y , the lemma follows from non-negativity of f . Thus, we may assume that z := x ∨ y -y > 0 and ∥ x ∥ ∞ > 0 . We have

where (6) holds by changing the integration variable to t ′ = t/ ∥ x ∥ ∞ , and the inequality follows from γ -weakly up-concavity of f , in particular because f is γ -weakly concave along the line segment [ y , y + t ′ . z ] ⊆ [0 , 1] d . To see that the last inclusion holds, note that, for every 1 ≤ i ≤ d , if x i ≤ y i , then y i + t ′ · z i = y i ≤ 1 , and if x i ≥ y i , then

Next we see that

where the inequality follows from non-negativity of f . The lemma now follows by plugging this inequality into Inequality (6) and rearranging the terms.

̸

Proof of Lemma 3. Clearly we have F ( x ) = 0 . For any x = x , the integrand in the definition of F is a continuous function of z that is bounded by

Therefore F is well-defined on [0 , 1] d . Moreover, we have 7

It follows that F is differentiable everywhere and E [ ∇ f ( Z 2 ∗ ( x -x ) + x ) ] = ∇ F ( x ) .

To prove the last claim, let x z := z 2 ∗ ( x -x ) + x and ω ( z ) = 1 8(1 -z 2 ) 3 . We have

Using Lemma 4, we have

Therefore

Next we bound ⟨∇ F ( x ) , x ⟩ . We have

For the first term, we have

which implies that

Combining Equations 7 and 8, we have

G Proof of Theorem 5

The algorithms are special cases of Algorithms 2 and 3 in [34] where the shrinking parameter and the smoothing parameter are equal. We include a description of the algorithms for completion.

The proof of Theorems 5 and 6 are similar to the proof of Theorems 6, 7 and 8 in [34]. The only difference being that we prove the result for α -regret instead of regret. We include a proof for completion.

Proof of Theorem 5.

Regret bound for STB :

Note that any realized adversary B ∈ Adv o 0 ( F , B 0 ) may be represented as a sequence of functions ( f 1 , · · · , f T ) and a corresponding sequence of query oracles ( Q 1 , · · · , Q T ) . For such realized adversary B , we define ˆ B to be the realized adversary corresponding to ( ˆ f 1 , · · · , ˆ f T ) with the stochastic gradient oracles

where v is a random vector, taking its values uniformly from S 1 ∩ L 0 = S 1 ∩ (aff( K ) -z ) , for any z ∈ K and k = dim( L 0 ) . Since Q t is a stochastic value oracle for f t , according to Remark 4 in [37], ˆ Q t ( x ) is an unbiased estimator of ∇ ˆ f t ( x ) . 8 Hence we have ˆ B ∈ Adv o 1 ( ˆ F , k δ B 0 ) . Using Equation 9 and the definition of the Algorithm 6, we see that the responses of the queries are the same between the game ( A , ˆ B ) and ( A ′ , B ) . It follows that the sequence of actions ( x 1 , · · · , x T ) in ( A , ˆ B ) corresponds to the sequence of actions ( x 1 + δ v 1 , · · · , x T + δ v T ) in ( A ′ , B ) .

Let u ∈ argmax u ∈U ∑ b t = a f t ( u t ) and ˆ u ∈ argmax u ∈ ˆ U ∑ b t = a ˆ f t ( u t ) . We have

According to Lemma 3 in [37], we have | ˆ f t ( x t ) -f t ( x t ) | ≤ δM 1 . By using Lipschitz property for the pair ( x t , x t + δ v t ) , we see that

On the other hand, we have

Therefore, using Equation 10, we see that

Therefore, we have

Regret bound for FOTZO :

The proof of the bounds for this case is similar to the previous case. As before, we see that the responses of the queries are the same between the game ( A , ˆ B ) and ( A ′ , B ) . It follows from the description of Algorithm 5 that the sequence of actions ( x 1 , · · · , x T ) in ( A , ˆ B ) corresponds to the same sequence of actions in ( A ′ , B ) .

Let u ∈ argmax u ∈U ∑ b t = a f t ( u t ) and ˆ u ∈ argmax u ∈ ˆ U ∑ b t = a ˆ f t ( u t ) . We have

To obtain the same bound as before, instead of Inequality 11, we have

The rest of the proof follows verbatim.

Proof of Theorem 6. Note that any realized adversary B ∈ Adv o 0 ( F ) may be represented as a sequence of functions ( f 1 , · · · , f T ) . For such realized adversary B , we define ˆ B to be the realized adversary corresponding to ( ˆ f 1 , · · · , ˆ f T ) with the stochastic gradient oracles

where v is a random vector, taking its values uniformly from S 1 ∩L 0 = S 1 ∩ (aff( K ) -z ) , for any z ∈ K and k = dim( L 0 ) . Since Q t is a stochastic value oracle for f t , according to Lemma 5 in [37], ˆ Q t ( x ) is an unbiased estimator of ∇ ˆ f t ( x ) . Moreover, we have

Hence we have ˆ B ∈ Adv o 1 ( ˆ F , kM 1 ) . Using Equation 13 and the definition of the Algorithm 5, we see that the responses of the queries are the same between the game ( A , ˆ B ) and ( A ′ , B ) . It follows that the sequence of actions ( x 1 , · · · , x T ) in ( A , ˆ B ) corresponds to the same sequence of actions in ( A ′ , B ) .

Let u ∈ argmax u ∈U ∑ b t = a f t ( u t ) and ˆ u ∈ argmax u ∈ ˆ U ∑ b t = a ˆ f t ( u t ) . We have

According to Lemma 3 in [37], we have | ˆ f t ( x t ) -f t ( x t ) | ≤ δM 1 . On the other hand, we have

Therefore, using Equation 14, we see that

Therefore, we have

Corollary 4. Under the assumptions of Theorem 5, if we have R A α, Adv o 1 ( F ,B 1 ) = O ( B 1 T η ) and δ = T ( η -1) / 2 , then we have

Corollary 5. Under the assumptions of Theorem 6, if we have δ = T -1 , then R A ′ α, Adv o 0 ( F ) has the same order of regret as that of R A α, Adv o 1 ( F ,B 1 ) with B 1 replaced with kM 1 .

H Proof of Theorem 7

Proof. Given a realized adversary B ∈ Adv o i ( F , B ) { T } , we may define ˆ B = Adv o i ( F , B ) { T/L } to be the realized adversary constructed by averaging each T/L block of length L . Specifically, if the functions chosen by B are f 1 , · · · , f T , the functions chosen by ˆ B are ˆ f q := 1 L ∑ qL t =( q -1) L +1 f t for 1 ≤ q ≤ T/L . Note that, for any x ∈ K and ( q -1) L < t ≤ qL , we have E [ f t ( x )] = ˆ f q ( x ) and if each f t is differentiable at x , then E [ ∇ f t ( x )] = ∇ ˆ f q ( x ) . If the query oracles selected by B are Q 1 , · · · , Q T , then for any 1 ≤ q ≤ T/L we define the query oracle ˆ Q q as the algorithm that first selects an integer ( q -1) L +1 ≤ t ≤ qL with uniform probability and then returns the output of Q t . It follows that ˆ Q q is a query oracle for ˆ f q . It is clear from the description of Algorithm 4

that, when the adversary is B , the output returned to the base algorithm corresponds to ˆ B . We have 1 ≤ ( a ′ -1) L +1 ≤ a ≤ b ≤ b ′ L ≤ T . Hence

Therefore

Remark 1 . Note that in the above proof, we did not need to assume that the query oracles are bounded. Specifically, what we require is that the set of query oracles to be closed under convex combinations. This holds when all query oracles are bounded by B , but it also holds under many other assumptions, e.g., if we assume all query oracles variances are bounded by some σ 2 > 0 .

Corollary 6. Under the assumptions of Theorem 7, if we have R A ′ α, Adv o i ( F ,B ) ( K T ⋆ )[ a, b ] = O ( BT η ) , K = O ( T θ ) and L = T 1+ θ -η 2 -η , then we have

As a special case, when K = O (1) , then we have

Proof. We have

I Proof of Theorem 8

Proof. Since A requires T θ queries per time-step, it requires a total of T 1+ θ queries. The expected error is bounded by the regret divided by time. Hence we have ϵ = O ( T η -1 ) after T 1+ θ queries. Therefore, the total number of queries to keep the error bounded by ϵ is O ( ϵ -1+ θ 1 -η ) .

J Projection-free adaptive regret

The SO -OGD algorithm in [17] is a deterministic algorithm with semi-bandit feedback, designed for online convex optimization with a deterministic gradient oracle. Here we assume that the separation oracle is deterministic.

Here we use the notation c , r and ˆ K δ described in Section 5.

Note that here we use a maximization version of the algorithm, which we denote by SO -OGA . Here P K denotes projection into the convex set K . The original version, which is designed for minimization, uses the update rule x ′ t +1 = x t -η o t in Algorithm 8 instead.

Lemma 5. Algorithm 9 stops after at most (dist( y 0 , ˆ K δ ) 2 -dist( y , ˆ K δ ) 2 ) /δ 2 + 1 iterations and returns y ∈ K such that ∀ z ∈ ˆ K δ , we have ∥ y -z ∥ ≤ ∥ y 0 -z ∥ .

Proof. We first note that this algorithm is invariant under translations. Hence it is sufficient to prove the result when c = 0 .

Let SO ′ K denote the following separation oracle. If y ∈ K or y / ∈ aff( K ) , then SO ′ K returns the same output as SO K . Otherwise, it returns P aff( K ) ( g ) where g ∈ R d is the output of SO K . To prove that this is indeed a separation oracle, we only need to consider the case where y ∈ aff( K ) \ K . We know that g is a vector such that

Since P aff( K ) is an orthogonal projection, we have

for all x ∈ K , which implies that SO ′ K is a separation oracle.

Nowweseethat Algorithm 9 is an instance of Algorithm 6 in [17] applied to the initial point y 1 using the separation oracle SO ′ K . Hence we may use Lemma 13 in [17] directly to see that Algorithm 9 stops after at most (dist( y 1 , ˆ K δ ) 2 -dist( y , ˆ K δ ) 2 ) /δ 2 + 1 iterations and returns y ∈ K such that ∀ z ∈ ˆ K δ , we have ∥ y -z ∥ ≤ ∥ y 1 -z ∥ Since y 1 is the projection of y over aff( K ) , we see that Algorithm 9 stops after at most

steps and

In the following, we use the notation

to denote the adaptive regret.

Theorem 9. Let L be a class of linear functions over K such that ∥ l ∥ ≤ M 1 for all l ∈ L and let D = diam( K ) . Fix v > 0 such that δ = vT -1 / 2 ∈ (0 , 1) and set η = vr 2 M 1 T -1 / 2 . Then we have

Proof. Since the algorithm is deterministic, according to Theorem 1 in [34], it is sufficient to prove this regret bound against the oblivious adversary Adv o 1 ( L ) .

Note that this algorithm is invariant under translations. Hence it is sufficient to prove the result when c = 0 . If aff( K ) = R d , then we have B d r ( 0 ) ⊆ K ⊆ B d R ( 0 ) and we may use Theorem 14 from [17] to obtain the desired result for the oblivious adversary Adv o 1 ( L ) . On the other hand, the assumption B d r ( 0 ) ⊆ K is only used in the proof of Lemma 13 in [17]. Here we use Lemma 5 instead which does not require this assumption.

The following corollary is an immediate consequence of the above theorem and Theorems 2, 3, 4, 8 and Corollaries 4, 5 and 6.

Corollary 7. Let SO -OGA denote the algorithm described above. Then the following are true.

Under the assumptions of Theorem 2, we have:

If we also assume F is bounded by M 0 and B 0 ≥ M 0 , then

b) Under the assumptions of Theorem 3, we have:

where A = OMBQ ( SO -OGA , BQM0 , Id) . Note that A is a first order full-information algorithm that requires a single query per time-step. If we also assume F is bounded by M 0 and B 0 ≥ M 0 , then

where

A semi-bandit = SFTT ( A ) , A full-info-0 = FOTZO ( A ) , A bandit = SFTT ( A full-info-0 ) .

Under the assumptions of Theorem 4, we have:

where A = OMBQ ( SO -OGA , BQN , x ↦→ x t + x 2 ) . Note that A is a first order full-information algorithm that requires a single query per time-step. If we also assume F is bounded by M 0 and B 0 ≥ M 0 , then

where

K Dynamic regret

Improved Ader ( IA ) algorithm [43] is a deterministic algorithm with semi-bandit feedback, designed for online convex optimization with a deterministic gradient oracle.

Algorithm 10: Improved Ader IA

Input : horizon T , constraint set K , step size λ , a set H containing step sizes for experts Activate a set of experts { E η | η ∈ H} by invoking Algorithm 11 for each step size η ∈ H Sort step sizes in ascending order η 1 ≤ · · · ≤ η N , and set w η i 1 = C i ( i +1) where C = 1 + 1 |H|

for t = 1 , 2 , . . . , T

Send the gradient o t to each expert E

end

Algorithm 11: Improved Ader : Expert algorithm

In Algorithm 11, P K denotes projection into the convex set K . Note that here we used the maximization version of this algorithm. The original version, which is designed for minimization, uses the update rule x η t +1 = P K ( x η t -η o t ) in Algorithm 11 instead.

Theorem 10. Let L be a class of linear functions over K such that ∥ l ∥ ≤ M 1 for all l ∈ L and let D = diam( K ) . Set H := { η i = 2 i -1 D M 1 √ 7 2 T | 1 ≤ i ≤ N } where N = ⌈ 1 2 log 2 (1 + 4 T/ 7) ⌉ +1 and λ = √ 2 / ( TM 2 1 D 2 ) . Then for any comparator sequence u ∈ K T , we have

Proof. If we use the oblivious adversary Adv o 1 ( L ) instead, this theorem is simply a restatement of the special case (i.e. when the functions are linear) of Theorem 4 in [43]. 9 Since the algorithm is deterministic, according to Theorem 1 in [34], the regret bound remains unchanged when we replace Adv o 1 ( L ) with Adv f 1 ( L ) .

The following corollary is an immediate consequence of the above theorem and Theorems 2, 3, 4, and Corollaries 4 and 5.

Note that we do not use the meta-algorithm OTB since Improved Ader is designed for non-stationary regret and does not offer any advantages in the offline case. On the other hand, we do not use the meta-algorithm SFTT in this case since Theorem 7 is only for the setting where the comparator is K T ∗ and does not allow us to convert bounds for dynamic regret.

Corollary 8. Let IA denote 'Improved Ader' described above. Then the following are true.

Under the assumptions of Theorem 2, we have:

If we also assume F is bounded by M 0 and B 0 ≥ M 0 , then

Under the assumptions of Theorem 3, we have:

where A = OMBQ ( IA , BQM0 , Id) . Note that A is a first order full-information algorithm that requires a single query per time-step. If we also assume F is bounded by M 0 and B 0 ≥ M 0 , then

where A full-info-0 = FOTZO ( A ) .

Under the assumptions of Theorem 4, we have:

where A = OMBQ ( IA , BQN , x ↦→ x t + x 2 ) . Note that A is a first order full-information algorithm that requires a single query per time-step. If we also assume F is bounded by M 0 and B 0 ≥ M 0 , then

where A full-info-0 = FOTZO ( A ) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the paper's contribution and scope. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The statements of the results have the precise assumptions. Further, some limitations are also discussed in the conclusions section as a future work direction.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have clearly stated the required assumptions and an accompanying complete proof in the appendix for each theory result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our paper is primarily of theoretical nature and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms, in every respect, to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is primarily of theoretical nature and has no immediate societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No high risk data or model have been used.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No existing asset has been used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new asset is introduced in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No experiments with human subjects were conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We conducted no experiments with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online optimization with upper linearizable/quadratizable functions which are a new class of objectives considered in this field. This class extends concavity and DR-submodularity in various settings, including monotone and non-monotone cases over different types of convex sets. A general meta-algorithm is proposed which can convert linear/quadratic maximization algorithms into ones that optimize upper quadratically functions. In this way, once can solve the concave and DR-submodular optimization problems through a unified approach.\n\nSTRENGTHS:\nThis paper proposes a novel framework and novel notion of quadratizable functions and relates the algorithms and regret guarantees for the optimization of linear functions to that of quadratizable functions.\n\nThis paper shows that the class of quadratizable function optimization is general, and includes not only concave, but up-concave optimization in several cases.\n\nThe authors design a new meta-algorithm, namely SFTT, that captures the idea of random permutations (sometimes referred to as blocking) as used in several papers such as [44, 46, 34]. While previous works used this idea in specific settings, the meta-algorithm is applicable in general settings.\n\nWEAKNESSES:\n1. The mentioned upper linearizable/quadratizable functions seem to extend the online optimization framework. It is good as a theoretical paper. Though it will be great if the authors can provide motivations for this kind of model from practial applications\n\n2. I think it will be better if the authors can provide numerical results or emperical intuitions other than the theoretical guaranttees?\n\n3. Another way to strengthen this work is providing a lower bound, or at least a discussion on it\n\n\nI am not familiar with this topic and will look at the comments by other reviewers.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new class of functions are introduced: upper linearizable/quadratizable functions, which extend concavity and DR-submodularity. It is shown how to apply algorithms for linear / quadratic maximization to this class, which offers a unified approach to DR-submodular optimiziation problems. The abstract framework is then applies to yield results for a broad variety of optimization settings, including online up-concave maximization with monotone functions, non-monotone, and a variety of feedback settings. Many of these settings had been studied previously in the literature, and the results obtained here either match or improve in many cases. They also convert the online results to offline ones, and apply the framework to non-stationary up-concave maximization.\n\nSTRENGTHS:\n- The abstraction of the class of linearizable functions unifies and improves results in many settings by making a connection with linear optimization. \n- In many cases, the results are achieved under fewer assumptions. For example, offline, monotone, maximization over a set containing the origin with access to the gradient of the function, the authors obtain the same sample complexity and approximation ratio as [18], but only require the function to be up-concave, differentiable, and Lipschitz, while [18] has more requirements, such as the function's Hessian to be Lipschitz.\n- Obtaining these results from one general framework enhances the understanding of these problems and the relationship between them\n- Figure 1 shows how to obtain various results from the theorem statements by following a directed path on a graph.\n\nWEAKNESSES:\nminor comments:\n- line 169: expect -> except\n- line 251: every function of F\n\nQUESTIONS:\n- Are there any implications for non-monotone maximization over downward closed sets?\n- The results for non-monotnoe functions don't seem to apply to gamma-weakly up-concave functions or improve with curvature. Can these results be extended to such cases?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper introduces a novel notion of upper linearizable / quadratizable functions. Using this notion of functions, the paper proposes a general meta algorithm that extends certain online linear / quadratic maximization algorithms to handle upper linearizable / quadratizable function classes. This general meta algorithm features multiple variants, obtaining results for different feedback settings.

STRENGTHS:
Disclaimer: This paper is out of my expertise area. I do not have a good understanding of this paper. My review should be considered an educated guess.

The technical contribution of this paper is strong. The meta algorithm is general and leads to several theoretical results.

WEAKNESSES:
I find this paper hard to follow. It is largely due to my lack of familiarity with this topic, but I think the authors could do a better job with the presentation. E.g. add examples and figures, when showing theoretical results, also add some explanation and intuition.

QUESTIONS:
I don't have specific questions.

LIMITATIONS:
The limitations are adequately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13