Title: Online Ad Procurement in Non-stationary Autobidding Worlds
Venue: NeurIPS 2023
Accepted: yes
Keywords: autobidding

online advertising

bandit online convex optimization

constrained optimization
Abstract: Today's online advertisers procure digital ad impressions through interacting with autobidding platforms: advertisers convey high level procurement goals via setting levers such as budget, target return-on-investment, max cost per click, etc.. Then ads platforms subsequently procure impressions on advertisers' behalf, and report final procurement conversions (e.g. click) to advertisers. In practice, advertisers may receive minimal information on platforms' procurement details, and procurement outcomes  are subject to non-stationary factors like seasonal patterns, occasional system corruptions, and market trends which make it difficult for advertisers to optimize lever decisions effectively. Motivated by this,  we present an online learning framework that helps advertisers dynamically optimize ad platform lever decisions while subject to general long-term constraints in a realistic bandit feedback environment with non-stationary procurement outcomes. In particular, we introduce a primal-dual algorithm for online decision making with multi-dimension decision variables, bandit feedback and long-term uncertain constraints. We show that our algorithm achieves low regret in many worlds when procurement outcomes are generated through procedures that are stochastic, adversarial, adversarially corrupted, periodic, and ergodic, respectively, without having to know which procedure is the ground truth. Finally, we emphasize that our proposed algorithm and theoretical results extend beyond the applications of online advertising.

Online Ad Procurement in Non-stationary Autobidding Worlds

Abstract

Today's online advertisers procure digital ad impressions through interacting with autobidding platforms: advertisers convey high level procurement goals via setting levers such as budget, target return-on-investment, max cost per click, etc. Then ads platforms subsequently procure impressions on advertisers' behalf, and report final procurement conversions (e.g. click) to advertisers. In practice, advertisers may receive minimal information on platforms' procurement details, and procurement outcomes are subject to non-stationary factors like seasonal patterns, occasional system corruptions, and market trends which make it difficult for advertisers to optimize lever decisions effectively. Motivated by this, we present an online learning framework that helps advertisers dynamically optimize ad platform lever decisions while subject to general long-term constraints in a realistic bandit feedback environment with non-stationary procurement outcomes. In particular, we introduce a primal-dual algorithm for online decision making with multi-dimensional decision variables, bandit feedback and long-term uncertain constraints. We show that our algorithm achieves low regret in many worlds when procurement outcomes are generated through procedures that are stochastic, adversarial, adversarially corrupted, periodic, and ergodic, respectively, without having to know which procedure is the ground truth. Finally, we emphasize that our proposed algorithm and theoretical results extend beyond the applications of online advertising.

1 Introduction

Automated bidding (or autobidding for short) has become the dominant mode for advertisers to procure digital ad inventories and impressions, contributing to more than $120 billion dollar ad spend in 2022 and more than 90% of total online ad transaction volumes [2, 1]. In an autobidding platform, advertisers only need to convey their high-level procurement goals for an ad campaign to the platform, which then takes charge of procuring ads on advertisers' behalf. Such procurement goals are communicated to a platform through platform levers , which are adjustable parameters that advertisers can control to influence the bidding process and campaign performance. To exemplify, Figure 1 displays certain several levers presented on the Google Ads interface, where an advertiser can set per-campaign budgets, target cost-per-actions, campaign duration, campaign schedules, targeting, etc; similar examples are also shown in related literature [21].

As the primary avenue for advertisers to run ad campaigns on autobidding platforms and influence ad conversion outcomes (e.g., clicks), making efficient lever decisions is vital to advertisers to achieve their procurement objectives. However, advertisers face many challenges in practice when optimizing for lever decisions, namely high-dimensional decision making under long-term constraints, non-stationary autobidding environments, and limited procurement feedback.

High-dimensional decision making and long-term constraint satisfaction. Making multiple lever decisions involves evaluating numerous possible combinations of lever configurations, which is computationally intensive and time-consuming in real-time decision making setups. Also, advertisers need to understand the potential interactions and dependencies between levers, as adjusting one lever may have unintended consequences or interactions with other levers, making it challenging to

Set your average daily budget for this campaign

Ad schedule

Saturdays - Sundays

12:00

to

19:00

Budget

US Dollar (USD S)

$100.00

Languages

English ×

Chinese (simplified)×

French ×

German ×

Select your bid strategy?

Pay for ?

Demographics

Gender

Age

Household income

Bidding

Target CPA

Interactions

Female

18 - 24

Top 10%

Target CPA

Male

25 - 34

√

11 - 20%

$1.00

Unknown ②

√

35 - 44

21- 30%

Start date

End date

√

45 - 54

31 - 40%

Start and end dates

Jan 23, 2023

Jan 25, 2023

55 - 64

41 - 50%

65+

Lower 50%

predict the overall impact of adjustments accurately. Furthermore, advertisers may need to satisfy certain long-term constraints over time, e.g., limiting total spend under a budget, or achieving certain return-on-investment targets. Hence in addition to analyzing the interactions between different levers, advertisers also need to concern long-term consequences of making certain lever decisions.

Limited feedback on procurement outcome and constraints. Despite the fact that autobidding via lever decisions greatly simplifies advertisers' ad procurement process as they no longer have to handle the intricacies of procuring individual ad impressions, the procurement procedure becomes a black box for advertisers, as advertisers only have limited visibility into the specific details of how the platform executes ad placement processes. This lack of transparency can make it challenging for advertisers to control the nuances of their campaign execution through lever decisions, and it amplifies the complexity to conduct counterfactual analyses on outcomes for past lever decisions.

Many non-stationary autobidding worlds. Autobidding procurement environments are highly non-stationary, as a wide spectrum of latent factors in online ad marketplaces may greatly vary procurement outcomes of the same lever decisions in different time periods. These latent factors include but are not limited to changing user preferences, seasonality effects, shifts in market trends, occasional malfunctions in autobidding platforms, etc. These dynamics may influence how users interact with different types of ads, which necessitates continuous adjustment of lever decision strategies to adapt to current and future market conditions.

To address these challenges, in this work we answer the following questions: How should an advertiser dynamically set multiple levers to optimize conversion outcomes subject to long-term constraints under limited information? And can she run a unified algorithm that can perform well while being agnostic to many types of non-stationary autobidding procurement environments?

Motivated by these questions, in this work we study an advertiser's online high-dimensional lever decision problem with long-term constraints under limited bandit feedback in many non-stationary worlds. We summarize our main contributions as followed:

Modelling online lever decisions in many worlds using online constrained optimization with bandit function-valued feedback and uncertain constraints (Section 2) . We cast the advertiser lever decision problem as an online constrained optimization with bandit function-valued feedback and uncertain constraints, where functions at which (lever) decisions are evaluated correspond to the conversion and constraint functions in our autobidding setup. Further, we model real-world nonstationarity in autobidding environments as possibly time-varying distributions from which conversion and constraint functions are sampled, and then further applied to lever decisions. Under this model, we discuss five different input procedures from which the sequence of functional distributions are sampled to model stochastic, adversarial, corrupted, periodic, and ergodic environments; see Section 2.2 for more details. To the best of our knowledge, from a modelling perspective this is the first work to model a high-dimensional lever decision problem in practical non-stationary autobidding worlds.

Proposing a constrained bandit optimization algorithm universally applicable to many worlds (Section 3). We develop a unified bandit optimization algorithm with robust performance guarantees across different worlds. Our algorithm incorporates four key designs to handle bandit function-valued feedback and unknown non-stationary environments. 1. we utilize dual descent to update dual variables associated with long-term constraints and decouple decisions over time; 2. we employ a primal-ascent-based bandit online convex optimization (BOCO) approach to make (primal) lever decisions to cope with bandit function valued feedback; 3. we implement an exponential weights

expert algorithm on top of primal-ascent BOCO, where each expert corresponds to a different primal ascent step size. This enables our algorithm to adapt to the optimal primal ascent step size for each world without prior knowledge on which world we are in; 4. our algorithm dynamically checks for realized constraint violations and applies "safe lever decisions" to ensure long-term constraint satisfaction. For further details, we refer the readers to Algorithm 1 in Section 3.

3. Analyzing the performance of proposed algorithms in many worlds (Section 4). We present theoretical analysis (Theorem 4.2) on the regret bound of the proposed algorithm, and we show that our unified algorithm can achieve reasonable regret bounds with all five input procedures. These regret bounds are summarized in the following Table 1.

Table 1: T -period regret bounds for Algorithm 1 under different input models. Here A > 0 is some algorithm dependent factor which we will later specify in Section 4.

The parameters ( δ, ξ, OPT ( P 1: T ) , q, κ ) are formally defined in Section 2.2 and Theorem 4.2. Finally, in Section 5 we remark all results are applicable to other problems.

1.1 Related works.

Autobidding. In an autobidding framework, there is a considerable body of works that study the price of anarchy, which aims to improve worst case individual or total advertiser welfare guarantees w.r.t. the optimal welfare via mechanism design frameworks; see e.g. [23, 7, 20, 38, 22]. We remark that this line of work does not concern developing online learning algorithms. On the other hand, numerous works have concentrated on developing online bidding algorithms for repeated ad auctions [41, 10, 31, 30], as well as designing repeated selling mechanisms for ad impression allocation; see [12, 27] and references therein. However, as discussed in the introduction, autobidding platforms conduct bidding on behalf of advertisers while keeping bidding and selling mechanism details undisclosed. In this study, we treat the bidding procedure and selling mechanism as a black box and directly model bidding and auction outcomes using conversion functions; see Section 2. To the best of our knowledge, the most pertinent work to this paper is [21], which explores a similar ad procurement problem by optimizing levers within a bandit environment. However, this work solely focuses on the stochastic world and optimizing a single lever (i.e., a 1-dimensional decision space). In contrast, our paper develops a unified algorithm capable of making high-dimensional lever decisions in many worlds.

Online convex optimization. In Section 2, we cast the advertiser's online learning problem of interest to a high-dimensional bandit online convex optimization problem with uncertain long-term constraints, and develop an algorithm that yields good performance guarantees under different procedures from which the objective and constraint functions are generated. There has been a rich line of works that study bandit convex optimization with no long-term constraints in stochastic and adversarial worlds [25, 32, 42, 17, 44], as well as works that study (full-information feedback) online convex optimization with long-term constraints [36, 33, 43, 35]. Further, works that study both bandit feedback and long-term constraints either consider single-dimensional (such as multi-arm bandits with constraints) [40, 6, 4, 24], or consider static regret (i.e., benchmarking performance to that of a single optimal action) [13, 16, 14, 37]. This paper distinguishes itself from these two streams of works by considering high-dimensional decisions as well as dynamic regret, i.e., comparing to the best sequence of actions instead of a single action; see Eq. (1). Finally, all aforementioned works only study algorithms in stochastic or adversarial setups, whereas in this work we go beyond these two worlds and address more complex learning environment such as periodic, corrupted, ergodic, and finite switching. To the best of our knowledge, the only related work that develops a universal algorithm under 'many world' setups is [10]. However, [10] considers a full-information scenario where the online decisions in period can be made after observing realized objective and constraint functions during that period. In this work, we present an efficient algorithm to handle bandit feedback.

Notation. For any vector z ∈ R d , let ∥ z ∥ be its Euclidean norm. Denote B = { z ∈ R d : ∥ z ∥ ≤ 1 } as the d -dimensional unit ball centered at 0 , and let S = { z ∈ R d : ∥ z ∥ = 1 } be the unit sphere. For any set S , let U ( S ) be the uniform distribution over S . Let e denote the vector whose components

are all 1's, and e j be the unit vector whose j 'th position is 1. We use O notation to represent the asymptotic order of a term when the period T →∞ and ignore the log T terms.

2 Autobidding with bandit feedback in many worlds (input models)

2.1 Autobidding as a bandit online optimization with long-term uncertain constraints

Consider an advertiser repeatedly interacting with an ad platform over T ∈ N periods, where each period can be interpreted as a single ad campaign that is run on the platform. During each period t ∈ [ T ] , the advertiser makes d ≥ 1 lever decisions denoted as x t ∈ X ⊆ R d , e.g., setting the per-campaign budget, campaign duration, per-campaign target return-on-investment, and max spend per conversion; see Figure 1 for example lever decisions in practice. Here, X is some compact and convex decision set whose diameter is D = sup { x , x ′ }⊆X ∥ x -x ′ ∥ . For simplicity, assume 0 ∈ X so ∥ x ∥ ≤ D for any x ∈ X . After the lever decisions x t are made the campaign is fully executed via autobidding, the advertiser observes bandit feedback for her campaign outcomes: she only observes her realized conversion f t ( x t ) ∈ R ≥ 0 (e.g., number of clicks on her ads), as well as some constraint balance g t ( x t ) ∈ R K . The conversion and constraint functions ( f t , g t ) are sampled from some (possibly infinite) support S according to distribution P t ∈ ∆( S ) (we will discuss how P t 's are generated by nature in Section 2.2). Using the notation P 1: T = ( P t ) t ∈ [ T ] the advertiser's hindsight optimization problem is

Here, we use a constraint function g k,t : X → R to characterize general performance metrics of the advertiser for her ad campaigns. In the following, we present several examples for constraint functions that are widely used in practice or studied in literature. For illustrative purposes, we assume x t, 1 = e ⊤ 1 x t (i.e., the first lever) represents the per-campaign budget for the t 'th campaign.

Long-term budget constraint. The advertiser has a total budget BT > 0 . Then by letting constraint function g k,t ( x ) = B -x t, 1 , we have ∑ t ∈ [ T ] g k,t ( x t ) ≥ 0 = ⇒ ∑ t ∈ [ T ] x t, 1 ≤ BT , which means total campaign spent over T periods (assuming each campaign fully depletes the campaign budget) must be less than the advertiser's total budget BT .

Long-term return-on-investment constraint . The advertiser intends to safeguard a long-term returnon-investment γ > 0 , i.e., she attains a long-term average of at least γ conversions per dollar spent. Then by considering g k,t ( x ) = f t ( x ) -γx t, 1 , we have ∑ t ∈ [ T ] g k,t ( x t ) ≥ 0 = ⇒ ∑ t ∈ [ T ] f t ( x t ) ≥ γ ∑ t ∈ [ T ] x t, 1 , which means total conversion over T periods is at least γ times total spend.

Finally, we make the following mild assumptions on conversion and constraint functions. 1

Assumption 2.1 (Mild assumptions on conversion and constraint functions) . For any ( f, g ) ∈ S , f , g 1 . . . g K are all bounded concave functions, i.e., we assume sup x ∈X ∥ g ( x ) ∥ ∞ ≤ ¯ G and sup x ∈X | f ( x ) | ≤ ¯ F for some ¯ G, ¯ F < ∞ . Moreover, f and g are L -Lipschitz, i.e., for any { x , x ′ } ⊆ X , we have | f ( x ) -f ( x ′ ) | ≤ L ∥ x -x ′ ∥ and ∥ g ( x ) -g ( x ′ ) ∥ ≤ L ∥ x -x ′ ∥ . Furthermore, there exists ( f, g ) ∈ S such that min x ∈X min k ∈ [ K ] g k ( x ) < 0 .

2.2 Five input models characterizing many autobidding worlds

In this subsection, we describe structural properties of the input distribution sequence P 1: T , and shed light on how we utilize various properties to model a wide spectrum of autobidding environments (called worlds) such as time-varying user preferences, seasonality, shifts in market trends, etc., that may potentially lead to different procurement outcomes for the same ad campaign lever decisions.

Stochastic : There exists some probability distribution P ∈ ∆( S ) such that P 1 = . . . = P T = P . This stochastic world represents a stationary autobidding environment where the underlying latent factors influencing user behaviors (and correspondingly conversion results) remain constant over time; see [31, 9, 21].

δ -corrupted. There exists P ∈ ∆( S ) as well as δ ∈ N periods T = { τ 1 . . . τ δ } ⊂ [ T ] such that P t = P for all t / ∈ T . This δ -corrupted input sequence represents occasional anomalies in the autobidding environment that may be caused by systematic malfunctions in the autobidding platform, or deliberate attempts by malicious competitors to exploit the system for their own benefit; e.g., some competitors may engage in click fraud to inflate the number of clicks on our ads to exhaust our budget or generate false data to manipulate autobidding algorithms (for example, see [29]).

Adversarial. P 1: T is adversarially chosen by nature before the process starts, and the distributions over time can possibly be non-identical and/or dependent. This adversarial world can be viewed a hypothetical extreme case for the δ -corrupted world where each period the procurement outcomes can potentially be corrupted by an adversary. Adversarial input sequences have been widely studied in the literature to assess algorithmic performances in worst-case scenarios; see [3, 30].

Periodic. There exists period length q ∈ N such that T = cq for some integer c ≥ 2 with P 1: T satisfying P 1: q = P q +1:2 q = · · · = P ( c -1) q +1: T . This periodic world captures regular cyclic patterns or fluctuations in the user behavior over specific time intervals; e.g., seasonality, day-of-week patterns, and time of a day.

Ergodic. P 1: T is an ergodic process (e.g., an irreducible and aperiodic Markov chain or stationary autoregressive processes), where there exists some κ > 0 and a stationary distribution such that the distance between κ -step transition probabilities and this stationary distribution decreases exponentially fast in κ . Mathematically, given the input distribution sequence P 1: T , denote P ( t + κ ) | [ t -1] as the conditional distribution of ( f t + κ , g t + κ ) conditioned on the realizations { ( f τ , g τ ) } τ ∈ [ t ] . Then, for the ergodic input model there exists a stationary distribution ˜ P ∈ ∆( S ) and absolute constant R > 0 such that

We remark that exponentially decaying ergodic models have been studied widely in the context of online optimization; see e.g. [10, Section 5.2] for further discussions. To make the problem of interest tractable, we assume κ > log( T ) . Intuitively, an ergodic input sequence signifies that the procurement outcomes in close time proximity are correlated, which is commonly observed in real-world autobidding systems, as they often involve iterative processes that enable procurement algorithms (operated on behalf of advertisers) to converge to a stable state; see details in [9, 28].

2.3 Minimizing regret subject to long-term constraints with a universal algorithm

In this work, we take the perspective of an advertiser making repeated lever decisions as described in Section 2.1. We focus on designing a single online algorithm that determines a lever decision x t ∈ X in each period t with the goal to minimize regret R T (defined as follows) under any input sequence P 1: T while satisfying long-term constraints

Here, the expectation is taken w.r.t. randomness from the input sequence as well as any randomness in our algorithm. We highlight that our desired policy should be agnostic to the input model as well as the input sequence P 1: T . We remark that our work distinguishes itself from related work that study solely stochastic and/or adversarial input models, as we aim to design a universal algorithm that achieves salient performance over any input model described in Section 2.2.

3 A universal constrained BOCO framework for many worlds

In this section, we focus on designing an algorithm that achieves low regret (per Eq. (3)) while maintaining long-term constraint satisfaction in any world described in Section 2.2, without having to know which world we are in. We first highlight three main challenges for our problem of interest.

Dynamic benchmark lever sequence and unknown input sequence P 1: T . Recall the hindsight optimal problem in Eq. (1), with which we are comparing our algorithm's performance to the optimal sequence of lever decisions over time given any input sequence P 1: T , instead of comparing to a single optimal lever decision as in many related works; e.g., see [15] and references therein. This dynamic optimal sequence presents a very strong benchmark and makes learning very difficult as we (ideally) need to account for variations in the underlying non-stationary ground truth input sequence P 1: T . Nevertheless, we do not know the input sequence P 1: T in advance, nor do we know in which world the sequence belongs (Section 2.2).

Bandit function-value (zeroth-order) feedback. Our setup concerns a bandit function-value feedback model, where the realized conversion and constraint functions in each period, namley f t : X → R ≥ 0 and g t : X → R K are never revealed to us, and we can only access their function values at our single lever decisions during the period. We also contrast our setup with a 'two-point feedback' setup in online convex optimization literature, where in each period one can access function values twice (see [44] and references therein). Using the context of our setup, in a 'two-point feedback' after nature samples f t : X → R ≥ 0 and g t : X → R K from P t , we can observe the values for two lever decisions { x (1) t , x (2) t } ⊆ X , namely ( f t ( x (1) t ) , g t ( x (1) t )) and ( f t ( x (2) t ) , g t ( x (2) t )) .

Universal algorithm for many worlds. Instead of developing separate algorithms that cope with each individual world described in Section 2.1, we aim to develop a single universal algorithm that performs well in every world without exploiting any knowledge on the realized input sequence P 1: T or its structural properties.

The key component of our algorithm to handle the aforementioned challenges is the following Lagrangian function: for any f t : X → R ≥ 0 and g t : X → R K , let's define L t : X × R K ≥ 0 → R such that

Surrounding this Lagrangian function, our proposed algorithm is formally stated in Algorithm 1 with four key components described as follows.

1. Dual descent to decouple decisions across time. As our lever decisions over time should be somewhat intertwined with one another due to the presence of long term constraints, we decouple this cross-period dependency by lagrangifying the hindsight problem in Eq. (1) w.r.t. some benchmark dual variables. In each period t , we maintain an estimate of dual variables λ t ∈ R K ≥ 0 corresponding to each of our K constraints, and adopt a standard dual descent approach to dynamically adjust these estimates; see Eq. (9). In particular, after observing ( f t ( x t ) , g t ( x t )) , we update our dual variable by projecting each coordinate of λ t -η ∇ λ L t ( x t , λ t ) = λ t -η g t ( x t ) onto some interval [0 , ¯ F β ] where η > 0 is the dual descent step size, ¯ F is the maximum achievable conversion, and β > 0 is some 'safety buffer' to be defined later.

2. Primal ascent to handle bandit function-valued feedback in many worlds. Given a dual variable λ t during period t , standard optimization frameworks suggest optimizing primal decisions (i.e., lever decisions) by setting x t = arg max x ∈X E P t [ L t ( x , λ t )] . However, in our setting this is not possible due to the bandit function-valued feedback structure: we do not know f t ( · ) , g t ( · ) nor P t , and hence cannot optimize for E P t [ L t ( · , λ t )] . To handle this, we view the primal decision problem as a bandit online convex optimization problem (BOCO) where the adversarial objective functions are L t ( · , λ t ) , and run a BOCO algorithm to make primal lever decisions in each period.

Mathematically, we define the BOCO objective function h t : X → R as

Then, the BOCO algorithm constructs an estimate ∇ t of ∇ h t ( x ) based on a random perturbation approach with perturbation parameter ρ > 0 (see Eq. (7) and [25] for more details in random perturbations methods to estimate gradients). Next, the BOCO algorithm computes primal decision x t by an ascent type update, i.e., projecting x t -1 + γ ∇ t back onto X , where γ > 0 is the primal ascent step size; see Eq. (9). The loss we incur by running BOCO primal update instead of setting arg max x ∈X E P t [ h t ( x t )] in the full information setting depends on the specific type of world we are in, i.e., the input sequence P 1: T . We later show in Lemma 4.3 that input sequences in each world induce some comparator sequence y 1 . . . y T ∈ X , so the conversion loss due to BOCO primal ascent updates can be characterized as some BOCO 'dynamic' regret ∑ t ∈ [ T ] h t ( y t ) -h t ( x BOCO t ) . Later in Lemma 4.6, we show that our BOCO algorithm achieves low dynamic regret against any comparator sequence y 1 . . . y T ∈ X .

3. Choosing primal ascent step sizes from expert advice for different worlds. Different worlds (and correspondingly different BOCO comparator sequences) may require different 'optimal' BOCO primal ascent step sizes in order to achieve optimal regret guarantees. Since we are unaware of the world from which the input sequence P 1: T is generated, on top of our primal ascent procedure we run a meta 'expert' algorithm that allows us to adaptively approximate optimal step sizes in each world. In particular, we consider N experts, each corresponding to a primal ascent step size γ i > 0 ,

and produce independent lever decision based on primal ascent with her own step size; see Eq. (9). Then, we dynamically maintain exponential weights w t ∈ ∆ N in the N -dimensional simplex over N experts, based on their past performance measured by some surrogate loss function defined in Eq. (8). Finally we set the weighted average primal decision over all experts as our ultimate primal decision; see Eq. (6). We later show in Lemma A.3 that such exponentially weighted average BOCO primal ascent decisions achieve low conversion loss compared to the 'optimal' BOCO primal ascent expert (i.e., step size) in every world, without having to know which world we are in.

Constraint violation check to ensure constraint satisfaction. To ensure long-term constraint satisfaction, we maintain a constraint balance B k,t = ∑ τ ∈ [ t -1] g k,t ( x τ ) for each constraint k ∈ [ K ] that keeps track of our deviation from satisfying the constraint. To make our problem tractable, we make the assumption that there is always a 'safe action' with which we are always guaranteed to attain some small positive constraint balance. 2

Assumption 3.1 (Safe action) . Assume there exists some ¯ β > 0 and an action ˜ x β ∈ X such that for any ( f, g ) ∈ S , we have g k ( ˜ x β ) > ¯ β for all k ∈ [ K ] .

With the constraint balance, in each period before making a lever decision, we check the following: if by playing the safe action in all future rounds nearly violates constraints (step 2 in Algorithm 1), then we hard stop our algorithm and play the safe action in all subsequent rounds. We remark that we do not necessarily know the constraint function value lower bound ¯ β > 0 in Assumption 3.1. However, we show later in Theorem 4.1 that by considering any safety buffer β < ¯ β (e.g., taking β = 1 / log( T ) for large enough T ), we are hard stopping more conservatively, and thus maintain constraint satisfaction. Note that our final regret scales with 1 β 2 ; see Lemma 4.6.

Algorithm 1

Input: Initial dual variable λ 1 = 0 , primal expert solutions ˜ x i 1 = 0 for any i ∈ [ N ] ; perturbation parameter ρ > 0 and α ∈ (0 , 1) ; primal ascent step sizes for experts { γ 1 , . . . , γ N } > 0 ; dual descent step size η ; learning rate of the meta-algorithm ϵ ; initial expert weights w i, 1 = 1 for all i ∈ [ N ] ; safety buffer β > 0 .

1: Initialize constraint balance B k, 1 = 0 for all k ∈ [ K ]

2: while {for all k ∈ [ K ] , B k,t -¯ G + β ( T -t -1) ≥ 0 } do

3: Compute exponentially weighted average forecaster :

4: Sample u t ∼ U ( S ) uniformly at random from the unit sphere.

5: Set x t = x t + ρ u t and observe f t ( x t ) and g t ( x t ) . Update B k,t +1 = B k,t + g k,t ( x t ) .

6: Construct gradient estimate for ∇ x L ( x t , λ t )

7: Update exponential weights for experts : Let ℓ t : X → R be a surrogate loss function to measure the performance of each expert. Then we update expert weights by

8: Primal ascent per expert and dual descent :

9: end while

10: Set stopping time τ A = t . For t = τ A . . . T set safety option x t = ˜ x β .

4 Performance analysis of our constrained BOCO algorithm

In this section, we analyze the performance of Algorithm 1 under input sequence P 1: T which may be generated from any world described in Section 2.2. Our first result Lemma 4.1 shows that our algorithm maintains long-term constraint satisfaction almost surely for any input sequence.

˜

Lemma 4.1 (Strict constraint satisfaction) . Suppose Assumption 3.1 hold, and T is large enough so that the safety buffer β = 1 log( T ) < ¯ β , where ¯ β is defined in Assumption 3.1. Then, for any k ∈ [ K ] , we have ∑ t ∈ [ T ] g k,t ( x t ) > 0 .

The main result of this paper is the following Theorem 4.2, where we bound the regret of our proposed Algorithm 1 in all worlds specified in Section 2.2.

Theorem 4.2 (Bounding regret in many worlds) . Let the safety buffer β = 1 log( T ) , and the number of experts N = max ( 1 , ⌈ -log 2 ( K -1 6 (1 + DT ) 1 2 T -3 4 )⌉ +1 ) , where K is the total number of constraints and D is diameter of the decision set X . Choose the corresponding primal ascent step sizes for the N experts as { γ 1 . . . γ N } = { 2 -i K -1 6 (1 + DT ) 1 2 T -3 4 : i = 0 . . . N -1 } . Then by taking dual descent step size η = 1 √ KT , random perturbation parameter ρ = K 1 3 T -1 4 and α = min ( 1 2 , K 1 3 T -1 4 ) , and exponential weighted expert learning rate ϵ = T -1 2 , we obtain the following bounds on R T in each input setting of interest: (1) Stochastic: R T ≤ O ( T 3 / 4 ) ; (2) Adversarial: R T ≤ ( 1 -1 ξ ) OPT ( P 1: T ) + ˜ O ( √ 1 + P ( ˜ y 1: T ) · T 3 4 ) , where ξ = 1 -1 ¯ β min ( f, g ) ∈S min k ∈ [ K ] , x ∈X g k ( x ) > 1 under Assumption 2.1, ¯ β > 0 is defined in Assumption 3.1, ˜ y t = arg max x ∈X f t ( x t ) + λ ⊤ t g t ( x t ) , and P ( ˜ y 1: T ) = ∑ t ∈ [ T -1] ∥ ˜ y t -˜ y t +1 ∥ ; (3) δ -corrupted: R T ≤ O ( T 3 / 4 + δ ) ; (4) Periodic : R T ≤ O ( T 3 / 4 + q √ T ) ; (5) Ergodic : R T ≤ O ( T 3 / 4 + κ √ T ) .

We also summarize our regret bounds in Table 1. Although it is always challenging to identify which input procedure the data currently comes from, Theorem 4.2 shows that the proposed Algorithm 1 achieves reasonable regret bound in five input settings without knowing which setting we are in. We remark that in the adversarial setting, our algorithm is (1 -1 ξ ) -competitive, namely, it can achieve at least a 1 ξ -portion of the reward compared to OPT; later in Section 5 we comment that no sublinear regret is achievable. For the δ -corrupted world, we corrupt δ instances in the input sequence over time, so if δ = 0 , we recover the stochastic world and achieve diminishing O ( T 3 / 4 ) regret; in the case we corrupt all samples, then we recover the adversarial case, and the regret becomes O ( T ) as expected. In the periodic world, if period length q = 1 , we recover the stochastic case.

Remark 4.1. We remark that our regret upper bound in the adversarial case depends on the dual variables { λ t } t ∈ [ T ] generated throughout the algorithm, which contrasts adversarial regret bounds in [44] that only depend on the total variance of the optimal sequence of 1. This is due to that fact that [44] handles a non-constrained online optimization problem, whereas we tackle a more difficult problem with long-term adversarial constraints that leads to more severe regret degradation as we ensure satisfaction of long-term adversarial constraints. We acknowledge that this bound should ideally be independent of dual variables, and we leave this for future work.

In the rest of this section, we present a proof scratch of Theorem 4.2, and the formal proof of the results can be found in Appendix A. The high-level idea of the proof is that we decompose total regret into three components as in Lemma 4.3 bellow, namely the conversion losses due to hard stopping, dual descent, and primal ascent, respectively. Then, we bound each component's regret.

Lemma 4.3 (Regret decomposition) . Let ( λ t ∈ R K ≥ 0 ) t ∈ [ T ] be the sequence of dual variables generated from Algorithm 1, and define h t ( x ) = L t ( x , λ t ) where there Lagrangian function L t ( x , λ ) is defined in Eq. (4) . Then we have

OPT ( P 1: T ) -∑ t ∈ [ T ] E [ f t ( x t )] ≤ ¯ F ( T -τ A ) + ∑ t ∈ [ τ A ] λ ⊤ t g t ( x t ) + R BOCO ( τ A ) (10) where R BOCO ( τ A ) admits the following bounds in each input setting: (1) Stochastic: max x ∈X ∑ t ∈ [ τ A ] h t ( x ) -h t ( x t ) ; (2) Adversarial: ∑ t ∈ [ τ A ] h t ( ˜ y t ) -h t ( x t )+ ( 1 -1 ξ ) OPT ( P 1: T ) where ˜ y t = argmax x ∈X E P t [ f t ( x ) + λ ⊤ t g t ( x )] ; (3) δ -corrupted: max x ∈X ∑ t ∈ [ τ A ] h t ( x ) -h t ( x t ) + O ( δ ) ; (4) Periodic : max x ∈X ∑ t ∈ [ τ A ] h t ( x ) -h t ( x t ) + O ( ηqT ) ; (5) Ergodic : max x ∈X ∑ t ∈ [ τ A ] h t ( x ) -h t ( x t ) + O ( ηκT ) .

Here, we remark that the first term ¯ F ( T -τ A ) is the loss due to hard stopping to maintain long-term constraint satisfaction; the second term ∑ t ∈ [ τ A ] λ ⊤ t g t ( x t ) is the loss due to dual descent; and the

final term R BOCO ( τ A ) is the loss due to primal ascent. The first two components turn out to be identical in different settings, and the input model only affects the last term R BOCO ( τ A ) . We next present our key results with which we use to bound first two components (dual descent and hard stop loss), and the third component (primal ascent loss), respectively.

Bounding dual descent and hard stop loss. Our strategy to bound the loss due to dual descent and hard stopping relies on bounding some cumulative 'complementary slackness' induced from dual descent in the following Lemma 4.4. We note that the bound in Lemma 4.4 holds for any λ ∈ [ 0 , ¯ F β e ] , and thus by choosing appropriate λ , we can 'internalize' the losses due to dual descent and hard stopping, as demonstrted in Lemma 4.5.

Lemma 4.4 (Bounding complementary slackness) . For any λ ∈ [ 0 , ¯ F β e ] and t ∈ [ T ] , we have ∑ τ ∈ [ t ] ( λ τ -λ ) ⊤ g τ ( x τ ) = ∑ τ ∈ [ t ] ( λ τ -λ ) ⊤ ∇ λ L τ ( x τ , λ τ ) ≤ η 2 tK ¯ G 2 + 1 2 η ∥ λ ∥ 2 2

Lemma 4.5 (Internalizing dual descent and hard stop losses.) . Given the stopping time τ A ∈ [ T ] , w.p. 1 we have ¯ F ( T -τ A ) + ∑ t ∈ [ τ A ] λ ⊤ t g t ( x t ) ≤ η 2 TK ¯ G 2 + 1 2 η ( ¯ F β ) 2 + ¯ F + ¯ F β ¯ G .

Bounding primal ascent loss (dynamic BOCO regret). We present our main result Lemma 4.6, which bounds the loss due to primal ascent for given any comparator sequence y 1 . . . y T ∈ X .

Lemma 4.6 (Bounding primal ascent (dynamic BOCO regret)) . For any i ∈ [ N ] , t ∈ [ T ] and any sequence y 1: t ∈ X t we have ∑ τ ∈ [ t ] h τ ( y τ ) -h τ ( x τ ) ≤ O ( ( ρ + α ) T β + 1+ P ( y 1: T ) γ i + γ i KT β 2 ρ 2 + Tϵ + 1 ϵ ) , where P ( y 1: T ) = ∑ t ∈ [ T -1] ∥ y t -y t +1 ∥ , and parameters ( α, γ i , ϵ, ρ ) are specified in Algorithm 1.

Proof sketch. We define the smoothed-versions of BOCO rewards h t : X → R as ˆ h t ( x ) = E v ∼ U ( B ) [ L t ( x + ρ v , λ t )] . Then, we decompose the primal ascent loss (BOCO dynamic regret) ∑ τ ∈ [ t ] ( h τ ( y τ ) -h τ ( x τ )) into 3 terms:

where we recall the ˜ x τ is the un-perturbed version of our primal lever decision in Algorithm 1. The first and third summands can be directly be bounded via exploiting Lipschtiz continuity properties for h t (see Lemma A.1). The second summand can further be upper bounded (using Lemma A.2) by the surrogate loss difference between the primal decision weighted over all expert decisions and the comparator decision, namely ℓ τ ( ˜ x τ ) -ℓ τ ((1 -α ) y τ ) . Then, we choose any expert i ∈ [ N ] corresponding to primal ascent step size γ i > 0 as an 'intermediary' and decompose cumulative surrogate loss as ∑ t ∈ [ T ] ( ℓ t ( ˜ x t ) -ℓ t ((1 -α ) y t )) = ∑ t ∈ [ T ] ( ℓ t ( ˜ x t ) -ℓ t ( ˜ x i t ) )+ ∑ t ∈ [ T ] ( ℓ t ( ˜ x i t ) -ℓ t ((1 -α ) y t )) . The first summand corresponds to the cumulative loss of our unperturbed primal decisions against any expert, and the second summand represents the cumulative loss of any expert w.r.t. the comparator sequence. Both summands are bounded in Lemma A.3.

We note that choosing different experts as the intermediary may yield different bounds. But since Lemma A.3 holds for any expert i ∈ [ N ] , we can consider the expert with the 'optimal' primal ascent step size in each world, respectively.

5 Additional discussions

Lower bounds. Anatural question to ask is whether regret bounds in Theorem 4.2 are optimal. Here, we look at lower bounds in 'relaxed' problem settings, e.g., bandit online convex optimization with no-constraints [11, 44], or online constrained optimization under full-information feedback [8, 10]:

These relaxed lower bounds are applicable to our setting (though they may not be the tightest bounds), and we can see that it may be possible to employ more complex machinery to attain better regret performances for our bandit convex online optimization setup with uncertain constraints. Yet, to identify an algorithm that can achieve the optimal regret in many worlds is an extremely challenging

open problem. For instance, even in the relaxed setting of bandit convex optimization with noconstraints, to the best of our knowledge there is no universal algorithm that achieves optimal regret in just the stochastic and adversarial worlds.

Other applications. Our proposed Algorithm 1 addresses a more general problem of online decision making with multi-dimensional decisions, bandit feedback and long-time uncertain constraints. That being said, our algorithm can also be applicable to other problem settings, for example:

Personalized recommendation and assortment in online retailing. Online retail platforms aim to optimize sales or revenue by recommending a limited assortment of products to customers. However, the likelihood of customer purchases given an assortment as well as their preferences are unknown, and different customer types may change over time due to evolving market trends. Platforms only observe bandit binary purchase decisions. Hence in our context, the decision set X represents a probability simplex over d items. The online decisions x t ∈ X correspond to recommendation probabilities for each item. The function f t ( x t ) reflects the revenue generated from customer purchases, while g t ( x t ) captures assortment capacity constraints, product inventory limits, or fairness considerations to ensure equal visibility for each product [18].

Real time posted pricing in E-commerce and cloud computing. In online E-commerce platforms such as Amazon and eBay, sellers set prices to sell various items to sequentially arriving customers [5, 34, 26]; and in cloud computing, operators of cloud services such as Alibaba Cloud (Alicloud) or Amazon Web Services (AWS) set prices for renting out different computing capacities (virtual machines, VMs etc.) upon customer requests [39]. The arrival of customer types as well as their demand may differ significantly over time (think about demand for sanitary products during pandemics outbursts, or computing resource demands during periodic business hours). Further, decision makers only get to observe (bandit) demand at the realized pricing decisions. Thereby, we can view x t ∈ X as the vector of prices for various sold products, f t ( x t ) as the generated revenue over a given period, g t ( x t ) as real time product/resource capacity constraints, operating cost constraints, etc.

References

Programmatic advertising trends, stats, & news. https://roirevolution.com/blog/ programmatic-advertising-trends-stats-news/ . Accessed: 2023-03-30.

Programmatic digital display advertising in 2022: Ad spend, formats, and forecast. https://www. insiderintelligence.com/insights/programmatic-digital-display-ad-spending/ . Accessed: 2023-03-30.

Shipra Agrawal, Zizhuo Wang, and Yinyu Ye. A dynamic near-optimal algorithm for online linear programming. Operations Research , 62(4):876-890, 2014.

Sanae Amani, Mahnoosh Alizadeh, and Christos Thrampoulidis. Linear stochastic bandits under safety constraints. Advances in Neural Information Processing Systems , 32, 2019.

Moshe Babaioff, Shaddin Dughmi, Robert Kleinberg, and Aleksandrs Slivkins. Dynamic pricing with limited supply, 2015.

Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. Journal of the ACM (JACM) , 65(3):1-55, 2018.

Santiago Balseiro, Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo. Robust auction design in the auto-bidding world. Advances in Neural Information Processing Systems , 34:17777-17788, 2021.

Santiago Balseiro and Yonatan Gur. Learning in repeated auctions with budgets: Regret minimization and equilibrium. Management Science , 65(9):3952-3968, 2019.

Santiago Balseiro, Anthony Kim, Mohammad Mahdian, and Vahab Mirrokni. Budget-management strategies in repeated auctions. Operations Research , 69(3):859-876, 2021.

Santiago Balseiro, Haihao Lu, and Vahab Mirrokni. The best of many worlds: Dual mirror descent for online allocation problems. Operations Research , 71(1):101-119, 2023.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. Advances in Neural Information Processing Systems , 27, 2014.

Mark Braverman, Jieming Mao, Jon Schneider, and Matt Weinberg. Selling to a no-regret buyer. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 523-538, 2018.

Matteo Castiglioni, Andrea Celli, and Christian Kroer. Online learning with knapsacks: the best of both worlds. In International Conference on Machine Learning , pages 2767-2783. PMLR, 2022.

Matteo Castiglioni, Andrea Celli, and Christian Kroer. Online bidding in repeated non-truthful auctions under budget and roi constraints. arXiv preprint arXiv:2302.01203 , 2023.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Giulia Romano, and Nicola Gatti. A unifying framework for online optimization with long-term constraints. Advances in Neural Information Processing Systems , 35:33589-33602, 2022.

Sapana Chaudhary and Dileep Kalathil. Safe online convex optimization with unknown linear safety constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6175-6182, 2022.

Lin Chen, Mingrui Zhang, and Amin Karbasi. Projection-free bandit convex optimization. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2047-2056. PMLR, 2019.

Qinyi Chen, Negin Golrezaei, Fransisca Susan, and Edy Baskoro. Fair assortment planning. arXiv preprint arXiv:2208.07341 , 2022.

Varsha Dani, Sham M Kakade, and Thomas Hayes. The price of bandit information for online optimization. Advances in Neural Information Processing Systems , 20, 2007.

Yuan Deng, Negin Golrezaei, Patrick Jaillet, Jason Cheuk Nam Liang, and Vahab Mirrokni. Fairness in the autobidding world with machine-learned advice. arXiv preprint arXiv:2209.04748 , 2022.

Yuan Deng, Negin Golrezaei, Patrick Jaillet, Jason Cheuk Nam Liang, and Vahab Mirrokni. Multi-channel autobidding with budget and roi constraints. arXiv preprint arXiv:2302.01523 , 2023.

Yuan Deng, Jieming Mao, Vahab Mirrokni, Hanrui Zhang, and Song Zuo. Efficiency of the first-price auction in the autobidding world. arXiv preprint arXiv:2208.10650 , 2022.

Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo. Towards efficient auctions in an auto-bidding world. In Proceedings of the Web Conference 2021 , pages 3965-3973, 2021.

Zhe Feng, Swati Padmanabhan, and Di Wang. Online bidding algorithms for return-on-spend constrained advertisers. In Proceedings of the ACM Web Conference 2023 , pages 3550-3560, 2023.

Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms , pages 385-394, 2005.

Negin Golrezaei, Patrick Jaillet, and Jason Cheuk Nam Liang. No-regret learning in price competitions under consumer reference effects. Advances in Neural Information Processing Systems , 33:21416-21427, 2020.

Negin Golrezaei, Patrick Jaillet, and Jason Cheuk Nam Liang. Incentive-aware contextual pricing with non-parametric market noise. In International Conference on Artificial Intelligence and Statistics , pages 9331-9361. PMLR, 2023.

Negin Golrezaei, Patrick Jaillet, Jason Cheuk Nam Liang, and Vahab Mirrokni. Bidding and pricing in budget and ROI constrained markets. arXiv preprint arXiv:2107.07725 , 2021.

Negin Golrezaei, Vahideh Manshadi, Jon Schneider, and Shreyas Sekar. Learning product rankings robust to fake users. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 560-561, 2021.

Yanjun Han, Zhengyuan Zhou, Aaron Flores, Erik Ordentlich, and Tsachy Weissman. Learning to bid optimally and efficiently in adversarial first-price auctions. arXiv preprint arXiv:2007.04568 , 2020.

Yanjun Han, Zhengyuan Zhou, and Tsachy Weissman. Optimal no-regret learning in repeated first-price auctions. arXiv preprint arXiv:2003.09795 , 2020.

Elad Hazan and Yuanzhi Li. An optimal algorithm for bandit convex optimization. arXiv preprint arXiv:1603.04350 , 2016.

Rodolphe Jenatton, Jim Huang, and Cédric Archambeau. Adaptive algorithms for online convex optimization with long-term constraints. In International Conference on Machine Learning , pages 402-411. PMLR, 2016.

Chia-Wei Kuo, Hyun-Soo Ahn, and Göker Aydın. Dynamic pricing of limited inventories when customers negotiate. Operations Research , 59(4):882-897, 2011.

Nikolaos Liakopoulos, Apostolos Destounis, Georgios Paschos, Thrasyvoulos Spyropoulos, and Panayotis Mertikopoulos. Cautious regret minimization: Online optimization with long-term budget constraints. In International Conference on Machine Learning , pages 3944-3952. PMLR, 2019.

Mehrdad Mahdavi, Rong Jin, and Tianbao Yang. Trading regret for efficiency: online convex optimization with long term constraints. The Journal of Machine Learning Research , 13(1):2503-2528, 2012.

Shie Mannor, John N Tsitsiklis, and Jia Yuan Yu. Online learning with sample path constraints. Journal of Machine Learning Research , 10(3), 2009.

Aranyak Mehta. Auction design in an auto-bidding setting: Randomization improves efficiency beyond vcg. In Proceedings of the ACM Web Conference 2022 , pages 173-181, 2022.

Huijie Peng, Yan Cheng, and Xingyuan Li. Real-time pricing method for spot cloud services with non-stationary excess capacity. Sustainability , 15(4):3363, 2023.

Wen Sun, Debadeepta Dey, and Ashish Kapoor. Safety-aware algorithms for adversarial contextual bandit. In International Conference on Machine Learning , pages 3280-3288. PMLR, 2017.

Jonathan Weed, Vianney Perchet, and Philippe Rigollet. Online learning in repeated auctions. In Conference on Learning Theory , pages 1562-1583. PMLR, 2016.

Scott Yang and Mehryar Mohri. Optimistic bandit convex optimization. Advances in Neural Information Processing Systems , 29, 2016.

Jianjun Yuan and Andrew Lamperski. Online convex optimization for cumulative constraints. Advances in Neural Information Processing Systems , 31, 2018.

Peng Zhao, Guanghui Wang, Lijun Zhang, and Zhi-Hua Zhou. Bandit convex optimization in non-stationary environments. The Journal of Machine Learning Research , 22(1):5562-5606, 2021.

Appendices for

Online Ad Procurement in Non-stationary Autobidding Worlds

A Proofs for Section 4

A.1 Additional definitions for Section 4

Definition A.1 (Total variation between probability distributions) . Consider two distributions {P , P ′ } ⊆ ∆( S ) . Then we define their total variation as ∥P - P ′ ∥ TV = 1 2 ∫ S |P ( s ) -P ′ ( s ) | ds .

We also define the smoothed version of h t : X → R (see Eq. (5)) for any t as follows:

where we recall the Lagrangian function L t is defined in Eq. (4).

A.2 Additional lemmas for Section 4

Lemma A.1 (Lipschitz continuity) . Let Assumption 2.1 hold, and we recall the definitions of h t ( x ) and ˆ h t ( x ) from Eqs. (5) and (11) , respectively, and recall λ 1 . . . λ T as the dual variables generated from Algorithm 1. Then for any { x , x ′ } ⊆ X , we have | h t ( x ) -h t ( x ′ ) | ≤ (1 + K ¯ F β ) L · ∥ x -x ′ ∥ and ∣ ∣ ∣ h t ( x ) -ˆ h t ( x ) ∣ ∣ ∣ ≤ (1 + K ¯ F β ) Lρ .

Lemma A.2 (Bounding BOCO dynamic regret with surrogate loss) . Recall the definition ˆ h t ( x ) = E v ∼ U ( B ) [ L t ( x + ρ v , λ t )] . Then, ˆ h t ( x ) is concave. Further, For any y ∈ (1 -α ) X , we have ˆ h t ( y ) -ˆ h t ( ˜ x t ) ≤ E u t ∼ U ( S ) [ ℓ t ( ˜ x t ) -ℓ t ( y )] , where ˜ x t is defined in Eq. (6) , and the surrogate loss function ℓ t : X → R is defined in Eq. (8) .

Lemma A.3 (Bounding surrogate loss for each expert) . Recall the definition of individual forecasters ˜ x i t defined in Eq. (9) , and the surrogate loss function ℓ t : X → R defined in Eq. (8) . Then for any i ∈ [ N ] and any sequence y 1: T ∈ X T , by defining P ( y 1: T ) = ∑ t ∈ [ T -1] ∥ y t -y t +1 ∥ (see Lemma 4.6), we have (i) ∑ t ∈ [ T ] ( ℓ t ( ˜ x i t ) -ℓ t ((1 -α ) y t )) ≤ O ( 1+ P ( y 1: T ) γ i + γ i β 2 ρ 2 T ) and (ii) ∑ t ∈ [ T ] ( ℓ t ( ˜ x t ) -ℓ t ( ˜ x i t ) ) ≤ O ( Tϵ + 1 ϵ ) . where the constant β is specified in Algorithm 1. Here, recall D is the diameter of the decision set X .

The proofs of Lemmas A.1, A.2,A.3 are shown in Appendices A.9, A.10, and A.11, respectively.

A.3 Proof for Lemma 4.1

Proof. For any k ∈ [ K ] we have

where in ( a ) we set x t = ˜ x β for all t = τ A . . . T and g k,t ( ˜ x β ) ≥ ¯ β for any k ∈ [ K ] ; ( b ) follows from the definition of the stopping time such that for any t ′ < τ A and k ∈ [ K ] we have ∑ t ∈ [ t ′ ] g k,t ( x t ) -¯ G + β ( T -t ′ -1) ≥ 0 .

A.4 Proof for Lemma 4.4

Proof. It is easy to see λ t +1 = Π [ 0 , ¯ F β e ] ( λ t -η ∇ λ L t ( x t , λ t )) + = arg min λ ∈ [ 0 , ¯ F β e ] ∇ λ L t ( x t , λ t ) ⊤ λ + 1 2 η ∥ λ -λ t ∥ 2 . By the first-order stationary condition

at λ t +1 , we have for any λ ∈ [ 0 , ¯ F β e ]

Then for all λ ∈ R K ≥ 0 , it follows that

By a telescoping argument, we have

where in the final equality we used λ 1 = 0 . Also,

Hence, combining Eqs. (13) and (14), we get the desired bound.

A.5 Proof of Lemma 4.5

Proof. If τ A = T , taking λ = 0 in Lemma 4.4 yields ∑ t ∈ [ T ] λ ⊤ t g t ( x t ) ≤ η 2 TK ¯ G 2 and thus the desired inequality holds. If τ A < T , then there exists some k ∈ [ K ] such that ∑ t ∈ [ τ A ] g k,t ( x t ) -¯ G + β ( T -τ A -1) < 0 , so by taking λ = ¯ F β e k ( e k ∈ R K is the unit vector whose k th entry is 1) in Lemma 4.4 yields

which completes the proof.

A.6 Proof of Lemma 4.6

Proof. Recall the definition of h t ( x ) in Eq. (5). Then, we have

Bounding A .

where (a) follows from Lemma A.1; (b) follows from ∥ y τ ∥ = ∥ y τ -0 ∥ ≤ D since we assumed 0 ∈ X .

Bounding B .

where (a) follows from Lemma A.2 and (b) follows from Lemma A.3 (i) and (ii).

Bounding C .

where (a) follows Lemma A.1; (b) is from the definition x τ = ˜ x τ + ρ u τ in Algorithm 1.

A.7 Proof of Lemma 4.3

In this section, we provide upper bounds of the regret term under five different environments of input procedures: stochastic, adversarial, δ -corrupted, periodic, and ergodic. The structure of the following proof might seem similar to [10]; see [10, Sections 3-5]. However, we note that the proof techniques are fundamentally different since our paper considers the bandit feedback environment while [10] makes sequential decisions after observations.

Stochastic.

Proof. In the stochastic regime, we have P = P 1 = · · · = P T for some P , and therefore we can rewrite OPT ( P 1: T ) in Eq. (1) as followed

where we defined F ( x ) = E ( f, g ) ∼P [ f ( x )] , and G ( x ) = E ( f, g ) ∼P [ g ( x )] for any x ∈ X . Hence, for any λ ≥ 0 we have

where in the inequality we applied Assumption 2.1 which states sup x ∈X | f ( x ) | ≤ ¯ F for all ( f, g ) ∈ S . Choosing λ = ¯ λ τ A := 1 τ A ∑ t ∈ [ τ A ] λ t we have

where in ( a ) we used the fact that λ t is H t -1 -measurable; in ( b ) we used definitions h t ( x ) = L t ( x ; λ t ) and L t ( x ; λ ) = f t ( x ) + λ ⊤ g t ( x ) in Eqs. (4) and (5) respectively. On the other hand, we have

so combining this with Eq. (20) we have

where we also used the fact that f t ( x ) ≥ 0 for all t = τ A +1 . . . T and x ∈ X .

Adversarial.

Proof. Recall the definition of ξ is Theorem 4.2:

For any t ∈ [ T ] , define ˜ y t = arg max x f t ( x ) + λ ⊤ t g t ( x ) .

By comparing to the safety action x β ∈ X which ensures g k ( x β ) ≥ ¯ β for any k ∈ [ K ] and ( f, g ) ∈ S , as well as the optimal hindsight action x ∗ t ∈ X (i.e., x ∗ 1 . . . x ∗ T is the optimal decision sequence to OPT ( P 1: T ) ), we have

We further have

where (a) follows Eq. (24); in (b) we used the fact that g k,t ( x ∗ t ) + ( ξ -1) ¯ β ≥ 0 since we have min ( f, g ) ∈S min k ∈ [ K ] , x ∈X ( g k,t ( x ) + ( ξ -1) ¯ β ) ≥ 0 (see Eq. (23)). Hence we have

which completes the proof.

δ -corrupted.

Here, we will prove a more general δ -corrupted model where the input distribution sequence P 1: T satisfies the following:

where the total variation norm is defined in Definition A.1. In fact, the definition in Section 2.2 for the δ -corrupted regime satisfies the above property: recall the definition in Section 2.2, there exists P ∈ ∆( S ) as well as δ ∈ N periods T = { τ 1 . . . τ δ } ⊂ [ T ] such that P t = P for all t / ∈ T , so for any t / ∈ T , we have

On the other hand, we have for any t ∈ T , ∥P t -1 T ∑ s ∈ [ T ] P s ∥ TV ≤ 1 2 . After summing it up, we get

which coincides with our general definition of δ -corruption in Eq. (27).

We now prove the δ -corruption regime under the general definition in Eq. (27). Define ˜ P = 1 T ∑ s ∈ [ T ] P s , ˜ F ( x ) = E ( f, g ) ∼ ˜ P [ f ( x )] , ˜ G ( x ) = E ( f, g ) ∼ ˜ P [ g ( x )] , F t ( x ) = E ( f, g ) ∼P t [ f ( x )] and G t ( x ) = E ( f, g ) ∼P t [ g ( x )] for all t ∈ [ T ] and any x ∈ X . Then for any λ ∈ [ 0 , ¯ F β e ] , we have

where the last inequality follows the definitions of ( ˜ F, ˜ G ) , Assumption 2.1, and the general definition of δ -corruption in Eq. (27). After choosing ¯ λ = 1 τ A ∑ t ∈ [ τ A ] λ t , similar to our proof in Eq. (20) for the stochastic case we have

where (a) follows from Eq. (29); (b) follows from the definition of general δ -corruption in Eq. (27). Finally, we complete the proof by using the definition f t ( x t ) = h t ( x t ) -λ ⊤ t g t ( x t ) and following the same argument as in Eq. (22) for the stochastic regime.

Periodic.

Recall in Section 2.2 that in the periodic regime, there exists cycle length q ∈ N such that T = cq for some integer c ≥ 2 with P 1: T as P 1: q = P q +1:2 q = · · · = P ( c -1) q +1: T . For any t ∈ [ T ] , define c t ∈ [ c ] such that ( c t -1) q +1 ≤ t ≤ c t q . After denoting ˜ P = 1 q ∑ t ∈ [ q ] P t , we define the mean deviation within a single cycle of length q as

where the equality follows the nature of periodic setting and the last inequality follows the definitions of ( ˜ F, ˜ G ) , Assumption 2.1, and (31). After choosing λ = ∑ ˆ c ∈ [ c τ A -1] q τ A λ (ˆ c -1) q +1 +

From Eq. (9) in Algorithm 1, we know that ∥ λ t +1 -λ t ∥ 1 ≤ η ¯ GK , which further implies ∥ λ t + i -λ t ∥ 1 ≤ η ¯ GKi for any i ∈ [ q -1] and thus

After combining the two equations above, it follows that

where the second last inequality follows from c τ A q ≤ cq = T .

Finally, we complete the proof by using the definition f t ( x t ) = h t ( x t ) -λ ⊤ t g t ( x t ) and following the same argument as in Eq. (22) for the stochastic regime.

Ergodic.

Consider some κ ≥ log( T ) . Given the input distribution sequence P 1: T , denote P ( t + κ ) | [ t -1] as the conditional distribution of ( f t + κ , g t + κ ) conditioned on the { ( f τ , g τ ) } τ ∈ [ t ] . Then, in the ergodic regime, there exists a stationary distribution ˜ P ∈ ∆( S ) and absolute constant R > 0 such that

see [10, Section 5.2] for further discussions. By defining ˜ F ( x ) = E ( f, g ) ∼ ˜ P [ f ( x )] , ˜ G ( x ) = E ( f, g ) ∼ ˜ P [ g ( x )] , ˆ F t + κ ( x ) = E ( f, g ) ∼P ( t + κ ) | [ t -1] [ f ( x )] , ˆ G t + κ ( x ) = E ( f, g ) ∼P ( t + κ ) | [ t -1] [ g ( x )] , F t ( x ) = E ( f, g ) ∼P t [ f ( x )] and G t ( x ) = E ( f, g ) ∼P t [ g ( x )] for all t ∈ [ T ] and any x ∈ X , we know that for any λ ∈ [ 0 , ¯ F β e ] , it follows that

By choosing λ = 1 τ A ∑ t ∈ [ τ A ] λ t , we further have

where in (a), from (9) in Algorithm 1, we know that ∥ λ t +1 -λ t ∥ 1 ≤ η ¯ GK , which further implies ∥ λ t + κ -λ t ∥ 1 ≤ κη ¯ GK and thus

In (b), we used the fact that for any t ≥ κ +1 , we have

In (c) we used the fact that κ ≥ log( T ) , so δ = R exp( -κ ) ≥ R .

Finally, we complete the proof by using the definition f t ( x t ) = h t ( x t ) -λ ⊤ t g t ( x t ) and following the same argument as in Eq. (22) for the stochastic regime.

A.8 Proof of Theorem 4.2

Proof. We bound the regret in every world as followed

where (a) follows from Lemma 4.3, and (b) follows from Lemma 4.5. Recall R BOCO ( τ A ) is specified in Lemma 4.3 for each world.

In the following we bound R BOCO ( τ A ) for each world.

Stochastic.

where (a) follows from Lemma 4.6 by taking the comparator sequence y t = arg max x ∈X ∑ t ∈ [ τ A ] h t ( x ) for all t ∈ [ τ A ] such that P ( y 1: T ) = 1 , as well as any primal ascent expert i ∈ [ N ] ; (b) follows from taking η = 1 √ KT , ρ = K 1 3 T -1 4 , ϵ = T -1 2 , β = 1 log( T ) , and finally choosing γ i = K -1 6 (1 + DT ) 1 2 T -3 4 . Recall all primal ascent expert stepsizes are { γ 1 . . . γ N } = { 2 -i K -1 6 (1 + DT ) 1 2 T -3 4 : i = 0 . . . N -1 } .

δ -corrupted, Periodic, and Ergodic. The proof is nearly identical with that of the stochastic world in Eq. (37) given that we still consider the comparator sequence y t = arg max x ∈X ∑ t ∈ [ τ A ] h t ( x ) for all t ∈ [ τ A ] such that P ( y 1: T ) = 1 . Hence we will omit the proof.

Adversarial. Recall the definition ˜ y t = arg max x ∈X f t ( x t ) + λ ⊤ t g ( x t ) . Then we have

where we chose the primal ascent stepsize γ i s.t.

We note that such a γ i must exist because P ( ˜ y 1: T ) ≤ DT given all ˜ y t ∈ X , so that the largest element in the primal ascent stepsize set, namely K -1 6 (1+ DT ) 1 2 T -3 4 is larger than the upper bound above, namely K -1 6 (1 + P ( ˜ y 1: T )) 1 2 T -3 4 .

A.9 Proof for Lemma A.1

Proof. Recall the definition h t ( x ) = f t ( x ) + λ ⊤ t g t ( x ) in Eq. (5). Then we have

where (a) follows from the fact that any ( f, g ) ∈ S are L -lipschitz under Assumption 2.1.

On the other hand, recall the definition ˆ h t ( x ) = E v ∼ U ( B ) [ L t ( x + ρ v , λ t )] in Eq. (11). Then we have

where the inequality follows from the first part of this lemma.

A.10 Proof of Lemma A.2

Proof. Recall the definitions h t ( x ) = f t ( x ) + λ ⊤ t g t ( x ) in Eq. (5), and ˆ h t ( x ) = E v ∼ U ( B ) [ L t ( x + ρ v , λ t )] in Eq. (11). Then, we have

where (a) follows from concavity of ˆ h t ( · ) ; (b) follows from Lemma B.2 by taking h = -h t , so that in the lemma -∇ x E v ∼ U ( B ) [ h ( x + ρ v )] = ∇ ˆ h t ( x ) and -E u ∼ U ( S ) [ h ( x + ρ u ) · u ] = E u ∼ U ( S ) [ h t ( x + ρ u ) · u ] ; (c) follows from the gradient estimate in Eq. (7) where

Finally, (d) follows from the definition of surrogate loss functions in Eq. (8).

A.11 Proof of Lemma A.3

Proving (i):

Proof. Let's denote ∇ t = ∥ ∇ t ∥ . Since ˜ x i t +1 = Π (1 -α ) X ( ˜ x i t + γ i ∇ t ) we have ∥ y -˜ x i t +1 ∥ ≤ ∥ y -( ˜ x i t + γ i ∇ t ) ∥ for any y ∈ (1 -α ) X . Then

Hence by taking y = (1 -α ) y t ∈ (1 -α ) X and rearranging we get

Telescoping with τ = 1 . . . t we get

where P ( y 1: T ) is defined in the lemma statement.

Proving (ii):

Proof. First, we have for any t ∈ [ T ] , i ∈ [ N ]

where we recall D = sup { x , x ′ }⊆X ∥ x -x ′ ∥ is the diameter of X , and both { ˜ x i t , ˜ x t } ⊆ (1 -α ) X . Define W t = ∑ i ∈ [ N ] w i,t for all t ∈ [ T ] , then

Here (a) follows from Hoeffding's Lemma as described in Lemma B.1 where we take X = ℓ t ( ˜ x I t t ) , a = -d ρ ( ¯ F + K ¯ F β ¯ G ) · (1 -α ) D and b = d ρ ( ¯ F + K ¯ F β ¯ G ) · (1 -α ) D ; (b) follows from the definition that ℓ t ( ˜ x ) = ∇ T t ( ˜ x -˜ x t ) is a linear function in ˜ x ; (c) follows from Eq.(6).

Hence, telescoping the above we get

On the other hand, we have

Hence, after combining Eqs. (47) and (48), and dividing both sides by ϵ > 0 , we get

which completes the proof.

B Supplementary lemmas

Lemma B.1 (Hoeffding's lemma) . Let X be some random variable such that a ≤ X ≤ b almost surely for some a, b ∈ R . Then for any ϵ ∈ R , we have E [exp( -ϵX )] ≤ exp ( -ϵ E [ X ] + ϵ 2 ( b -a ) 2 8 ) .

Lemma B.2 ([25] Lemma 2.1) . Let h : X → R be some convex function (not necessarily differentiable). Then for any x ∈ X ⊆ R d and δ > 0 we have

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies an advertiser's online high-dimensional lever decision problem with long-tern constraints under limited bandit feedback for different input models. The authors' main contributions include: (1) model formulation; (2) proposing an algorithm universally applicable to input models; (3) theoretical regret analysis. \n\nThere is an advertiser that repeatedly interact with an ad platform during a time horizon $T$, aiming to maximize her total conversions subject to multiple constraints. At each time $t$, the advertiser need to make a multi-dimensional lever decision and observed her realized conversion as well as her multi-dimensional realized cost. The authors propose an algorithm with universally good performance and provide regret lower bounds of this problem with respect to different input procedures.\n\nSTRENGTHS:\n1. The proposed algorithm is oblivious to input models such that it can achieve high performance without knowing which setting the decision-maker is in.\n\n2. It is novel to adopt the random perturbation approach and the expert-based decision-making in online autobidding. The regret analysis requires non-trivial insights and techniques. Particularly, the proof of lemma 4.6 that bounds the primal ascent regret stands out from standard primal-dual frameworks.\n\n3. The organization of this paper is great.\n\nWEAKNESSES:\n1. The idea of solving a problem in many worlds is not new. In addition to commonly studied stochastic and adversarial input models, [1] also considers $\\delta$-corrupted, periodic, ergodic input models. Actually, the proofs for the latter three cases are analogous to the stochastic case as they still assume stationary distributions to some extent.\n\n[1] Santiago Balseiro, Haihao Lu, and Vahab Mirrokni. The best of many worlds: Dual mirror descent for online allocation problems.\n\n2. Some mistakes are spotted. Some statements are not clear enough. See the questions below.\n\nQUESTIONS:\n1. Line 13, multi-dimension -> multi-dimensional. \n\n2. Line 14. What does \"uncertain\" mean throughout this paper? The bidder should known their constraints at the beginning, e.g. known budget.\n\n3. Line 168, a redundant \"of\". \n\n4. Line 219, realzied -> realized. \n\n5. Line 283, violates constraints violation?\n\n6. Line 664, missing values for $a$ and $b$.\n\n7. Line 288, what does $\\alpha$ do in algorithm 1? To my understanding, it makes sure that the opinion of an expert lies in the interior of the domain. However, the value of $\\alpha$ is not specified. It seems fatal if $\\alpha$ is close to 1.\n\n8. Line 299, the number of experts $N$ seems to be negative when $T$ goes to infinity.\n\n9. Line 301, N experts or (N+1) experts?\n\n10. Line 539, the first term in the RHS of equation (15) is not bounded after timing $T$.\n\n11. Line 542, $\\tau$ -> $t$. The authors should check $t\\in [T]$ and $\\tau\\in [t]$ throughout the appendix. the order of $\\beta$ also seems wrong in equation (16).\n\n12. Line 626, observe that the sum operator is over $t\\in [\\tau_A]$ so the optimal $x$ does not vary to $t$. Does it implies all $y_t$ are the same?\n\n13. Line 629, arer -> are.\n\n14. Line 636, according to the theorem 4.2 one has $\\gamma_{0} = K^{-1/6} (1+DT)^{1/2} T^{-3/4}$ and $\\gamma_{N} = 1$. Then $\\gamma_{0}$ is not the largest element in the stepsize set since it goes to $0$.\n\nBe willing to raise my rating if it is confirmed there is no technical flaw.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of dynamic online allocation under constraints with bandit feedback, and derives a generic algorithm applicable to various input settings (stochastic, adversarial, $\\delta$-corrupted, ergodic, periodic). It recovers regret rates close to those of the lower bounds in each of these settings. The algorithm uses a dual gradient descent over $\\lambda_t$ to decouple the decisions over time by considering the lagrangian, a gradient ascent for the optimal choice of $x_t$ (that also uses the technique from Flaxman et al (2004) to handle the bandit feedback), and finally a multiplicative weight update to adapt the learning rates used to the correct input setting.\n\nSTRENGTHS:\n- The algorithm generalizes to multiple input settings and general constraints the problems related to Bandits with knapsack constraints, and of online learning with constraints. (to be clear this is an important strength of this paper!)\n- The problem is well motivated through the consideration of running multiple ad campaigns\n\nWEAKNESSES:\n- The assumption that there exists some safe action of level $\\beta$ do simplify the problem of constraint satisfaction by guaranteeing to be able to satisfy the constraint at the end\n- Some of the upper bounds (stochastic, periodic, corrupted) are a bit loose compared to their respective lower bounds\n- The writing can be improved, in particular in the proofs which are lacking discussion about their main ideas and intuition. As an example page $8$ of the appendices is almost only a sequence of inequality and is hard to follow. For instance the inequalities could be cut in multiple parts, and some comments could be added to better explain the goal of the proof. There are also some typos.\n\nQUESTIONS:\n- Do the $T^{2/3}$ rates come from the use of the method to handle bandit feedback? How would a generic gradient feedback affect the rates, would we be able to derive tight rates with respect to the lower bounds presented? I think it might have been better to first present the results with gradient feedback, and mention in the appendix that bandit feedback can be handled by a standard technique. This would allow the main part of the paper to focus on the new contributions. \n- Could you give some intuition on what this optimal dynamic sequence looks like in the various settings? For instance, if my understanding is correct, the optimal dynamic sequence in the stochastic setting is simply a unique point (because the data is i.i.d). Could the optimal sequence for the ergodic setting be a function of $\\kappa$ close to the unique optimal point with respect to the stationary measure? \n- How much does the meta algorithm degrade the regret? (compared to assuming that we know the input setting)\n- l 657 Why is it $... +D \\dots$ and not $...+(t-1)D\\dots $ ?\n \nComments/Typos:\n- L304 and l324 stochstic ->  stochastic\n- I find the notation of $\\lambda \\in [0,F e/ \\beta]$ confusing, is it $\\lambda \\in [0,F/\\beta]^K$?\n- L553 ‘which states $\\max_{x \\in \\mathcal{X}} f(x)$: is a word missing?\n- L608 I am not sure to understand this statement, could you include a reference for this result? Is the sup taken over $(g_{\\tau},f_{\\tau})$ - over $[t]$ or $[T]$? \n- I think it would be nice to cite the paper of Mannor et al (Online learning with sample path constraints), which deals with very similar problem and started the works on online convex optimization with varying constraints\n- Equation below l643: $v$ -> $\\Vert v \\Vert$, and the last equality should be an inequality (as it is in the ball, not the sphere)\n- Equation below l655 I think some of the gradients are missing $\\Vert \\nabla_t \\Vert^2$ and $x^i_{t+}$ ->  $\\tilde{x}^{i}_{t}$. Why are some of the gradients bolded and not the others? \n- Same thing for the gradients below l657, in addition I am not sure where $P(y_1:T)$ is defined.\n- L 664 ‘$a=$ and $b=$’ it is unfinished \n\nI have not read through all the proofs, but I would recommend to read it again to look for additional typos\n\nLIMITATIONS:\nYes the authors address some limitations in the section $5$ of the paper regarding the lower bounds.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concerns a two-stage autobidding scenario, such as an advertising platform environment.  Each advertiser wants to maximize value received (e.g., clicks) subject to long-run constraints (e.g., budget or ROI).  As actions, the advertiser can specify certain instructions to an autobidding agent (e.g., a spend rate or ROAS target) and observe the results.  The advertiser observes bandit feedback, and wishes to tune their choice of actions/instructions to solve their long-run optimization problem.\n\nBecause the autobidders themselves are learning over time and potentially facing changing market conditions, the evolution of payoffs observed by the advertiser may not be stationary.  The paper considers a variety of different payoff evolution models, including partial adversarial corruption, periodic, and ergodic payoffs.  \n\nThe main result is a universal learning method for the advertiser that achieves good regret for each of these payoff evolution models.  The idea is to combine dual descent methods for constraints with a modified online convex optimization approach to adequately explore lever decisions given the dual variables.  This combination leads to vanishing regret in each of the settings considered.  The resulting regret rates are then compared with known lower bounds for each of these settings (many of which apply in relaxed settings, such a full feedback rather than bandit feedback).\n\nSTRENGTHS:\nI like this paper.  The modeling framework that separates true \"long-run\" objectives from \"short-term\" directive levers is extremely natural and, as far as I'm aware, novel.  It also tracks my understanding of how autobidding works in practice: advertisers need not keep their specified constraints (to the autobidder) fixed over time, but can manipulate them online as tunable knobs.  \n\nThe assumption of a safe action is likewise very reasonable, and the proposed algorithm makes use of it to good effect.  \n\nThe proposed algorithm combines multiple well-established ideas from the online optimization literature in a reasonable way.  The fact that this comes together into a unifying framework is an appealing feature, as is the need for only bandit feedback.  The regret rate suffers somewhat compared to known bounds, but not by too much --- what amounts to a rate of T^{3/4} for each of the non-fully-adversarial settings is quite good, while leaving room for future work to improve.  Getting these regret rates down to sqrt{T} (or showing this isn't possible) is a nice open challenge.\n\nWEAKNESSES:\nThe biggest question for me is how \"real\" autobidding (e.g., competitive uniform bidding) falls into this scenario.  For instance: even in a stationary environment in terms of competitors, the relationship between levers and outcomes is not necessarily stationary for advertiser i because the underlying autobidder is learning over time how best to satisfy the directive communicated by a given lever setting.  So my understanding is that this scenario would fall under the ergodic setting.  But what if the competitors are not stationary, but are learning as well?  Can these theoretical frameworks be linked back to the motivating setting of autobidders that simultaneously learn?  Either way, it would be nice to have a more thorough discussion of this in the body of the paper.\n\nAnother potential weakness is that the technical contributions largely synthesizes known approaches, so the marginal technical contribution is not extremely high.  I therefore view the conceptual and modeling contributions as the main selling points for the submission.\n\nQUESTIONS:\nAre there natural conditions under which a scenario with mutually competing autobidders would fall into one of the analyzed worlds?\n\nLIMITATIONS:\nI feel that the paper is sufficiently up-front about its limitations, and the authors do an adequate job of describing what their paper does and does not do.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes a universally constrained online learning framework for ad procurement in non-stationary autobidding worlds. The paper makes contributions to the field by addressing the challenges of ad procurement in non-stationary autobidding worlds and developing a unified algorithm that can perform well in autobidding world while satisfying long-term constraints.

STRENGTHS:
- The paper addresses the challenges of ad procurement in non-stationary autobidding worlds and develops a unified algorithm that can perform well in autobidding world while satisfying long-term constraints.
- The paper makes contributions to the field by developing an algorithm that yields good performance guarantees under different procedures.
- The paper is well-written and clearly presents the problem and the proposed solution.

WEAKNESSES:
- The paper does not have experiments to validate the theoretical findings.
- The paper could provide more insights into the practical implications of the proposed algorithm and how it can be applied in real-world settings.

QUESTIONS:
What are the challenges with applying the algorithm to obtain experimental results?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10