SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jason Cheuk Nam Liang
jcnliang@mit.edu
Haihao Lu
haihao.lu@chicagobooth.edu
Baoyu Zhou
zbaoyu@umich.edu
Today's online advertisers procure digital ad impressions through interacting with autobidding platforms: advertisers convey high level procurement goals via setting levers such as budget, target return-on-investment, max cost per click, etc. Then ads platforms subsequently procure impressions on advertisers' behalf, and report final procurement conversions (e.g. click) to advertisers. In practice, advertisers may receive minimal information on platforms' procurement details, and procurement outcomes are subject to non-stationary factors like seasonal patterns, occasional system corruptions, and market trends which make it difficult for advertisers to optimize lever decisions effectively. Motivated by this, we present an online learning framework that helps advertisers dynamically optimize ad platform lever decisions while subject to general long-term constraints in a realistic bandit feedback environment with non-stationary procurement outcomes. In particular, we introduce a primal-dual algorithm for online decision making with multi-dimensional decision variables, bandit feedback and long-term uncertain constraints. We show that our algorithm achieves low regret in many worlds when procurement outcomes are generated through procedures that are stochastic, adversarial, adversarially corrupted, periodic, and ergodic, respectively, without having to know which procedure is the ground truth. Finally, we emphasize that our proposed algorithm and theoretical results extend beyond the applications of online advertising.
Automated bidding (or autobidding for short) has become the dominant mode for advertisers to procure digital ad inventories and impressions, contributing to more than $120 billion dollar ad spend in 2022 and more than  $90\%$  of total online ad transaction volumes [2, 1]. In an autobidding platform, advertisers only need to convey their high-level procurement goals for an ad campaign to the platform, which then takes charge of procuring ads on advertisers' behalf. Such procurement goals are communicated to a platform through platform levers, which are adjustable parameters that advertisers can control to influence the bidding process and campaign performance. To exemplify, Figure 1 displays certain several levers presented on the Google Ads interface, where an advertiser can set per-campaign budgets, target cost-per-actions, campaign duration, campaign schedules, targeting, etc; similar examples are also shown in related literature [21].
As the primary avenue for advertisers to run ad campaigns on autobidding platforms and influence ad conversion outcomes (e.g., clicks), making efficient lever decisions is vital to advertisers to achieve their procurement objectives. However, advertisers face many challenges in practice when optimizing for lever decisions, namely high-dimensional decision making under long-term constraints, non-stationary autobidding environments, and limited procurement feedback.
High-dimensional decision making and long-term constraint satisfaction. Making multiple lever decisions involves evaluating numerous possible combinations of lever configurations, which is computationally intensive and time-consuming in real-time decision making setups. Also, advertisers need to understand the potential interactions and dependencies between levers, as adjusting one lever may have unintended consequences or interactions with other levers, making it challenging to
Figure 1: Example levers on the Google Ads interface to create digital advertising campaigns.
predict the overall impact of adjustments accurately. Furthermore, advertisers may need to satisfy certain long-term constraints over time, e.g., limiting total spend under a budget, or achieving certain return-on-investment targets. Hence in addition to analyzing the interactions between different levers, advertisers also need to concern long-term consequences of making certain lever decisions.
Limited feedback on procurement outcome and constraints. Despite the fact that autobidding via lever decisions greatly simplifies advertisers' ad procurement process as they no longer have to handle the intricacies of procuring individual ad impressions, the procurement procedure becomes a black box for advertisers, as advertisers only have limited visibility into the specific details of how the platform executes ad placement processes. This lack of transparency can make it challenging for advertisers to control the nuances of their campaign execution through lever decisions, and it amplifies the complexity to conduct counterfactual analyses on outcomes for past lever decisions.
Many non-stationary autobidding worlds. Autobidding procurement environments are highly non-stationary, as a wide spectrum of latent factors in online ad marketplaces may greatly vary procurement outcomes of the same lever decisions in different time periods. These latent factors include but are not limited to changing user preferences, seasonality effects, shifts in market trends, occasional malfunctions in autobidding platforms, etc. These dynamics may influence how users interact with different types of ads, which necessitates continuous adjustment of lever decision strategies to adapt to current and future market conditions.
To address these challenges, in this work we answer the following questions: How should an advertiser dynamically set multiple levers to optimize conversion outcomes subject to long-term constraints under limited information? And can she run a unified algorithm that can perform well while being agnostic to many types of non-stationary autobidding procurement environments?
Motivated by these questions, in this work we study an advertiser's online high-dimensional lever decision problem with long-term constraints under limited bandit feedback in many non-stationary worlds. We summarize our main contributions as followed:
1. Modelling online lever decisions in many worlds using online constrained optimization with bandit function-valued feedback and uncertain constraints (Section 2). We cast the advertiser lever decision problem as an online constrained optimization with bandit function-valued feedback and uncertain constraints, where functions at which (lever) decisions are evaluated correspond to the conversion and constraint functions in our autobidding setup. Further, we model real-world nonstationarity in autobidding environments as possibly time-varying distributions from which conversion and constraint functions are sampled, and then further applied to lever decisions. Under this model, we discuss five different input procedures from which the sequence of functional distributions are sampled to model stochastic, adversarial, corrupted, periodic, and ergodic environments; see Section 2.2 for more details. To the best of our knowledge, from a modelling perspective this is the first work to model a high-dimensional lever decision problem in practical non-stationary autobidding worlds.
2. Proposing a constrained bandit optimization algorithm universally applicable to many worlds (Section 3). We develop a unified bandit optimization algorithm with robust performance guarantees across different worlds. Our algorithm incorporates four key designs to handle bandit function-valued feedback and unknown non-stationary environments. 1. we utilize dual descent to update dual variables associated with long-term constraints and decouple decisions over time; 2. we employ a primal-ascent-based bandit online convex optimization (BOCO) approach to make (primal) lever decisions to cope with bandit function valued feedback; 3. we implement an exponential weights
expert algorithm on top of primal-ascent BOCO, where each expert corresponds to a different primal ascent step size. This enables our algorithm to adapt to the optimal primal ascent step size for each world without prior knowledge on which world we are in; 4. our algorithm dynamically checks for realized constraint violations and applies "safe lever decisions" to ensure long-term constraint satisfaction. For further details, we refer the readers to Algorithm 1 in Section 3.
3. Analyzing the performance of proposed algorithms in many worlds (Section 4). We present theoretical analysis (Theorem 4.2) on the regret bound of the proposed algorithm, and we show that our unified algorithm can achieve reasonable regret bounds with all five input procedures. These regret bounds are summarized in the following Table 1.
Stochastic	δ-corrupted	Adversarial	Periodic	Ergodic
O(T3/4)	O(T3/4 + δ)	(1 - 1/ξ) OPT(P1:T) + O(A · T3/4)	O(T3/4 + q√T)	O(T3/4 + κ√T)
Table 1:  $T$ -period regret bounds for Algorithm 1 under different input models. Here  $A > 0$  is some algorithm dependent factor which we will later specify in Section 4.
The parameters  $(\delta, \xi, \mathrm{OPT}(\mathcal{P}_{1:T}), q, \kappa)$  are formally defined in Section 2.2 and Theorem 4.2. Finally, in Section 5 we remark all results are applicable to other problems.
1.1 Related works.
Autobidding. In an autobidding framework, there is a considerable body of works that study the price of anarchy, which aims to improve worst case individual or total advertiser welfare guarantees w.r.t. the optimal welfare via mechanism design frameworks; see e.g. [23, 7, 20, 38, 22]. We remark that this line of work does not concern developing online learning algorithms. On the other hand, numerous works have concentrated on developing online bidding algorithms for repeated ad auctions [41, 10, 31, 30], as well as designing repeated selling mechanisms for ad impression allocation; see [12, 27] and references therein. However, as discussed in the introduction, autobidding platforms conduct bidding on behalf of advertisers while keeping bidding and selling mechanism details undisclosed. In this study, we treat the bidding procedure and selling mechanism as a black box and directly model bidding and auction outcomes using conversion functions; see Section 2. To the best of our knowledge, the most pertinent work to this paper is [21], which explores a similar ad procurement problem by optimizing levers within a bandit environment. However, this work solely focuses on the stochastic world and optimizing a single lever (i.e., a 1-dimensional decision space). In contrast, our paper develops a unified algorithm capable of making high-dimensional lever decisions in many worlds.
Online convex optimization. In Section 2, we cast the advertiser's online learning problem of interest to a high-dimensional bandit online convex optimization problem with uncertain long-term constraints, and develop an algorithm that yields good performance guarantees under different procedures from which the objective and constraint functions are generated. There has been a rich line of works that study bandit convex optimization with no long-term constraints in stochastic and adversarial worlds [25, 32, 42, 17, 44], as well as works that study (full-information feedback) online convex optimization with long-term constraints [36, 33, 43, 35]. Further, works that study both bandit feedback and long-term constraints either consider single-dimensional (such as multi-arm bandits with constraints) [40, 6, 4, 24], or consider static regret (i.e., benchmarking performance to that of a single optimal action) [13, 16, 14, 37]. This paper distinguishes itself from these two streams of works by considering high-dimensional decisions as well as dynamic regret, i.e., comparing to the best sequence of actions instead of a single action; see Eq. (1). Finally, all aforementioned works only study algorithms in stochastic or adversarial setups, whereas in this work we go beyond these two worlds and address more complex learning environment such as periodic, corrupted, ergodic, and finite switching. To the best of our knowledge, the only related work that develops a universal algorithm under "many world" setups is [10]. However, [10] considers a full-information scenario where the online decisions in period can be made after observing realized objective and constraint functions during that period. In this work, we present an efficient algorithm to handle bandit feedback.
Notation. For any vector  $\mathbf{z} \in \mathbb{R}^d$ , let  $\| \mathbf{z} \|$  be its Euclidean norm. Denote  $\mathbb{B} = \{ \mathbf{z} \in \mathbb{R}^d : \| \mathbf{z} \| \leq 1 \}$  as the  $d$ -dimensional unit ball centered at  $\mathbf{0}$ , and let  $\mathbb{S} = \{ \mathbf{z} \in \mathbb{R}^d : \| \mathbf{z} \| = 1 \}$  be the unit sphere. For any set  $S$ , let  $U(S)$  be the uniform distribution over  $S$ . Let  $e$  denote the vector whose components
are all 1's, and  $e_j$  be the unit vector whose  $j$ 'th position is 1. We use  $\mathcal{O}$  notation to represent the asymptotic order of a term when the period  $T \to \infty$  and ignore the  $\log T$  terms.
2 Autobidding with bandit feedback in many worlds (input models)
2.1 Autobidding as a bandit online optimization with long-term uncertain constraints
Consider an advertiser repeatedly interacting with an ad platform over  $T \in \mathbb{N}$  periods, where each period can be interpreted as a single ad campaign that is run on the platform. During each period  $t \in [T]$ , the advertiser makes  $d \geq 1$  lever decisions denoted as  $\boldsymbol{x}_t \in \mathcal{X} \subseteq \mathbb{R}^d$ , e.g., setting the per-campaign budget, campaign duration, per-campaign target return-on-investment, and max spend per conversion; see Figure 1 for example lever decisions in practice. Here,  $\mathcal{X}$  is some compact and convex decision set whose diameter is  $D = \sup_{\{\boldsymbol{x}, \boldsymbol{x}'\} \subseteq \mathcal{X}} \| \boldsymbol{x} - \boldsymbol{x}' \|$ . For simplicity, assume  $\mathbf{0} \in \mathcal{X}$  so  $\| \boldsymbol{x} \| \leq D$  for any  $\boldsymbol{x} \in \mathcal{X}$ . After the lever decisions  $\boldsymbol{x}_t$  are made the campaign is fully executed via autobidding, the advertiser observes bandit feedback for her campaign outcomes: she only observes her realized conversion  $f_t(\boldsymbol{x}_t) \in \mathbb{R}_{\geq 0}$  (e.g., number of clicks on her ads), as well as some constraint balance  $g_t(\boldsymbol{x}_t) \in \mathbb{R}^K$ . The conversion and constraint functions  $(f_t, g_t)$  are sampled from some (possibly infinite) support  $S$  according to distribution  $\mathcal{P}_t \in \Delta(S)$  (we will discuss how  $\mathcal{P}_t$ 's are generated by nature in Section 2.2). Using the notation  $\mathcal{P}_{1:T} = (\mathcal{P}_t)_{t \in [T]}$  the advertiser's hindsight optimization problem is
$$
\operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) = \max  _ {\boldsymbol {x} \in \mathcal {X} ^ {T}} \sum_ {t \in [ T ]} \mathbb {E} _ {\mathcal {P} _ {t}} \left[ f _ {t} \left(\boldsymbol {x} _ {t}\right) \right] \text {s . t .} \sum_ {t \in [ T ]} \mathbb {E} _ {\mathcal {P} _ {t}} \left[ g _ {k, t} \left(\boldsymbol {x} _ {t}\right) \right] \geq 0 \quad k = 1 \dots K. \tag {1}
$$
Here, we use a constraint function  $g_{k,t} : \mathcal{X} \to \mathbb{R}$  to characterize general performance metrics of the advertiser for her ad campaigns. In the following, we present several examples for constraint functions that are widely used in practice or studied in literature. For illustrative purposes, we assume  $x_{t,1} = e_1^\top x_t$  (i.e., the first lever) represents the per-campaign budget for the  $t$ 'th campaign.
- Long-term budget constraint. The advertiser has a total budget  $BT > 0$ . Then by letting constraint function  $g_{k,t}(\boldsymbol{x}) = B - x_{t,1}$ , we have  $\sum_{t \in [T]} g_{k,t}(\boldsymbol{x}_t) \geq 0 \Rightarrow \sum_{t \in [T]} x_{t,1} \leq BT$ , which means total campaign spent over  $T$  periods (assuming each campaign fully depletes the campaign budget) must be less than the advertiser's total budget  $BT$ .
- Long-term return-on-investment constraint. The advertiser intends to safeguard a long-term return-on-investment  $\gamma > 0$ , i.e., she attains a long-term average of at least  $\gamma$  conversions per dollar spent. Then by considering  $g_{k,t}(\pmb{x}) = f_t(\pmb{x}) - \gamma x_{t,1}$ , we have  $\sum_{t \in [T]} g_{k,t}(\pmb{x}_t) \geq 0 \Rightarrow \sum_{t \in [T]} f_t(\pmb{x}_t) \geq \gamma \sum_{t \in [T]} x_{t,1}$ , which means total conversion over  $T$  periods is at least  $\gamma$  times total spend.
Finally, we make the following mild assumptions on conversion and constraint functions.
Assumption 2.1 (Mild assumptions on conversion and constraint functions). For any  $(f, g) \in S$ ,  $f, g_1 \ldots g_K$  are all bounded concave functions, i.e., we assume  $\sup_{\boldsymbol{x} \in \mathcal{X}} \| \boldsymbol{g}(\boldsymbol{x}) \|_{\infty} \leq \bar{G}$  and  $\sup_{\boldsymbol{x} \in \mathcal{X}} |f(\boldsymbol{x})| \leq \bar{F}$  for some  $\bar{G}, \bar{F} < \infty$ . Moreover,  $f$  and  $\boldsymbol{g}$  are  $L$ -Lipschitz, i.e., for any  $\{\boldsymbol{x}, \boldsymbol{x}'\} \subseteq \mathcal{X}$ , we have  $|f(\boldsymbol{x}) - f(\boldsymbol{x}')| \leq L \| \boldsymbol{x} - \boldsymbol{x}' \|$  and  $\| \boldsymbol{g}(\boldsymbol{x}) - \boldsymbol{g}(\boldsymbol{x}') \| \leq L \| \boldsymbol{x} - \boldsymbol{x}' \|$ . Furthermore, there exists  $(f, g) \in S$  such that  $\min_{\boldsymbol{x} \in \mathcal{X}} \min_{k \in [K]} g_k(\boldsymbol{x}) < 0$ .
2.2 Five input models characterizing many autobidding worlds
In this subsection, we describe structural properties of the input distribution sequence  $\mathcal{P}_{1:T}$ , and shed light on how we utilize various properties to model a wide spectrum of autobidding environments (called worlds) such as time-varying user preferences, seasonality, shifts in market trends, etc., that may potentially lead to different procurement outcomes for the same ad campaign lever decisions.
Stochastic: There exists some probability distribution  $\mathcal{P} \in \Delta(S)$  such that  $\mathcal{P}_1 = \ldots = \mathcal{P}_T = \mathcal{P}$ . This stochastic world represents a stationary autobidding environment where the underlying latent factors influencing user behaviors (and correspondingly conversion results) remain constant over time; see [31, 9, 21].
<sup>1</sup>These assumptions are justified in many related works in autobidding; see [21, 14] and references therein. For example, [21] shows that the conversion function is concave and piecewise-linear when autobidding platforms procure ad impressions on behalf of advertisers in standard second price or VCG auctions.
$\delta$ -corrupted. There exists  $\mathcal{P} \in \Delta(S)$  as well as  $\delta \in \mathbb{N}$  periods  $\mathcal{T} = \{\tau_1 \dots \tau_\delta\} \subset [T]$  such that  $\mathcal{P}_t = \mathcal{P}$  for all  $t \notin \mathcal{T}$ . This  $\delta$ -corrupted input sequence represents occasional anomalies in the autobidding environment that may be caused by systematic malfunctions in the autobidding platform, or deliberate attempts by malicious competitors to exploit the system for their own benefit; e.g., some competitors may engage in click fraud to inflate the number of clicks on our ads to exhaust our budget or generate false data to manipulate autobidding algorithms (for example, see [29]).
Adversarial.  $\mathcal{P}_{1:T}$  is adversarially chosen by nature before the process starts, and the distributions over time can possibly be non-identical and/or dependent. This adversarial world can be viewed as a hypothetical extreme case for the  $\delta$ -corrupted world where each period the procurement outcomes can potentially be corrupted by an adversary. Adversarial input sequences have been widely studied in the literature to assess algorithmic performances in worst-case scenarios; see [3, 30].
Periodic. There exists period length  $q \in \mathbb{N}$  such that  $T = cq$  for some integer  $c \geq 2$  with  $\mathcal{P}_{1:T}$  satisfying  $\mathcal{P}_{1:q} = \mathcal{P}_{q+1:2q} = \dots = \mathcal{P}_{(c-1)q+1:T}$ . This periodic world captures regular cyclic patterns or fluctuations in the user behavior over specific time intervals; e.g., seasonality, day-of-week patterns, and time of a day.
Ergodic.  $\mathcal{P}_{1:T}$  is an ergodic process (e.g., an irreducible and aperiodic Markov chain or stationary autoregressive processes), where there exists some  $\kappa > 0$  and a stationary distribution such that the distance between  $\kappa$ -step transition probabilities and this stationary distribution decreases exponentially fast in  $\kappa$ . Mathematically, given the input distribution sequence  $\mathcal{P}_{1:T}$ , denote  $\mathcal{P}_{(t + \kappa)[t - 1]}$  as the conditional distribution of  $(f_{t + \kappa}, g_{t + \kappa})$  conditioned on the realizations  $\{(f_{\tau}, g_{\tau})\}_{\tau \in [t]}$ . Then, for the ergodic input model there exists a stationary distribution  $\widetilde{\mathcal{P}} \in \Delta(S)$  and absolute constant  $R > 0$  such that
$$
\sup  _ {\left\{\left(f _ {t}, \boldsymbol {g} _ {t}\right) \right\} _ {t \in [ T ]} \in \mathcal {S} ^ {T}} \sup  _ {t \in [ T - \kappa ]} \| \mathcal {P} _ {(t + \kappa) | [ t - 1 ]} - \widetilde {\mathcal {P}} \| _ {T V} \leq \delta := R \exp (- \kappa); \tag {2}
$$
We remark that exponentially decaying ergodic models have been studied widely in the context of online optimization; see e.g. [10, Section 5.2] for further discussions. To make the problem of interest tractable, we assume  $\kappa >\log (T)$ . Intuitively, an ergodic input sequence signifies that the procurement outcomes in close time proximity are correlated, which is commonly observed in real-world autobidding systems, as they often involve iterative processes that enable procurement algorithms (operated on behalf of advertisers) to converge to a stable state; see details in [9, 28].
2.3 Minimizing regret subject to long-term constraints with a universal algorithm
In this work, we take the perspective of an advertiser making repeated lever decisions as described in Section 2.1. We focus on designing a single online algorithm that determines a lever decision  $\boldsymbol{x}_t \in \mathcal{X}$  in each period  $t$  with the goal to minimize regret  $\mathcal{R}_T$  (defined as follows) under any input sequence  $\mathcal{P}_{1:T}$  while satisfying long-term constraints
$$
\mathcal {R} _ {T} = \operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) - \sum_ {t \in [ T ]} \mathbb {E} \left[ f _ {t} \left(\boldsymbol {x} _ {t}\right) \right] \text {a n d} \sum_ {t \in [ T ]} \mathbb {E} \left[ \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}\right) \right] \geq \mathbf {0}. \tag {3}
$$
Here, the expectation is taken w.r.t. randomness from the input sequence as well as any randomness in our algorithm. We highlight that our desired policy should be agnostic to the input model as well as the input sequence  $\mathcal{P}_{1:T}$ . We remark that our work distinguishes itself from related work that study solely stochastic and/or adversarial input models, as we aim to design a universal algorithm that achieves salient performance over any input model described in Section 2.2.
3 A universal constrained BOCO framework for many worlds
In this section, we focus on designing an algorithm that achieves low regret (per Eq. (3)) while maintaining long-term constraint satisfaction in any world described in Section 2.2, without having to know which world we are in. We first highlight three main challenges for our problem of interest.
Dynamic benchmark lever sequence and unknown input sequence  $\mathcal{P}_{1:T}$ . Recall the hindsight optimal problem in Eq. (1), with which we are comparing our algorithm's performance to the optimal sequence of lever decisions over time given any input sequence  $\mathcal{P}_{1:T}$ , instead of comparing to a single optimal lever decision as in many related works; e.g., see [15] and references therein. This dynamic optimal sequence presents a very strong benchmark and makes learning very difficult as we (ideally) need to account for variations in the underlying non-stationary ground truth input sequence  $\mathcal{P}_{1:T}$ . Nevertheless, we do not know the input sequence  $\mathcal{P}_{1:T}$  in advance, nor do we know in which world the sequence belongs (Section 2.2).