Sustainable Online Reinforcement Learning for Auto-bidding
Authors: Zhiyu Mou
Yusen Huo
Rongquan Bai
Mingzhou Xie
Chuan Yu
Jian Xu
Bo Zheng
Venue: NeurIPS 2022 Accept (2022)
Keywords: online reinforcement learning
offline RL
auto-bidding

Abstract: Recently, auto-bidding technique has become an essential tool to increase the revenue of advertisers. Facing the complex and ever-changing bidding environments in the real-world advertising system (RAS), state-of-the-art auto-bidding policies usually leverage reinforcement learning (RL) algorithms to generate real-time bids on behalf of the advertisers. Due to safety concerns, it was believed that the RL training process can only be carried out in an offline virtual advertising system (VAS) that is built based on the historical data generated in the RAS. In this paper, we argue that there exists significant gaps between the VAS and RAS, making the RL training process suffer from the problem of inconsistency between online and offline (IBOO). Firstly, we formally define the IBOO and systematically analyze its causes and influences. Then, to avoid the IBOO, we propose a sustainable online RL (SORL) framework that trains the auto-bidding policy by directly interacting with the RAS, instead of learning in the VAS. Specifically, based on our proof of the Lipschitz smooth property of the Q function, we design a safe and efficient online exploration (SER) policy for continuously collecting data from the RAS. Meanwhile, we derive the theoretical lower bound on the safety degree of the SER policy. We also develop a variance-suppressed conservative Q-learning (V-CQL) method to effectively and stably learn the auto-bidding policy with the collected data. Finally, extensive simulated and real-world experiments validate the superiority of our approach over the state-of-the-art auto-bidding algorithm.

Sustainable Online Reinforcement Learning for Auto-bidding
Zhiyu Mou
Yusen Huo', Rongquan Bai', Mingzhou Xie', Chuan Yu',
Jian Xu Bo Zheng't
Alibaba Group, Beijing, China
Department Automation, Tsinghua University, Beijing, China
mouzy208mails. wttpiiiisssssinnuuaaeaaea tsinghua. edu. cn
boooyusen.huouueee rongquan.. br, mingzhou xmz, yuchuan. xiyu.xj, bozheng aalieaaa-bbbbaieeeeeeeaa
Abstract
Recently, auto-bidding technique has become essential tool increase the revenue of advertisers. Facing the complex and ever-changing bidding environ- ments in the real-world advertising system (RAS), state-of-the-art auto-bidding policies usually leverage reinforcement learning (RL) algorithms to generate real- time bids behalf of the advertisers. Due to safety concerns, it was believed that the RL training process can only be carried out in an offline virtual advertising system (VAS) that is built based on the historical data generated in the RAS. In this paper, we argue that there exists significant gaps between the VAS and RAS, making the RL training process suffer from the problem of inconsistency between online and offline (IBOO). Firstly, we formally define the IBOO and systematically analyze its causes and influences. Then, avoid the IBOO, we propose a sustain- able online RL (SORL) framework that trains the auto auto-bidding policy by directly interacting with the RAS, instead of learning in the VAS. Specifically, based on our proof of the Lipschitz smooth property of the Q function, we design safe and efficient online exploration (SER) policy for continuously collecting data from the RAS. Meanwhile, we derive the theoretical lower bound the safety degree of the SER policy. We also develop variance-suppressed conservative Q-learning (V-CQL) method to effectively and stably learn the auto-bidding policy with the collected data. Finally, extensive simulated and real-world experiments validate the superiority of our approach over the state-of-the-ad auto-bidding algorithm.
Introduction
In the era of Internet, online advertising business has become one of the main profit models for many companies. such as Google [6] and Alibaba [3], which, the same time, benefits millions of advertisers who are willing to bid for impression opportunities. Contemporary online advertising systems, auctioneers, usually have large amount candidate advertisers contesting for numerous impression opportunities at every moment. Making auction decisions based on accurate evaluation of each impression opportunity for all advertisers within several milli-seconds is computationally infeasible. Therefore, a real- world advertising system (RAS) adopts a cascade architecture [7, 10]. In this architecture, the auction of each impression opportunity completed through several stages. Without loss of generality, we here simply view the RAS as system with two stages: stage and stage 2. The auction process each impression opportunity as follows: stage rough
Work was done during an internship Alibaba Group. Cooresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

evaluations are conducted for all the candidate advertisers (~ 10%), and group of the most promising advertisers (~ 102) are fed to the stage 2; and in stage 2, accurate valuations and auction are carried out to determine the winning advertiser of the impression opportunity. The winner will gain the value of the impression opportunity and pay the market price. See Appendix A.1.1 for detailed explanations on the RAS structures. Faced with huge amount of impression opportunities at every moment, advertisers cannot bid in the granularity of individual opportunities. Recently, many auto-bidding policies have emerged to realize automatic real-time biddings for each impression opportunity on behalf of advertisers [1. 2, 4, 5], which significantly increase their revenues. State-of-the-ae auto- bidding policies are usually learned with reinforcement learning (RL) algorithms [5]. It was believed that the auto- badto-iddiig policy being trained cannot directly access to the RAS during the RL training
process due to safety concerns [1]. A common solution in most existing works [1. 2, 4, 5] is to train the buto-bidding policy in a virtual advertising system (VAS) - an offiie simulated environment that is built based on the advertisers' historical data generated in the RAS. See Appendix 1.1.2 for the details of the VAS structures.
IBOO Challenges. However, we argue that there exists gaps between the RAS and the VAS, which makes this common solution suffer from the problem of inconsistencies between online and offline (IBOO). Here, online refers to the RAS, while offline refers to the VAS, and inconsistencies refer to the gaps. Formally, we can define the IBOO as follows.
Definition 1 (Inconsistencies Between Online and Offline, IBOO). The IBOO refers to the gaps between the RAS and the VAS that prevent the VAS from accurately simulating the RAS.
Specifically, there two dominated gaps between the RAS and the VAS. One is that the VAS cannot accurately simulate the cascade auction architecture of the RAS. For example, due to the constraint on computing power, the VAS built only based on the historical data generated in stage 2 of the RAS", which makes the VAS lack the mechanism of stage 1 in the RAS. In addition, the VAS does not incorporate the exact influences of other advertisers RAS does, which makes up the second dominated gap. For example, in the RAS, the market prices are determined by the bids of all advertisers which can change during the training process. However, the VAS always provides constant market prices for the auto-bidding policy being trained. See figures in Appendix A.3 for illustrations. Essentially, the IBOO makes the VAS provide the auto-bidding policy with false rewards and state transitions during RL training process. As presented in Table. 1, the auto-bidding policy with better performance in the VAS (i.e., higher R/R*) may yield poorer performance in the RAS (i.e., lower A/B test value). Hence, the IBOO can seriously degrade the performance of the auto- titto-biddin policy in the RAS. One way to address the IBOO challenge is to improve the VAS and reduce the gaps as much as possible. However, due to the complex and ever-changing nature of the RAS, the reduction in IBOO is usually very limited, resulting in little improvement in the auto-bidding policy performance. Hence, new schemes need to be devised so as to avoid the IBOO and improve the performance of buto-bidding policies. Besides, is worth noting that the IBOO resembles the sim2real problem studied in other realms, such computer visions [8] and robotics [9]. Nonetheless, to the best of our knowledge, this is the first work that formally put forward the concept of IBOO in the field of auto-bidding and systematically analyzes and resolves it.
Table I: Influence of IBOO. R/ R* and A/B test values are the performance evaluations of the auto-bidding policies in the VAS and RAS, respectively. Both are the higher the better. We rank the ten policies accordingly. means that the performance in the RAS is higher than that in the VAS, while I means the opposite. means that the evaluations in the RAS and VAS are the same.
Policy
R/R+ R+ (rank)
A/B Tests (rank)
Policy
R/ R+ (rank)
A/B Tests (rank)
0.9118 (1)
-2.50% (9) 4
0.8563 (6)
+4.30% (2)
2
0.8731 (2)
+3.10% (6) L
7
0.8535 (7)
+4.40% (1)
3
0.8656 (3)
-3.20% (10) +
8
0.8434 (8)
-1.50% (8)
4
0,8656 (4)
+1.20% (7)
0.8428 (9)
+3.60% (5)
5
0.8594 (5)
+4.30% (2) f
10
0.8111 (10)
+3.80% (4)
In this paper, we propose a novel sustainable online reinforcement learning (SORL) framework to address the IBOO challenge. For the first time, the SORL abandons the way of learning with the VAS and trains the auto-bidding policy by directly interacting with the RAS. Notably, the SORL can obtain true rewards and state transitions from the RAS and thereby does not suffer from the IBOO problem.
Detailed reasons shown in Appendix A.1.2.

---

The SORL contains two main algorithms. The first one is safe and efficient online exploration policy for collecting data from the RAS, named as the SER policy. Specifically, to guarantee the safety of explorations, we design a safety zone to sample actions from based on the Lipschitz smooth property of the Q function we theoretically proved. We also derive the lower bound on the safety degree. To increase the efficiency of explorations, we develop sampling distribution that can make the collected data give more feedbacks to the auto-bidding policy being trained. The second main algorithm is an effective and stable offline RL method to train the auto-bidding policy based on the collected data, named as the variance- suppeesse conservative Q-learning (V-CQL). Specifically, motivated by the observation that the optimal* Q function is in the quadratic form, we design a regularization term in the V-CQL to optimize the shape of the Q function. The V-CQL can train the auto- bitto-idding policy with hence stable, in performance under different
high average hence effective, and low variance random seeds. The whole SORI, works in an iterative manner, alternating between collecting data and training the auto-bidding policy. Extensive simulated and real-world experiments validate the effectiveness of our approach.
2 Related Work
VAS-based RL Auto-bidding Methods. Impressed by powerful contextual learning and sequential decision-making capabilities of RL, modern auto-bidding policies, such as DRLB [1], RLB [4], MSBCB [2] and USCB (state-of-the-art) [5], are usually Icarned by RL algorithms in a manually built VAS. However, as stated before, they all suffer from the IBOO problem. The SORL avoids using the VAS and thereby completely address the IBOO challenge.
Offline RL. Offline RL (also known as batch RL) [11, 13, 26, 27,30, 30, 31, 38, 39, 40] aims to learn better policies based on a fixed offline dataset collected by some behavior policies. The main challenge offline RL addressed is the extrapolation error caused by missing data [27]. Specifically, offline RL algorithms usually address this challenge in three ways, including policy constraint methods such as BCQ [11), BEAR [30], and conservative regularization methods such as CQL [13], BRAC [40], as well as modifications of imitation learning [37] such as ABM [31], CRR [38], BAIL[39]. However, there is another important challenge that offline RL cannot solve: the fixed offline dataset cannot be guaranteed to contain sufficient state-action pairs from high-reward regions [27]. This challenge exists in many practical applications, including the auto-bidding problem, and can inherently prevent offline RL algorithms from learning excellent policies. Hence, a great advantage of the SORL is its ability to continuously collect data from OOD high-reward regions that can give new feedbacks to the bidd-n policy being trained.
Off-Policy Evaluation (OPE). OPE is an algorithm to evaluate the performance of the policy with offline data collected by other policies [12]. The state-of-the-art OPE algorithm in auto-bidding is to calculate the R/ R+ of the evaluated policy in the VAS (see Appendix B for details). However, as shown in Table. 1, the OPE in auto-bidding is not accurate. This indicates that we cannot rely on the OPE to select auto-bidding policies with good performance for directing further online explorations. Notably, the proposed offline training algorithm in the SORL, V-CQL, outperforms existing offline RL algorithms in training stabilities and helps to reduce the OPE process during iterations in SORL.
Online RL. Safety is of vital importance in the online exploration of many real-world industrial systems, including the RAS. Many safe online RL algorithms have been proposed to guarantee the safety of explorations [17, 18, 19, 20, 21, 23, 34, 35). However, many of them are either developed based on the constraints that are not suitable for the auto-bidding problem or designed for systems with specific assumptions (see Appendix for details). Besides, many existing works [33, 34, 35, 21] assume that there exists a specific set of safe state-action pairs that can be explored. Some work [33] even requires to know the safe states in advance. However, in the auto-bidding problem, no specific actions at any state cannot be explored as long as the expected accumulative reward maintains at high level. This requires the exploration policy to maintain high performance throughout the iterations, which is more challenging. Notably, the SER policy can meet this requirement with theoretical guarantees. Recently, with the development of the offline RL field, many algorithms for efficient online explorations on the premise of having an offline dataset [14, 23] have emerged. However, they often only focus on the efficiency of data collections but ignore the safety of it. Notably, the SER policy in the SORL strikes good balance between the efficiency and safety.
'Here "optimal" refers the optimal Q function in the simulated experiments. See Section 4.2 for details.
3

---

Supplementary related works are described in Appendix B.
3 Problem Settings
We consider the auto-bidding problem from the perspective of single advertiser, which can be viewed as an episodic task with T E N+ time steps. Specifically, between time step t and t 1, there are N; E N+ impression opportunities, each of which has a positive value U/et S UM and is sold to the winning advertiser at market price Pj,t S PM, where UM O and pM > 0 are the upper bounds for values and market prices, respectively. j E {1.2, N.). E (1, ..., T). Denote Pis S PM and u/e, I Uar the market price and rough value of impression opportunity j in stage 1, and p²,, S PM and 5,, S UM as the market price and accurate value of it in stage 2. Note that they are all positive values. Let the total budget of the advertiser be B > 0. The auto-bidding problem can be modeled as a Constraint Markov Decision Process (CMDP) [15] < S, A, R, P.Y, >. The state 8, - [b,, - B - b.] E S is composed of three elements including the budget left (1) b,, the I, time left (2) T and the consumed budget (3) B 6, The action a, E A - [Amin Amax] is the "a-bid" at time step f. where Amax > Amin > 0 are the upper and lower bounds, respectively. The real bidding price for impression opportunity j under action ae a1U),.t See [5] for detailed explanations. In addition, the reward function r (t( at) and the constraint function ca az) are the total value of impression opportunities won between time step and t++ and the corresponding costs, respectively, and IP denotes the state transition rule. Note that R, C and IP are all directly determined by the RAS. Moreover, y E [0, 1] is the discounted factor. We denote the auto-bidding policy as u S + A, and let TT be the policy space. The goal of the auto-bidding problem is to maximize the total values of the impression opportunities carned by the advertiser under the budget constraint, which can be expressed as
Σ Exm((sras subject to ass 0r) B.
max V(n) A E EsIti P Pare EE
(1)
One can leverage standard RL algorithms to train state-of-the-ar auto-bidding policy [5] with the VAS, where the constraint (1) is met by terminating the training episode early once the budget runs out. However, as stated before, this way of learning suffers from the challenge of IBOO. Hence, we propose the SORL framework to avoid the IBOO in the following section.
4 Our Approach
The SORL framework consists of two algorithms, including the SER policy for online explorations and data collections, as well as the V-CQL method for offline training of the auto-bidding policy based on the collected data. The SORL works in iterative manner, alternating between online data collections and offline training.
4.1 Online Exploration: SER Policy
There are two requirements on the online exploration policy. As the exploration policy is directly deployed in the RAS, the primary requirement is safety. Unlike the safety in other realms such as robotics, it is not appropriate to construct an immediate- evaluuate safety function to assess the safety degree of each state-action pair merely based on their values in auto-bidding. Actually, any action in any state would be safe as long as the summation of all rewards of the whole episode maintains at high level. See Appendix F.1.3 for detailed explanations. Let μt, be the auto-biddin originally deployed in the RAS. Note that p. is safe. Denote the online exploration policy as ne. We can formally define the safety requirement as: the performance of ne cannot be lower than that of
policy
"I fact, more elements be added the state, and we claim that the under mild additional assumptions. the propositions and theorems in this paper still hold.
'There is always an auto-biddee bidding policy μ. deployed in the RAS. For example, the state-of of of-the-ar art method USCB [5] will do.

---

R.n o
88
function Q"(S,, 4e) St
Rew wih
50
R. -hh
wi1 1,0-
0.012
40
25
Le
bg
30
Safety Zone
0.010
25
+
0.038
20
Pr.
Actions a,
(a) The Q function of any auto budo-nidding policy is Lo Lipschcchit smooth. The safety zone is de- signed as the range between paa (*t) & and (x.) + &, where <
(b) Sampling distribution of the SER policy, ne. under different syper-parameters a and X, as well as Te,V with different hyper-parameter a.
"
Figure I: Design of the safety zone and sampling distributions in the SER policy.
He by a threshold e, > O, Le., V(u,) - (mTe) S cs. Besides, ne needs to be efficient, i.c., it should collect data that give new feedbacks the auto-bidding policy being trained, which constitutes the second requirement. We next propose the SER policy for online explorations that can satisfy both requirements.
4.1.1 Theory: Lipschitz Q Function
Our basic idea of ensuring the safety of exploration to design a safety zone for taking actions around the outputs of Hg. The theoretical foundation of this idea is our proof of the Lipschitz smooth property of the Q function in the auto-bidding problem. We next report the theorem of this Lipschitz smooth property, well as corresponding propositions and assumptions. Let μ be arbitrary auto-bidding policy. According to the Bellman equation, we have:
Q" az) re(st, ax) + PP- 1Q" ((
(2)
Based on the mechanism of RAS, we formulate the reward function R, the constraint function C' and the transition rule P as follows, and the proof can be found in Appendix D.I.
Proposition 1 (Analytical expressions of R, C and P). Based on the characteristic of the two-stage cascaded auction in the RAS, can formulate the reward function R re(si, az) Σ, Marot 2 p),,, atu3,, 2 FU,, the constraint function C' as (e 0t) = E; Ha,u), 2 Pj auy 2 2 Pj.lPp. and the state transition rule P as St+1 = St + [As (1), (2),,, (3)) where (1, (3) E; 1arr",, 2 P/... atv3, 2 p3, and (s; = -1
We make the following assumption on the impression opportunities, whose rationalities can be found in Appendix C.I.
Assumption (Bounded Impression Distributions). Between time step t and t + 1, assume the numbers of winning impressions with action ag in the first stage n4,1 and the second stage n4,2 can both be bounded by linear functions, i.e., 04,1 S kia,, n4,2 I k2aa, where k, k2 > 0 are constants.
Based on this assumption, we claim that the reward function re at) is Lipschitz smooth with respect actions a₂ any given state st. See Appendix E.1 for the proof.
Theorem (Lipschitz Smooth of re(st, a.)). Under Assumption I, the reward function (5 az) is Ly-Lipschitz smooth with respect to actions at at any given state st, where L,- = (k1 k2)
We make the following mild assumptions, whose rationalities can be found in Appendix C.2.
Assumption 2 (Bounded Partial Derivations of Q" (St, a.)). We assume that the partial derivation of < k4.
Q" ax) with respect to (1) and se(3) is bounded, i.e., aQ" (ar.as) where K3, ka > 0 are constants.
< K3 and (s..))
(s
(s.
5

---

Equipped with Theorem 1, we can prove that the Q function Q" 0z) is also Lipschitz smooth. See Appendix E.2 for proof.
Theorem 2 (Lipschitz Smooth of (sr, at)). Under Assumption I and 2, the Q function Q" (st, 0z) is an Le-Lipschitz smooth function with respect to the actions ot any given state 86 where Lo = [uas + 7(k + ka)pw](ki + k2).
This means that the decrease rate of Q", the subsequent accumulated rewards, due to action offset at any time step is bounded by LQ. which gives us a way to design the safety zone. Specifically, as shown in Fig. (a), the safety zone at state can be design as the neighborhood of action Jes (s²), i... [st(ss) E. ps (st) + c). where & > is the range. In this way, the online exploration policy Te can be designed as: sampling within the safety zone (1,(8)) (a()) E] in certain AT 2 1 consecutive time steps, and sticking to μs in the rest of T
AT time steps, i.e..
(s = sampling from [Na(*e) - (.)) + E (s)),
41 S 1 12;
Vt.
(3)
otherwise.
where 0 S 4 < l2 - 1, 1, AT = =t 1121-111 + 1, and &,, l2 E N+ In the following theorem, we give the upper bound of |V (T²) - V(Ha)l-
Theorem 3 (Upper Bound of (VTTT) - ((s)))) The expected accumulated reward V (Te) satisfies
V(me) v(m.)) s E uM + 7 (Kis + ka)PM (Ki + k2) AT.
See Appendix E.3 for the detailed proofs. Hence, to meet the safety requirement, we design the range of the safety zone as & S T.77"17'
4.1.2 Sampling Method
With the safety guarantee, we can further design the sampling method in ne to increase the efficiency of exploration.
Vanilla Sampling Method. A vanilla sampling method in the safety zone can directly be sampling from a (truncated) Gaussian distribution Re.N = N (s)( o²) with mean (s((s) and variance o². However, Æe_Nr resembles the policy 14, to a large extent and cannot change with the auto-bidding policy being trained, which makes the explorations conservative and lack of new feedbacks.
SER Sampling Method. To lift up the efficiency of explorations, we shift the distribution Ro,N' towards the actions of the auto-bidding policy being trained, while constraining the deviations within threshold & > 0. This can be formulated as functional optimization problem:
Be (s) Ne.
max
s.t. DxL (mte Re,N) S te:
where the optimization variable मe denotes the shifted distribution, Q is the Q function of the auto-bidding policy being trained. Using Euler equations, we can derive the form of Te from (5) as
नe = #e,N
exp { Holsr.ai)
(at (.)))))))
1 [Qss,aa) }
exp
(6)
C(st)
C(s))
20
safely
efficiency
where C(st)=== = (op(-lo--uefo)1 + ( (Q(s;,;))dd ad), acts as the normalization factor. The complete deductions of #.e given in Appendix F.1.1. We can interpret n. deviation from #e,,v towards the Q function Q(+,, at), where Re.. further guarantees the safety and the Q function ensures the efficiency. Note that and À in (6) are both and deviation degrees from μ, the SER policy, respectively. As shown in Fig. ((b), the smaller the value of X, the larger the deviation degree of #. from μai besides, the bigger the value of a, the greater the randomness degree of म.. Hence, we can control the randomness and deviation degrees from the safe policy μ4 of exploration policy easily by adjusting the value of and X. In addition, we describe the way of practically implementing the sampling of distribution मe in Appendix F.1.2 Then the complete SER policy is
that can control the randomness
(samlling from (st) (.Ho + + with #e. ti tistsay tzj otherwise.
(st)) (st)s
WL.

---

Warm booting
fs
Tel
2,22
e,
SER policy
RAS
RAS
D3
0o
om,1
on.2
D on,T
Iteration
Iteration
lteration
Iterution
V-COL
V-CQL
+ Dag
V-COL
D3 + Dons Dona
V-COL
D. Dan
Auto-
0o
Q1
biddligg
140
U2
policy
Figure Sustainable online reinforcement learning (SORL. framework which learns the auto- auto-bidding policy directly with the RAS in safe and efficient way. The SORL works in iterative manner.
4.2 Offline Training: V-CQL
Due to the safety constraint on the explorations, the SER policy can only collect data within the safety zone and the data outside the safety zone will be missed. Hence, we leverage a strong baseline offline RL algorithm, CQL, to address the extrapolation error and make the offline training effective. Moreover, as the sampling distribution in the SER policy involves the Q function Q the training result will affect the directions of further explorations. However, on the one hand, as stated in Section the OPE in the auto-bidding is not reliable; and on the other hand, the performance variance under different random seeds of existing offline RI. algorithms can be large, as shown in Fig. 5.2 in the experiment. Hence, we further design a novel regularization term in the CQL loss increase the stability of offline training and thereby reduce the OPE process. This forms the V-CQL method. Specifically, we observe that the functions the state-of-the-ard auto-bidding policy are always in nearly quadratic forms. In addition, we conduct simulated experiments where the auto-bidding policy directly trained simulated RAS? with traditional RL algorithms. We find out the optimal Q functions in the simulated experiments are also in nearly quadratic forms. See Appendix F.2.1 for illustrations. Based on these observations, we assume that the optimal Q functions also in nearly quadratic forms. Hence, the key idea of V-CQL is to restrict the Q function to remain nearly quadratic, and the regularization term the V-CQL is designed as:
RAS are
exp(((sx.))
expl (82: ") exp( QQuua Caua(( (ok,
R())= SL DKL
Ea exp (xp(Q(sk))) a))
distributionffomm) of
distribution(form)) Qun..
where Qqua is selected as the Q function of the state-of-the-art auto-bidding policy, (k.(;)) represents the KL.-divergence, and 3 > 0 is a constant controlling the weight of R(u).. Moreover, we can interpret (8) from another aspect, i.e., R(x²) tries to limit the distance between the distribution of the function and the distribution Qqua- This can limit the derivation of the buto-bidding policy from the corresponding state-of-the-arr policy, which can increase the training stability. Complete implementations of the V-CQL. are shown in Appendix F.2.2.
4.3 Iterative Updated Structure
Fig. 2 shows the whole structure of the SORL framework. Specifically, the SORL works in an iterative manner, alternating between collecting data with the SER policy from the RAS and training the auto-bidding policy with the V-CQL method.
Warm Booting, To start with, we use the safe policy μ4 to boot the explorations the RAS. Denote the data collected by M D. {(k, ag, Tk. syy) ht. and the auto-bidding policy trained by D, with the V-CQL as No-
Iteration Process. Denote the SER policy and the auto-bidding policy in the 7-th iteration as Te. and μr., and the data collected in the 7-th iteration as D Don,Ts where 7 E (1,2,... ..... The design
The details of this simulated RAS described in Appendix A.2

---

for the sampling distribution मe,r in iteration T is ãe,r C.(a)) He.N e as)/Ar where (st) acts as the normalization factor and A is the hyper-parameter used in iteration т, and Qr is the Q function of Hy. We note that the exploration policies in each iteration are safe, ie., (V [(u))l S Es, since they all taking actions within the safety zone only with different sampling distributions. We leverage the V-CQL to in each iteration improve the auto-bidding policy. Specifically, at iteration T, we substitute Q in (8) with QT-1, and train a new Q function Q + Qr for the next iteration. The auto-bidding policy are expected to be continuously improved. A summary of the SORL framework is present in Appendix F.3.
5 Experiments
We conduct both simulated and real-world experiments validate the effectiveness of our approach* The following three questions are mainly studied in the experiments: (1) What is the performance of the whole SORL framework? Is the SER policy safe during iterations? Can the V-CQL method continuously improve the auto- -uto-bidding policy and outperform existing offline RL algorithms and the state-of-the-an art auto-bidding policy? (2) Is the safety zone reliable? Is the SER policy still safe when using Q functions of auto-bidding policies with bad performance? (3) Does the V-CQI. really help to reduce the performance variance compared to existing offline RI. algorithms?
Experiment Setup. We conduct the real-world experiments on one of the world's largest E-commerce platforms. TaoBao. See Appendix G.I for details. The simulated experiments are conducted in manually built offline RAS and the corresponding VAS. See Appendix A.2 for details. The safe auto-bidding policy μs used for warm booting and constructing the safety zone is trained by the state-of-thee of-the-artt art auto- auto-bidding policy, USCB[5]. The safety threshold is set as 6; = 5%V(u,)
Performance Index. The objective function V (p)) in (1), i... the total value of impression opportuni- ties won by the advertiser in the episode, acts as the main performance index in our experiments and is referred as BuyCnt in the following. In addition, we utilize three other metrics that are commonly used in the buto-bidding problem to evaluate the performance of our approach. The first metric the total consumed budget (ConBdg) of the advertiser. The second metric the return on investment (ROI) which is defined as the ratio between the total revenue and the ConBdg of the advertiser. The third metric is the cost per acquisition (CPA) which defined the average cost for each successfully converted impression. Note that larger values of BuyCnt, ROI, and ConBdg with a smaller value of CPA indicate better performance of the auto-bidding policy.
Baselines. We compare our approach with the state-of-the -he- auto-bidding policy, USCB [5], that is trained by RL in the VAS. We also compare the V-CQL. method with modern offline RL algorithms, including BCQ [11] and CQL [13]. Recall that many safe online RL algorithms are not suitable for explorations in the auto-bidding problem stated in Section 2, we compare the SER policy with Te,. in our experiments.
5.1 Main Results
To Answer Question (1): We first conduct simulated experiments with the SORL, and the results during iterations are shown in Fig. 5.1. Specifically, from Fig. 3(a), we can see that the decline rate in BuyCnt of both the SER policy and the vanilla exploration policy are smaller than 5%, which verifies the safety of the SER policy. Moreover, the BuyCnt of the SER policy Ne, rises with iterations and is alway higher than that of Re,As, which indicates that the SER policy is more efficient. From Fig. 3(b), we can see that the BuyCnt of the auto-bidding policy prr trained with V-CQL higher than that of BCQ, CQL and USCB, which indicates the effectiveness of the V-CQL. Besides, the BuyCnt rises with the number of iterations and converges to the optimal BuyCnt (i.e., the BuyCnt of the optimal policy) at the 5-th iteration. This validates the superiority of the whole SORL framework. This indicates the effectiveness of the V-CQL method. For real-world experiments, we utilize 10, 000 advertisers to collect data from the RAS with ne and compare the bido-ni policies in 4 iterations on 1, 500 advertisers using A/B tests, and the results are shown in Table. 2. We can see that the performances of budo-nidding policies are getting better with iterations and exceeds the state-of-the-art algorithm, USCB, which validates the superiority of the whole SORL framework.
'The codes of simulated experiments are available http:///ghhuubcccmmmmoooooossssp
Aiddiid
8

---

Varila expentoon pelic Fex
Inital poicy
policy ity
Insiaal infi palccy for
300
250
BeyOnt
BuyCCe
200
150
Polay unieed by BCQ
taaned (USCB)
Pelicy tained
100
Policy usiaed by (peapeggeapoool
Iniial unstrinnd
V-CCE
250
50
palicy
lnnal tonnee pollyy CVR.
0
Tierations
Dersiions
(a) BuyCnt of Ne, and Ne,N-
(b) BuyCnt μ using different algorithms.
Figure The change of BuyCnt of both the SER policy Te,r and auto-bidding policy prr with iterations 7 when applying the SORL framework.
Table 2: The results of SORL framework in the real-world experiments.
A/B Tests with pr-1
A/B Tests with the safe policy Ps
Itcrations
BuyCnt
ROI
CPA
ConBdg
BuyCnt
ROI
CPA
ConBdg
0-th: pio
+3.21 %
+1.28%
-2.01 %
+1.12%
+3.21 %
+1.28% %
-2.01%
+1.12%
1-th: pn
+0.65 %
+1.96%
-1.27% %
-0.62%
+3.41 %
+2.88% %
-0.93%
+2.45%
2-th: μi
+0.47% %
+0.26%
-0.13%
+0.33%
+3.57 %
+1.60%
-0.98%
+2.55%
3-th: pss
+0.95%
+3.20%
-1.01%
+0.06%
+3.75%
+2.48%
-3.91%
-0.15%
5.2
Ablation Study
To Answer Question (2): We fully examine the safety of the
SER policy using the Q function Q of auto-bidding policies with
Uilrnd polvess
different performance levels. Specifically, in the simulated exper- iment, we utilize seven different versions of auto- auto-bidding policies
{Mo)) with functions {Â(K) He=1 that have various per- formance levels to construct seven corresponding SER policies, where (((a)) < V(42)) ... (((z)) We also construct a vanilla exploration policy Te,N' for comparison. The hyper- parameters for all exploration policies
400
1, 0.1. We apply the Te.Ar and seven SER policies to the simulated RAS, and the BuyCnt of explorations policies shown in Fig. 4. We can see that the BuyCnt of the SER policy rises with the performance level of the auto-bidding policy. The worst BuyCnt of the SER policy drops about 3.52% compared with μts which meets the safety requirement. Moreover, we can balance between the safety and efficiency of the SER policy by regulating the byper-parameters a and A. The corresponding results are shown Appendix G.2.2.
Figure 4: BuyCnt of Te,As, We-
For real-world experiments, we apply मe and Ne, to 300 advertisers in the RAS and compare their perfor- mance to H4 in A/B tests, as shown in Table. 3. We can that the variations of the ConBdg and BuyCnt of Te are all with in 5%, and are better than those of Te,.. which indicates the safety of the SER policy.
Table 3: Real-world A/B tests between Te and (4s: well as between Te, and paa
Methods
BuyCnt
ConBdg
vanilla Re.. -3.32%
+1.11% +0.67%
SER policy Te
-1.83% %
leverage V-CQL, CQL, BCQ and USCB to train auto-bidding policies under 100 different random seeds in the simulated experiment, and the results shown in Fig. 5(a). We can see that the performance variance the V-CQL is much smaller than those of other algorithms. At the same time, the average performance of the V-CQL maintains at high level. In addition, we leverage the V-CQL and CQL algorithm to train the auto-bidding policies based on the real-world data under 100 different random seeds. To ensure faimess, also utilize the USCB to train the auto- piito-bidding policies in the VAS under exactly the same random seeds. We carry out the OPE for these three sets of auto-bidding policies, shown in Fig. 5(b). Recall that R/ Rs is the main
To Answer Question (3):

---

300
R/R==1.00
R/R=
1,0
V-CQL: R/R 9468
USCB: max R/R 8216
0.8
200
BayCa
150
0.6600
Me: R/Rx=0.7629
100
V.DOL
COL
BQQ
USCB
V-COL
COL
USCB
(a) BuyCnt the auto-bidding policy trained by different methods in the simulated RAS.
(b) OPF the auto-bidding policy trained by differ- ent methods with real-world data.
Figure 5: The performance of different methods under 100 random seeds.
main metric used in the OPE for auto-bidding (see Appendix B). We can see that the maximum R/R of the V-CQL much larger than that of the USCB, which indicates that the V-CQL is capable training a better policy. Although the maximum R/ R* of the V-CQL and the CQL about in the same level, the variance of R/ Rx of the V-CQL smaller than that of the CQL. For real-world experiments, we apply the auto-bidding policies trained by the V-CQI. and USCB to 1, 600 advertisers in the A/B test for seven days in the RAS, and the average values of the metrics are shown in Table 5. We can see that the V-CQL outperforms USCB method in almost all metrics. Moreover, we present the A/B test results under 10 random seeds in Table. 4. We can see that the V-CQL outperforms the USCB under 9 seeds, and only slightly worse under the other one seed. All these results indicate that the V-CQL can really help to reduce the performance variance and increase the training stability, while keeping the average performance a high level.
Table 4: Real-world A/B tests between the V-CQL and USCB under different random seeds.
Seeds
BuyCnt
Seeds
BuyCut
+1.70%
+1.94 %
2
+1.06 %
7
+1.38 %
3
+3.01 %
+1.19%
+1.89%
+0.31 %
5
-0.53%
10
+0.71 %
Table 5: world A/B Tests between the V-CQL and USCB.
USCB: policy trained with new data
USCB the safe policy μs
Methods
BuyCnt
ROI
CPA
ConBdg
BuyCnt
ROI
CPA
ConBdg
USCB
40,926
3.90
20.71
847,403.
35,627
3.82
21.58
768,832.64
V-CQL
42,236
3.95
20.29
856,913.14
37,090
3.97
20.61
764,467.39
variations
+3.20%
+1.28%
-2.03%
+1.12%
+4.11%
+3.93%
-4.49
-0.57%
Conclusions
In this paper, we study the auto-bidding problem in online advertisings. Firstly, we define the IBOO challenge in the auto-bidding problem and systematically analyze its causes and influences. Then, to avoid the IBOO, we propose the SORL framework that can directly learn the auto-bidding policy with the RAS. Specifically, the SORL framework contains two main algorithms, including safe and efficient online exploration policy, the SER policy, and an effective and stable offline training method, the V-CQL method. The whole SORL framework works in iterative manner, alternating between online explorations and offline training. Both simulated and real-world experiments validate the superiority of the whole SORI. framework over the state-of-the-arr auto-bidding algorithm. Moreover, the ablation study shows that the SER policy can guarantee the safety of explorations even under the guide of the auto-bidding policy with bad performance. The ablation study also verifies the stability of the V-CQL method under different random seeds.
Acknowledgments and Disclosure of Funding
This work is supported by Alibaba Research Intern Program. The authors would like to thank Mingyuan Cheng, Guan Wang, Zongtao Liu, Zhaoqing Peng, Lvyin Niu, Miao Xu, and Tianyu Wang for their valuable feedbacks and insightful discussions.
10

---

References
II] D. Wu, X. Chen, X. Yang, et al., Budget constrained bidding by model-free reinforcement learning in display advertising, in Proceedings of the 27th ACM International Conference Information and Knowledge Management, Oct. 2018.
[2] X. Hao, Z. Peng, Y. Ma, et al., Dynamic knapsack optimization towards efficient multi-channel sequential advertising, in Proceedings of the 37th International Conference on Machine Learning, Nov. 2020.
[3 Alibaba. 2022. alimama, https://www.alimane wtwiimamm.coo/ndeee...e.ee com/index.htm
[4] H. Cai, K. Ren, W. Zhang, et al.. Real-time bidding by reinforcement learning in display advertising, in Proceedings of the 10th ACM International Conference on Web Search and Data Mining, Jan. 2017.
[5] Y. He, X. Chen, D. Wu, et al., A unified solution to constrained bidding in online display advertising, in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Aug. 2021.
[6] Google. 2022. google ads, https://ads.googll.com
[7] Z. Wang. L. Zhao, B. Jiang B, ct al., Cold: Towards the next generation of pre-ranking system, arXiv preprint 2xXiv2007..16122 Aug. 2020.
[8] D. Carl, and A. Zisserman, Sim2real transfer learning for 3d human pose estimation: motion to the rescue, in Advances in Neural Information Processing Systems 32, 2019.
S. Höfer, K. Bekris, A. Handa, al., Sim2Real in robotics and automation: Applications and challenges, IEEE transactions on automation science and engineering, vol. 18, no. 2, pp. 398-400, Apr. 2021.
[10] X. Liu, C. Yu, Z. Zhang, et al., Neural auction: End-to-end learning of auction mechanisms for e-commerce advertising, in Proceedings the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, Aug. 2021.
[11] S. Fujimoto, D. Meger, and D. Precup, Off-policy deep reinforcement learning without exploration. in Proceedings of 36th International Conference on Machine Learning. May. 2019.
[12] C. Voloshin, HM. Le, N. Jiang, and Y. Yue, Empirical study of off-policy policy evaluation for reinforcement learning, in 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), Jun. 2021.
[13] A. Kumar, A. Zhou, G. Tucker, and S. Levine, Conservative q-learning for offline reinforcement learning. in Advances in Neural Information Processing Systems 33, 2020.
[14] A. Nair, A. Gupta, M. Dalal, and S. Levine, Awac: Accelerating online reinforcement learning with offlinc datasets, arXiv preprint #XXv:2006.00355 2020.
[15] E. Altman, Constrained Markov decision processes: stochastic modeling, Routledge, 1999.
[16] TP. Lillicrap, JJ. Hunt, A. Pritzel, et al., Continuous control with deep reinforcement learning, arXiv 2015.
preprint
[17] H. Bharadhwaj. A. Kumar, N. Rhinehart, et al., Conservative safety critics for exploration, arXiv preprint arXiv:2010. 14497, 2020.
[18] DRR. Scobee, SS. Sastry, Maximum likelihood constraint inference for inverse reinforcement learning, arXiv preprint (rXXiv1909.05477 2019.
[19] U. Anwar, S. Malik, A. Aghasi, et al., Inverse Constrained Reinforcement Learning, arXiv preprint
arXiv:2011 1009999, 2020.
[20] B. Thananjeyan, Balakrishna, U. Rosolia, et al., Safety augmented value estimation from demonstrations for sparse cost robotic tasks, IEEE Robotics and Automation Letters, vol. 2020.
(saved); Safe deep
5, no. 2, pp.
[21] F. Berkenkamp, M. Turchetta, Schoellig, et al., Safe model-hased reinforcement learning with stability guarantees, in Advances in neural information processing systems, 2017.
[22] T. Koller, F. Berkenkamp, M. Turchetta, al., Leaming-basee model predictive control for safe exploration, IEEE Conference on Decision and Control (CDC), 2018
[23] S. Lee, Y. Seo, K. Lee, et al., Offine-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble, in Conference on Robot Learning, 2022.
[24] TL. Paine, C. Paduraru. A. Michi, et al., Hyperparameter selection for offline reinforcement learning, arXiv preprint arXiv:2007.0005: 2020.
[25] X. Zhan, H. Xu, Y. Zhang, et al., Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning, arXiv preprint arXiv:2102.11492, 2021.
[26] R. Qin. S. Gao, X. Zhang, et al. NeoRL: A near real-world benchmark for offline reinforcement learning. arXiv preprint arXiv:2102.00714, 2021. 2021.
11

---

[27] S. Levine, A. Kumar, G. Tucker, et al.. Offline reinforcement learning: Tutorial, review, and perspectives open problems, arXiv preprint rrXiv:2005.01633 2020.
[28] P. Jan, M. Katharina, and A. Yasemin, Relative Entropy Policy Search, in Proceedings of the AAAI Conference on Artificial Intelligence, 2010.
[29] X. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, arXiv preprint arXiv:1910.00177, 2019.
[30] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems 32, 2019.
[31] N. Siegel, J. Springenberg. F. Berkenkamp, A. Abdolmaleki, M. Neunert,T. Lampe, R. Had M. Riedmiller, Keep doing what worked: Behavioral modelling priors for offiine reinforcement learning. arXiv preprint arXiv:2002.0 08396, 2020.
[32] A. Kumar, A. Singh, S. Tian, et al., A workflow for offline model-free robotic reinforcement learning, arXiv preprint arXiv:2109.
2021.
[33] M. Turchetta, F. Berkenkamp, and A. Krause, Safe exploration in finite markov decision processes with gaussian processes, in Advances in Neural Information Processing Systems 29, 2016.
[34] G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa, Safe exploration in continuous action spaces, arXiv preprint / 1801,88577 2018.
[35] R. Laroche, P. Trichelair, and Des Combes, Safe policy improvement with baseline bootstrapping, in Proceedings of 36th International Conference on Machine Learning, 2019.
[36] Z. Guan, H. Wu, Q. Cao, et al., Multi-Agent Cooperative Bidding Games for Multi-Objeatie Optimization in e-Commercial Sponsored Search, in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, 2021.
[37] D. Brandfonbrener, W. F. Whitney, R. Ranganath, and J. Bruna, Offline RI. Without Off-Policy Evaluation, in Advances in Neural Information Processing Systems 34, 2020.
[38] Z. Wang, A. Novikov, K. Zolna, J. Merel, J. S., S. Reed, B. Shahriari, N. Siegel, C. Gulcchre, N. Heess, et
al.. Critic regularized regression. Advances in Neural Information Processing Systems 33, 2019.
[39] X. Chen, Z. Zhou, Z. Wang, .C. Wang. Y. Wu, and K. Ross, Bail: Bestaction imitation learning for batch deep reinforcement learning, Advances in Newral Information Processing Systems 33, 2019.
[40] Y. Wu. G. Yucker, Nachum, Behavior Regularized Offline Reinforcement Learning. arXiv preprint arXiv: 1911 2019.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. . You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets?? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/]]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6 and social impact in Appendix H.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We believe there is no negative socictal impacts.
12

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 4.1.1 and Appendix C.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix D and Appendix E.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We include the codes for simulated experiments in Section 5 in the paper. The codes for real-world experiments are proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A] Our work uses data, and the data used in the simulation experiments and real-world experiments is released in supplementarymateriaa
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "jKHZGWOR_kx", "reviewer": "Reviewer_o6ML", "summary": "This paper tries to address the problem in auto-bidding that training can only be carried out offline due to safety concerns. The performance of the learned policy could deteriorate due to the difference between the offline simulation and online real environment. The authors propose a sustainable online reinforcement learning paradigm which uses a safe policy to explore the real enviroment.", "questions": "(1) What do you mean by $p_{j,t}$ and $v_{j,t}$ in proposition 2? Do you mean $p_{j,t}^2$ and $v_{j,t}^2$?\n(2) What do you mean by $\\mu_s$ in Figure 6?\n(3) Does the $\\pi_e$ policy mean the SER policy after 7 iterations?\n(4) Why do you only show results after three iterations in Table 3?", "limitations": "No such issues are identified in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) the problem to be addressed is important and novel.\n(2) the proposed idea is interesting and technical depth is deep.\n(3) experiments are conducted on a real world platform.\n\nWeaknesses (The revised paper and the authors' response addressed these major concerns.):\n(1) There are approximations/relaxations without proper justification or theoretical guarantee.\n(2) There could be flaws in some proofs.\n(3) The experiments fail to compare the proposed method to SOTA methods.\n\nDetailed comments:\n\nThe major problems with this work are:\n\n-There are approximations/relaxations without proper justification or theoretical guarantee. At the end of section 3, the authors said we can relax the constraint \"the safty zone is effective only when the subsequent actions strictly follow the saft policy $\\mu_s$\". But there is neither jutification for the relaxation nor theoretical analysis on how this relaxation could affect the safe property of the online exploration. Another issue lies in section 4.3, where the authors modify the SER policy without justification.\n\n-The proof for theorem 1 seems problematic. The upper bound in Eq.(12) could be violated by such a case: $\\frac{p_{j,t}^2}{v_{j,t}^2} > a_2, \\forall j \\in \\{1, \\dots, N_t\\}$, which means $a_2$ fails to win any impressions. In this case, subtracting the winning impression upperbound for $a_2$ seems to underestimate the bound.\n\n-The authors fail to compare to the SOTA works in the auto-bidding field, e.g., [\"Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search.\" SIGKDD 2021] develops an evolutionary strategy to learn the optimal policy and achieves better performance than RL methods. Furthermore, as the authors argue in Figure 2(b), the bidding involving all advertisers naturally falls into the realm of multi-agent RL. However, the proposed method is still a single agent solution which cannot well handle the market price change issue. In comparison, the above work sets up a multi-agent bidding game for policy learning.\n\n\nMinor issues:\n\n-The presentation could be improved. The figures are not detailedly explained in the main text. Some notations are confusing. See the first question in the question part for an example. Another example is that the authors use both $\\pi$ and $\\mu$ to denote a policy.\n\n-Writing could be improved. There are noticeable language errors, e.g., \"derivation\"-->\"deviation\", \"are in four-fold\", \"is substitute by\", etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WXxfrxyb4WP", "reviewer": "Reviewer_7gxf", "summary": "The paper tries to address the problem of inconsistency between the online and offline settings in the outbidding problem with safeguards. It proposes an online RL framework to improve the auto-bidding policies through direct interactions with the real environment. An online exploration policy is discussed trying to address safety and efficiency requirements. \n", "questions": "- It seems the discussion of the related literature is not very clear. For example, it is stated that if one trains the policy using data collected by a certain behavior policy directly from the RAS, then there will be certain extrapolation errors that degrade the policy performance. More discussion should be given here to avoid confusion. It is also helpful to be more specific and have some discussion on how this paper differs from the other safe online RL methods that try to explore the environment. It is also stated that there are methods using constraint functions, how does these methods related to this problem, and specifically why are they not good candidates?  \n\n- In the algorithm section, the algorithm details need more elaboration, also the description of the warm boosting section can be made more clear.\n", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper tries to address the problem of inconsistency between the online and offline settings in the outbidding problem with safeguards, and the experiments look encouraging.\n\nWeakness: In certain places, the presentation can be made clearer, and the algorithm details need more elaboration. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "2-P9OcXJDm3", "reviewer": "Reviewer_qefY", "summary": "In this paper, the authors provide a new online bidding algorithm using reinforcement learning and a safety function to accommodate the mismatch between online and offline data (IBOO challenge in the paper). The authors characterize the lipshitzness of the safety function and validate the effectiveness of the proposed algorithm through synthetic data and real advertising data.", "questions": "If I understand correctly, one of the main causes to IBOO problem is the mismatch of the data resource for RAS and VAS. Could the authors provide more details why we can only collect data in stage 2 in VAS?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This paper investigates a very important problem in practice. Auto-bidding is getting more and more popular in real production system and this paper provides a very reasonable approach to design real time bidding algorithm. The paper is well-written and clearly states the contribution. In addition, the authors show their algorithm can achieve very good performance empirically. I have several question as follows.\n\nWeaknesses: There is no theoretical guarantee for the proposed bidding algorithm, however, this doesn't really hurt the contribution of this paper since the authors show that their algorithm achieves good performance over real advertising.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept