Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Search
Authors: Zhiyu Mou
Yiqin Lv
Miao Xu
Cheems Wang
Yixiu Mao
Jinghao Chen
Qichen Ye
Chao Li
Rongquan Bai
Chuan Yu
Jian Xu
Bo Zheng
Venue: ICLR 2026 Oral (2026)
Keywords: auto-bidding
offline reinforcement learning
generative decision making
Primary area: applications to robotics, autonomy, planning

Abstract: Auto-bidding is a critical tool for advertisers to improve advertising performance. Recent progress has demonstrated that AI-Generated Bidding (AIGB), which learns a conditional generative planner from offline data, achieves superior performance compared to typical offline reinforcement learning (RL)-based auto-bidding methods. However, existing AIGB methods still face a performance bottleneck due to their inherent inability to explore beyond the static dataset with feedback. To address this, we propose AIGB-Pearl (Planning with EvaluAtor via RL), a novel method that integrates generative planning and policy optimization. The core of AIGB-Pearl lies in constructing a trajectory evaluator to assess the quality of generated scores and designing a provably sound KL-Lipschitz-constrained score-maximization scheme to ensure safe and efficient exploration beyond the offline dataset. A practical algorithm that incorporates the synchronous coupling technique is further developed to ensure the model regularity required by the proposed scheme. Extensive experiments on both simulated and real-world advertising systems demonstrate the state-of-the-art performance of our approach.

Published as a conference paper at ICLR 2026
ENHANCING GENERATIVE AUTO-BIDDING WITH OFFLINE REWARD EVALUATION AND POLICY SEARCH
Zhiyu Mou'
Yiqin Ly
Miao Xu'
Qi Wang Yixiu Mao
Jinghao Chen 1.2f
Qichen Ye!
Chao Li!
Rongquan Bai Chuan Yu'
Jian Xu
Bo Zheng!
'Taobao & Tmall Group of Alibaba, Beijing, China
DDepartment of Automation, Tsinghua University, Beijing, China wwwzhiyu..zy@taooee com, thh123gp gmail com, rongquan www.rabaa..ccom
ABSTRACT
Auto-bidding is critical tool for advertisers to improve advertising performance. Recent progress has demonstrated that AI-Generated Bidding (AIGB), which learns conditional generative planner from offline data, achieves superior per- formance compared to typical offline reinforcement learning (RL)-based auto- bidding methods. However, existing AIGB methods still face a performance bot- tleneck due to their inherent inability to explore beyond the static dataset with feedback. To address this, we propose AIGB-Pearl (Planning with EvaluAtor via RL), novel method that integrates generative planning and policy optimiza- tion. The core of AIGB-Pearl lies in constructing a trajectory evaluator to assess the quality of generated scores and designing a provably sound KL.-Lipschitz- constrained score-mas maximization scheme to ensure safe and efficient exploration beyond the offline dataset. A practical algorithm that incorporates the synchronous coupling technique is further developed to ensure the model regularity required by the proposed scheme. Extensive experiments on both simulated and real-world advertising systems demonstrate the state-of-the. the-rartt performance of our approach.
INTRODUCTION
The increasing demand for commercial digitalization has facilitated the development of the auto- bidding technique in online advertising. Distinguished from traditional manual bidding products, outo-bidding provides advertisers with an efficient and flexible scheme to automatically optimize bids in dynamic and competitive environments (Balseiro al., 2021a Deng al., [027] Balseiro let al. 2021b). Technically, auto-bidding constitutes an offline sequential decision-makinn problem that aims to maximize advertising performance over bidding episode, constrained static offline dataset due to operational safety concerns (Mou et al.) 2022).
As a standard approach offline decision-making problems, offline reinforcement learning (RL) (Kumar et al., 22200; Mao et al., 2024b) is widely adopted to solve the auto-bidding problem. By employing conservative policy search schemes, offline RL mitigates the infamous distribuiino (OOD) problem (Fujimoto et al., 2019), enabling reliable generalization beyond the offline dataset. However, their reliance on bootstrapped value estimates renders offline RL. methods prone training instability (Peng et al. 2024), potentially compromising policy performance.
Recent advances in generative models shed new light on offline decision-making problems (Zhu al. 2023 Kang et al. 2023). Specifically, Al-generated bidding (AIGB) models auto-bidding as task and employs a generative model to approximate the conditional trajec- tory distribution of the offline dataset (Guo et al. 2024h. AIGB avoids bootstrapping and exhibits more stable training and superior performance. However, the modeling approach in AIGB does not explicitly align with the relies on imitating trajectories from the offline dataset (Ajay et al. (023) and lacks the ability to improve its generation quality beyond the offline dataset based on the performance feedback. Con- sequently, its conditional generation in the extrapolation regime can become unreliable, potentially leading to performance degradation even risky trajectory generations.
a senectory-ggneratiinn
of the auto-bidding problem. Inherently, AIGB
"Equal contribution; Work was done during an internship at Alihaha Group: Corresponding authors.

---

Published as a conference paper at ICLR 2026
Hence, there arises a question: built on AIGB, the latest state-of-the-de auto-bidding method, can we devise a plausible scheme to involve policy optimization in its generative model? To this end, natural idea is to integrate offline RL methods into AIGB. However, it is nontrivial to implement in the auto-bidding problem since (i) there is lack of reward signals in AIGB to guide the gener- ative model. Specifically, the generation quality of the generative model remains unknown during training, making it infeasible to explore new trajectories beyond the offline dataset; (ii) no dedicated offline RL algorithm exists for AIGB. In particular, theoretical analysis that guarantees safe general- ization and mitigates OOD issues for generative models in auto-bidding remains largely unexplored.
To address these critical challenges, we propose AIGB-Pearl (Planning with EvaluAtor via RL), an RL-enbanced version of AIGB that learns a trajectory evaluator to score generation quality and drive exploration of the generative model through continuous interaction. The evaluator trained through supervised learning on the offine dataset. Crucially, to mitigate the OOD problem and ensure reliable use of the evaluator, we examine the theoretical upper bound on the evaluator's bias. Then, guided by this analysis, we formulate KL-LLpschhit-conntrraieed score-maximization objective with a provable suboptimality bound, enabling safe and effective exploration beyond the offline data. Morcover, to perform constrained score maximization, we design practical algorithm that incorporates the synchronous coupling technique to satisfy the generative model's Lipschitz condition. In addition, we note that AIGBB-Parl does not require bootstrapping and exhibits greater training stability than offline RI. methods.
To summarize, our contributions in this paper fourfold: (i) propose novel generative auto- bidding method, AIGB-Pearl, that enables continuous improvement in generation quality through exploration beyond the offline dataset; (ii) we propose provable KL-Lipschitz constrained score maximization objective with a sub-optimality bound, ensuring a safe and effective generalization beyond the offline dataset; (iii) we devise practical algorithm with synchronous coupling that effectively ensures the Lipschitz requirement for the generative model; (iv) extensive simulated and real-world experiments demonstrate that AIGB-Pearl achieves SOTA performance and verify the effectiveness of the developed techniques in enhancing safe and effective generalization.
2
PRELIMINARIES
2.1
PROBLEM STATEMENT
This work studies the auto-bidding problem for a single advertiser subject to a budget constraint. The auction mechanism follows a sealed-bid, second-price rule. The objective is devise bidding policy that maximizes the cumulative value of the impressions won over a finite bidding episode (e.g., day) within a budget B O. As established in (He al. 202I). the optimal bid for each impression i is proportional to its intrinsic value U; > O, scaled by non-negative factor a 2 0 that remains consistent across all impressions. Under this strategy, the advertiser wins an impression i an; 2 Pt and pays P; upon winning, where p; > 0 is the market price. The Return on Investment (ROI) of impression i is defined as / and we denote its upper bound as Rm - max; 0//P:-
However, the scaling factor is unknown a priori, and impression volatility drives its continual change throughout the bidding process. Hence, a standard practice involves recalibrating the scaling factor a at fixed intervals of T E N+ time steps (Guo et al., 2024) He al., 202T Mou al., 2022). This casts the auto-bidding to sequential decision-making problem.
Specifically, the auto-bidding problem can be modeled as a Markov Decision Process (MDP) < S, A. R, P > The state 8 - [t. -1 E S composed of the current time step t E [T], the cost ratio --1 = c > 0 where c;-1 is the advertiser's cost for impressions won between time step 1 - 1 and f, and static advertiser-specific feature x that includes the budget and many other individual information. The action a, E A denotes the calibrated scaling factor at time step t. The reward r. 2 0 describes the value of the impressions won between time steps t and t + 1, and P denotes the state transition rule. The auto-bidding problem can be formulated as:
T
max E PP4tsP(
Σ
r
s.t. Σ
C
(1)
01:02."
Offline Setting, Due to safety concerns common in real-world advertising systems- we are re- stricted to learning the optimal bidding policy from a static offline dataset DD comprising historical
2

---

Published as a conference paper at ICLR 2026
states and actions along with associated rewards. This makes the considered auto- auto-bidding problem an offline sequential decision making task.
2.2 OFFLINE RL METHODS
RL constitutes a standard approach for auto-bidding problems, seeking an optimal bidding policy S + A that maximizes cumulative reward. Specifically, this is typically achieved by learning
Q-value function, Q(=x; 01) A Fgg Tel, through temporal difference (TD) error minimization:
min B ((rrarres))) (Q(ss az) max ( 0c+1
(2)
where Q is a target Q-value function with parameters updated via Polyak averaging Mnih al, 2015). Upon convergence, the optimal bidding policy is derived as π (Tx)) = arg maxa, (s 0z).
Due to the offline setting of the considered auto-bidding problem, directly employing Eq. results in the infamous diitr-itdittiibttionn (OOD) problem (Fujimoto et al., 2019), making the policy emro- neously deviate from the offline dataset (Mao et al. 2002aa). As a standard solution, offline RL
Yu et al. 2020 Kumar et al. 2020 Mao et al. 2023 Kidambi et al.
2020) constrains the policy's
behavior near D during TD learning, enabling rehabble generalization beyond the offline dataset.
However, offline RL methods are notoriously unstable due to training instability caused by TD- learning (Peng et al. 2024), in which the bootstrapped value the Q-function serves as its training label, resulting in erroneous ground truth. Training stability is critical in auto- hidobidding due the absence of an accurate offiin policy evaluation method and the high cost of online policy evaluation in real-world advertising systems (Mou et al. 2022).
2.3 GENERATIVE
METHODS Definition 1 (Trajectory and Trajectory Quality). The trajectory is formalized as the state sequence ST). The trajectory quality is defined as the normalized cumulative reward of the trajectory, i.e. u(r) 2 Σ where Ft rt
throughout the bidding episode, i.e., é [s1, 82, 82,
Unlike RL methods, the AI- Al-generated auto- buto-biddding (AIGB) (Guo et 2024 treats the auto- biddin problem as a sequence generation task. Specifically, a conditional generative model is employed to fit the conditional trajectory distribution (o(T))(TT) within the offline dataset D, i.e.,
max E (log
(3)
where 0 denotes the parameter. Let ym 0 be the maximum trajectory quality in D, have Vy D,y y E [0, Vm]- During inference, AIGB follows a planning-and-come each time step. trajectory is sampled from the trained generative model that acts as the planner, with a manually set condition y* & (1+ where e > 0 is hyper-parameter Then, an extra off-the-shelf inverse dynamic model (Agrawal et al. 2016), acting as the controller, is employed compute the action. See Appendix B for detailed descriptions. AIGB avoids TD learning and generally outperforms offline RL methods (Guo et al.) 2024).
architecture. Specifically,
However, AIGB relies on imitating trajectories from the offline dataset (Ajay et al., 2023) and lacks the ability to improve its generation quality beyond the offline dataset based on the performance feedback. Consequently, AIGB's conditional generation can be unreliable without explicit reward guidance, rendering exploration undirected and potentially leading to performance degradation or even risky trajectory generation. Furthermore, no theoretical guarantee exists for the quality of the generated trajectory in this extrapolation regime.
the extrapolation regime (y* > ym)
METHOD
Enabling AIGB to explore higher-quality trajectories beyond the offline dataset with explicit reward guidance can enhance its performance and generalization. To this end, we propose AIGB-Pearl
Note that, as in real-world advertising systems, the bidding process will automatically suspend once the advertiser's 's budget runs out, and thereby, any action sequence will not violate the hudget constraint. Hence, the trajectory quality can he directly defined the cumulative reward of the trajectory.
3

---

Published as a conference paper at ICLR 2026
(Planning with EvaluAtor via RL) that constructs trajectory evaluator (hereinafter referred as the evaluator for simplicity) to integrate RL methods into AIGB's s planner training. Specifically, the evaluator learns a score (T) to estimate the trajectory quality у(T) via supervised learning based the offline dataset D, ie.. ming Br~DD(@o(T parameter. Then, with @ fixed, the planner tries to maximize the score of its generation through iterative interactions with the evaluator, shown in Fig. 2 Formally, this can be formulated as:
y(T))²,, where 6 denotes the evaluator
max L(0) & E, Po (T))
where the condition is fixed to yy both training and inference stages to ensure consistency.
As shown in Eq. 4 the effectiveness of AIGB-Pearl hinges critically on the evaluator's reliability. However, given the offline nature of the considered auto-bidding problem, the evaluator training is confined to the fixed dataset D. Although we incorporate several techniques into the evaluator's su- pervised training to enhance its prediction accuracy (as detailed in Section 3.2. and Appendix [E.]. directly applying Eq. 4 to the planner can still trigger the infamous OOD problem due to the evalu- ator's generalization limits outside the data support, potentially degrading the planner's 's true perfor- mance. This issue is particularly acute in auto-bidding, risk-sensitive domain in which suboptimal or anomalous trajectory generation can result in substantial monetary losses or campaign failures. Notably, there remains lack of theoretically principled approaches this challenge.
To address this challenge, we first analyze the theoretical bounds on the evaluator's bias in Sec- [.1] Then, guided by this analysis, we propose -L-Lipschit--conttrinedd score-maximization objective for the planner to ensure reliable use of the evaluator. Notably, this objective is theoreti- cally justified by sub-optimality bound established in Section 3.I.I] Finally, practical algorithm is presented in Section 3.2] which employs synchronous coupling method to satisfy the planner's Lipschitz constraint in Section 3.2.22
3.1 KL-LIPSCHITZ-CONSTRRANN SCORE MAXIMIZATION
This section focuses on the reliable exploitation of the evaluator-guice -ugide score maximization.
Our basic idea is to optimize within a domain where the gap between the planner's score L(0) and its true performance J(0) P Eep (v")
[[(T]] is bounded by a small certifiable upper bound. This ensures the score maximization occurs only in regions where the evaluator is reliable.
(J(0) (() [Er~po(Fle [v(T)] Er<pa(rlv (T)]l-
(5)
In the following, we investigate this gap. Specifically, find that the trajectory quality y(r) is Lipschitz continuous as stated in Theorem [] The proof is given in Appendix C.I
Theorem (Lipschitz Continuous of ((T).). The trajectory quality y(T) is -ipsschitt con- tinuous with respect to the Frobenius norm.
Motivated by Theorem we enforce a VTR -ipscchiz regularity on the evaluator's training to inherit the Lipschitz continuity of the true trajectory quality y(r) (as described in Section 3.2. As the trained evaluator's Lipschitz constant may not equal /T exactly, denote A T Rm.. where & 2 0 quantifies the degree of violation. Note that tighter adherence of the evaluator to the Lipschitz constraint yields a value of closer to
Equipped with Theorem I and the Lipschitz property of the evaluator, we derive the following upper bound on the performance gap between (8) and L(0), and the proof given Appendix C.2
Theorem 2 (Evaluator Bias in Planning Performance Bound). Let the upper bound of the evaluator's bias on its training dataset 'D be ōD > O, iee. ET-DlU(T) - ÚØ(T) ) S 8p, and let the Lipschitz (T)
constant of (o(T) be kyTRm. The gap beiween the planner's 's score L(0) and its true performance (() can bounded by:
(J(0) - -(0) < 6n + (1
W po(T)))) + W Wiipp(rrw)
Generation sensitivity
Imination error D
where W1 denotes the 1- 1-Wasserstein distance.
4

---

Published as a conference paper at ICLR 2026
Note that dD could be regulated to small value via supervised training of the evaluator, and de- pends on the Lipschitz property of the resulting evaluator as stated before Consequently, bounding the evaluator bias in the planner's performance requires constraining the following two factors:
* the planner's generation sensitivity to condition y (the first Wasserstein term)
the planner's 's imitation error on the offline dataset (the second Wasserstein term).
Specifically, we establish that the first Wasserstein term can be bounded by the Lipschitz constant Lipw, (Po(Tl))) of the planner with respect to the condition y measured by W1:
By-ro (v)
& (1 + c)ym (Po(TlU)).
(6)
The proof given Appendix C.3 Therefore, to ensure the boundedness of the first Wasserstein term, we constrain the planner's s Lipschitz constant Lipw, (Pa(r))) to positive hyperparameter Lp. A lower bound analysis L, is provided later in Eq. 10
Moreover, establish that constrained KL divergence By>pp(u) [DK (P S 6g could bound the expectation of the second Wasserstein distance term as follows, where åÆ > is hyperparameter and can be set small value, close zero. See Appendix (erendixC..4foor for the proof. Note that the KL divergence constraint here inherently makes the planner perform conditional behavior cloning the offline dataset D (Guo al., 2025).
Eyrpoty) [ (po(Tly) S Vōk.
Collectively, to effectively perform score maximization with small, certifiable evaluator bias, enforce Lipschitz continuity of the planner with respect to the condition y, while preserving its behavior cloning fidelity to the offline dataset D. Formally, Eq. 4 is transformed to:
max 0
L(O)
(Score Maximization)
(8) R
s.t.
(v) DKL (Po(l))))))l s 8g (KL Constraint)
Lipw, (Po(Tly)) S Ly
(Lipschitz Constraint)
Eq. forms the score maximization objective in AIGB-Pearl.
Remark 1. Intuitively, the KL and Lipschitz constraints jointly ensure the planner's 's gen- eration under condition y² remains within certified neighborhood of the high-quality tra- jectories in the offline dataset D. As illus- trated in Fig. П the green region represents the feasible set of trajectories po po(Tlu) satis- fying the KL constraint for y E D The Lipschitz constraint makes the generated tra- jectories po(T(yy") remain within a neighbor- hood of the best-quality trajectory in the offline dataset pet (T ym), as illustrated by the blue cir- cle with radius eLyyJm- The union of all such circles constitutes the total trajectory explo- ration range. Meanwhile, the evaluator trained on D maintains high accuracy within this 'D- proximal region, and the Lipschitz regulariza- tion on the evaluator bounds its sensitivity input perturbations, preventing drastic value fluctua- tions in OOD regions and promoting more reliable extrapolation. Overall, KL-Lipschitz constrained optimization, guided by Lipschitz continuity evaluator, enables safe trajectory improvement within a theoretically certified neighborhood of high-quality offline trajectories, effectively mitigating risky OOD exploration.
Innjetiery Qualitty Chattear
Mudie Goemand Traaeteyy
Coeatrained Irgochey ingrovement
Lipechii Canoznn:
sitC Preel
KL Contrain
Traletiry Improemes Pundy
Eqpionaiio Rinpe
Trijectry Exalaatar Cortear
Offiee Dun Quality (aange
Figure 1: Schematic illustration of the KL-Lipschitz- constrained moreemmaximzztioo. It enables safe trajec- tory improvement within theoretically certified neigh- borhood of the high quality trajectories in the offline dataset, where evaluator remains high accuracy.
Note that & cannot approach zero without compromising 6D. as excessively small prevents the evaluator from fitting the offline dataset DD
NNte that the KI. -Il-constri feasible region may not he regular topology since the generated trajectories (T(gg) should also satisfy the Lipschitz constraint under condition y E D.
5

---

Published as a conference paper at ICLR 2026
3.1.1 SUB-OPTIMALITY GAP BOUND
This section focuses on presenting and analyzing the sub-optimality bound of the solution the pro- posed Eq. 8 Specifically, denote the solution the true performance [0) 0" f arg max, J()),
and denote the solution the proposed Eq- 8 as ô. The following theorem gives the sub-optimality bound of the planner's performance, and the proof is given in Appendix C.5
Theorem 3 (Sub-optimality Gap Bound). Let êm 2 Ey--pn(y) [D (PD(Tlu) (ool))))lpp (Tly"))] be the expected distance between the optimal trajectory distribution and the trajectory distribution of the offline dataset 'D. The true performance gap between the optimal parameter 0* and the solution Eq. iis bounded by:
VRR [VoN + Vōx + (1 c)
J(0") - ((0)
J(0)) - 26D +
(1 +(1
(9)
Theoretical Result Analysis. In Theorem the constants dM; Rm, T, T,6, Um characterize domain- specific properties of the auto-bidding task and the offline dataset D. Nonetheless, lower training error dD of the evaluator and closer & to 1 correspond to smaller sub-optimality gap. Note that cannot be smaller than I without compromising ód, as excessively small prevents the evaluator from fitting the offline dataset D.
Moreover, in Theorem 3 a lower behavior cloning error 5g and a lower Lipschitz constant Lp of the planner lead to smaller sub-optimin uut-optimality gap. However, excessively small Lp prevents the planner from behavior cloning the offline dataset D (as required by the KL constraint), resulting in a large 5x- Actually, theoretical lower bound for T,p is given by the Lipschitz constant of the conditional trajectory distribution of the offline dataset (D(rly)
W (p (T (D(()
Lp sup
(10)
yy1 - 3/2
where V1. 1/2 E D. Consequently, we leverage this lower bound of L, in AIGB-Pearl. A further discussion on the theoretical performance range of AIGB-Pearl is given Appendix
3.2 PRACTICAL ALGORITHM DESIGN
This section focuses the practical algorithm implementation of Eq. 8 Section 3.2.1 first presents our reliabilit--ehhaneed evaluator architecture, followed by the synchronous-coupling based Lips- chitz planner design in Section 3.2.2
3.2.1 LIPSCHITZ TRAJECTORY EVALUATOR
As shown in Fig. 2 the evaluator processes the trajectory to predict score (T(T) for quality estimation. The evaluator trained via supervised learning using the offline dataset D. Besides, to satisfy VTR,, - Lipschitz constraint requirement according to Theorem 2 ization term to the training loss of the evaluator, which can be expressed as:
we add Lipschitz regular-
Le(0) ETAD (i+(T) - y())) +B1 Er,.. (Do(T)) (o( VTRm l
- 72
(11)
fitting the ground truth
Lipschite penalty
where B1 > is hyper-parameter, []+ max .). Moreover, further enhance the predic-
tion accuracy of the evaluator, we devise two specific techniques, including the LLM Embedding enhancement and pair-wise learning, whose details are given Appendix E.
3.2.2 LIPSCHITZ PLANNER WITH SYNCHRONOUS COUPLING
As shown in Fig. 2 the planner is implemented by Causal Transformer (Chen et al. 202 that generates trajectories an auto-regressive manner. Specifically, the model takes the condi- tion y and history states S1. input tokens, and predicts the next state Gaussian distribution, pn($:41|81:4 y) = N (Nu (S12. y. t), o (s y, t)), where μ4 denotes the mean and 0g > the stan- dard deviation. During the auto-regressive generation process, each output state is sampled from the

---

Published as a conference paper at ICLR 2026
-Conditional BC Score +ippchiit Penalty
Synchronaus Coupling Wasserstein Estimation
Trajectory Evaluator
Cumplete Trajedury
X2
+
+
A
Causal Transformer Planner
Cousal Transformer Planner
Causal Transformer Planner
+
Post Trijenory
Generated Trnjenory
The condition used in score maximization
State in the past trajectory
State in the gonersted Irajedicyy
Standard Gaussian Noise
Figure 2: AIGB-Pearl (Planning with EvaluAtor via RL) constructs trajectory evaluator to score the trajectory quality and let the planner maximize the obtained score under the KI-Lipschitz con- straint through continuous interaction with the evaluator. A synchronous coupling method used to estimate the Wasserstein term in the Lipschitz penalty.
Gaussian distribution using the reparameterization trick, i.., St41 po($1. v. +00 M, M.t). where ~ N(0, D
Regularized Planner Training Loss. To perform the score maximization in Eq. we involve two regularization terms in the planner's training loss (0). including a conditional behavior cloning
loss, corresponding to the KL constraint Eq. Sp and a Lipschitz penalty loss, corresponding to
Lipschitz constraint Eq. 8p i.e.,
(0) E popo(Tly yy [D)(T)]
.2 (T.)))) [log (T (Tl)
planner score L(6)
conditional behavior clone
+ B3 w.
Wi (p (i - Lplyi 1/2
(12)
Lipschitz penalty, where Wi (Po(Tlu1) (p((((/2))) replaced by Wi Wits
where B2, A33 > O are two syper-parameters, With prior RL works (Sutton et al. 1999), we can derive the closed-form expression of planner's score gradient (ol as shown in Appendix C.66 The core of the planner loss lies in the computation of W, (pa(Tly)) Po (1())))).
Wasserstein Upper Bound as Surrogate. Accurate computation this Wasserstein distance term is challenging, as it requires finding the optimal coupling between po(Tly)) and po(Tlu)) that minimizes the expected transportation cost. Nonetheless, we can choose a certain coupling y (((pe(Tl))) po(T)z))) to obtain an upper bound of this Wasserstein term, i.c.
inf (e (eT(po(llpc)uooorle))) Eam Σ |s sill
E Bm.ra((on)) Σ
||s sill
A W1(,
(13)
where W, (81, 32: 8) denotes the upper bound, and sf is the t-th state in trajectory T;. It can be that Wivn, y25 0) Lplyı vv y2l acts sufficient condition make the planner L,-Lipschitz continuous. Thus, we replace (Po((T))))) (o (T(())))) by this upper bound in the planner loss.
Synchronous Coupling Wasserstein. Instead of using random couplings, employ a synchronous coupling 7sync to make the upper bound tighter. Specifically, two trajectories and conditiinoddd on y and y2, rwwpectively-aaeeeeee are generated using the same sequence of Gaussian noise (πı, 72, ...
Note that the time step and the static advertiser feature x in the state generated. We only generate the next ratio e in practice.
[t, Ce-1, x]] do not need to be

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance (GMV) in simulated experiments with 30 advertisers. A indicates the
relative improvement of AIGB-Pearl against the most competitive baseline (which is underlined). Note that the absolute values are normalized without specific meanings; only A matters.
Budget
USCB
BCQ
CQL
IQL
MORL MOPO
DT
DiffBid AIGB-Pearl
A
1.5k
454.25
454.72
461.82
456.80
468.49
470.38
477.39
480.76
502.98
+4.62% %
2.0k
482.67
483.50
475.78
486.56
488.12
489.27
507.30
511.17
521.84
+2.09 %
2.5k
497.66
498.77
481.37
518.27
511.93
523.91
527.88
531.29
545.03
+2.59%
3.0k
500.60
501.86
491.36
549.19
553.91
549.01
550.66
556.32
574.17
+3.21%
Table 2: Overall performance in real-world A/B tests, involving 6k advertisers over 19 days.
Methods
GMV
BuyCnt
ROI
Cost
Methods
GMV
BuyCnt
ROI
Cost
DiffBid
76,390.174
650,962
5.31
14,395,290
USCB
52,182,805
516,994
4.92
10,598,486
AIGB-Pearl
78.676.009
665.173
5.41
14,551.054
AIGB-Pearl
53,973.101
520,796
10,515,772
A
+3.00%
+2.20%
+1.89%
+1.10%
A
+3.43%
+0.74%
+4.24%
-0.78%
Methods
GMV
BuyCnt
ROI
Cost
Methods
GMV
BuyCnt
ROI
Cost
DT
34,808.665
341,995
5.61
6,205,665
MOPO AIGB-Pearl
51,674,071
579.332
3.08
16,771,922
AlGB-Pearl A
35,957,933
344,194
5.77
6,246,512
53,292,945
591,741
3.23
16,475,670
%
+0.64%
+0.16%
+0.66%
A +3.13%
+2.14%
+4.87%
-1.77%
The definition of W1 (31, y2; 0) is given in Eq. 13 Compared to random couplings, the synchronous coupling reduces spurious variance in the trajectory comparison by aligning stochasticity through shared noise, resulting in a tighter upper bound on the Wasserstein distance (Lindvall (002).
Moreover, if we make the predicted variance da of the planner fixed constant, then the expression of Wi (U1, 3/; ()) can be further simplified to Wi (y1, y2; = Eg |1 (s /1, mo(s..e V2. I-|l-
The overall AIGB-Pearl algorithm is summarized in Algorithm in Appendix due to page limits.
4 EXPERIMENTS
We conduct both simulated and real-world experiments to validate the effectiveness of our approach. In the experiments, mainly investigate the following Research Questions (RQs): (1) Does hancing AIGB with policy optimization improve overall performance, and can it generalize better unseen data compared to existing AIGB methods? (Section 4.2) (2) How does the KI.-Lipschitz constraint affect the performance of the planner? (Section[4.3) (3) Can the proposed method guaran- tee the Lipschitz property of the evaluator and the planner? (Section (4) What is the evaluator's accuracy on the training data, and how well does it generalize to unseen data? (Section 4.5) The
training stability of AIGB- Pearl is studied Appendix F.5
4.1 EXPERIMENT SETUP
Experiment Environment. We conduct simulated experiments in an open-source offline advertis- ing system with 30 advertisers of four budget levels (1.5k, 2.0k, 2.5k, and 3.0k), as in (Mou et al.] 2022 Guo al. 2024). The offline dataset comprises 5k trajectories generated by 20 advertisers. Extra detailed settings of simulated experiments are given in Appendix F. For real-world experi- ments, we conduct online A/B tests on one of the world's largest E-commerce platforms, TaoBao. The offline dataset comprises 200k trajectories of 10k advertisers. See Appendix F.2 for extra de- tailed settings of real-world experiments. In both simulated and real-world experiments, we employ the same inverse dynamics model from Agrawal et al. (2016) as the controller in AIGB. Moreover, the evaluator is trained on the entire offline dataset, and its generalization ability evaluated using K-fold cross-validation with K = 5.
Baselines. We compare our method with state-of-the-art AIGB methods, including DiffBid (Guo et al. 2024) and DT (Chen et al.) 2021), which learn from conditional behavior cloning of offline datasets using a diffusion model and a Causal Transformer, respectively. We also compare our method with RL auto-bidding methods, including USCB (He et al.] 2021) that learns the auto- bidding policy in a manually constructed advertising system with DDPG (Silver et al.] 2014); and offline RL auto-bidding methods, including model-free offline RL methods BCQ (Fujimoto et al. 2019), CQL (Kumar et al., 2020), and IQL (Kostrikov et al. 2022). and model--baser offline RL methods MOPO (Yu et al. 2020) and MORL (Mou et al. 2025).
8

---

Published as a conference paper at ICLR 2026
Table 3: Generalization performance in real-world A/B tests with unseen advertisers against AIGB methods, involving 4k advertisers over 19 days.
Methods
GMV
BuyCnt
RO1
Cost
Methods
GMV
BuyCnt
ROI
Cost
DifTBid
67.092,973
553,020
5.39
12,444,306
DT
30,562, 007
300,271
5.61
5,450,573
AIGB-Pearl
69,252,539
565,776
5.53
12,534,379
AIGB-Pearl
31,502.309
305,202
5.74
5,484,473
A
+3.32%
+2.31%
+2.48%
+0.72%
A
+3.08%
+1.64%
+2.32%
+0.62%
Table 4: Ablation Study. The effectiveness of the KL constraint and the Lipschitz constraint in Real- world A/B tests, involving 6k advertisers over 8 days.
AIGB AIGB-Pearl|
GMV
BuyCnt
ROI
Cost
Methods
GMV
BuyCnt
ROI
Cost
w/o KL
30,906,963
292,605
4.25
7,269,018
w/o Lipschitz
32,284,972
268,551
5.73
5,634,304
with KL A
31,243.688
292,783
4.26
7,342.485
with Lipschitz
32,869.329
281,979
5.79
5,678,252
+1.09%
+0.06% %
+0.08%
+1.01%
A
+1.81%
+0.50%
+1.05%
+0.78%
Performance Index. The objective in the auto-bidding problem Eq. [ i... the cumulative rewards over the bidding episode, acts as the main performance index in our experiments and is referred to as the gross merchandise volume, GMV. In addition, utilize three other metrics commonly used in the auto-bidding problem to evaluate the performance of our approach. The first metric is the total number of impressions won over the bidding episode, referred to the BuyCnt. The second metric is the Cost over the bidding episode, and the third is the return on investment ROI, defined as the ratio of GMV to Cost. Note that larger values of GMV, BuyCnt, and ROI with Cost oscillating within an acceptable tolerance (+2%) indicate a better performance.
4.2 OVERALL PERFORMANCE
To answer RQ(): Table D shows that our method consistently outperforms all baselines in GMV across all four budget levels in simulated experiments. In real-world experiments, Table that our method also achieves superior performance on GMV, BuyCnt, and ROI, with Cost fluctuations of less than 2%. Notably, both simulated and real-world experiments consistently demonstrate that AIGB-Pearl achieves a +3% % improvement in GMV over the AIGB, the state-of-the-an auto-bidding method. Since our method and DiffBid share the same controller, the performance gain stems solely from the planner. This provides strong empirical evidence that the proposed conservative RL learn- ing for score maximization effectively enhances overall performance.
Notably, we also apply AIGB-Pearl to another important auto-bidding problem, named TargetROAS (See Appendix F.3) Real-world experiments show that AIGB-Pearl achieves +5% improvement in GMV compared to AIGB. It is worth noting that a GMV uplift exceeding 2% is highly significant, translating to millions of RMB in additional daily GMV on Taobao-scale advertising platforms.
Generalization Ability. We evaluate AIGB-Pearl on advertisers not used to generate trajectories in the offline dataset and compare it with existing AIGB methods. For simplicity, we refer to these ad- vertisers as advertisers outside the offline dataset. Sable(]reports the performance on 4k advertisers outside the offline dataset in real-world experiments. AIGB-Pearl consistently delivers better results in GMV (43 ),), BuyCnt, and ROI, while maintaining Cost fluctuations within 2% compared to the baselines. This indicates that the proposed method has better generalization ability than AIGB.
4.3 ABLATION STUDY
To answer RQ(2): We remove the KL and Lipschitz constraints from AIGB-Peal individually and evaluate the model's performance each ablated variant using real-worid A/B tests. The results are presented in Table E It can be seen that the KI constraint contributes +1.1 % improvement in GMV, and the Lipschitz constraint provides +1.8% improvement GMV, demonstrating their respective roles in enhancing AIGB-Pearl's performance.
Visualization. Three AIGB-Pearl-genereee trajectory examples are presented in Fig. 3 As can be observed, the trajectories generated by AIGB-Pearl are plausible. In contrast, the ablated variant without the KL and Lipschitz constraints produces trajectories that deviate significantly from the optimal trajectory in the offline dataset and exhibit clear pathological behaviors-sute sive budget consumption, backward- backward-trending trending pacing, and undliittt of available budgets (see Appendix F.4 for (xpplanation--whiin further validate the KL--Lipschizz constraint necessity.
exces-

---

Published as a conference paper at ICLR 2026
Case
Case 2
Casc 3
Bedatte Tha Sag
6:25-
Tickoardia
0,025
0..0
C/220
6015-
6015-
0.02
6010
6.00-
0.11
Excosise
0:000
0.000
Optnnal Tegeedery Doteset
Peul Gesented Togextar
KL.Cllaii Genereed Trgestery
Figure 3: Trajectory Generation Visualization. Three cases are presented. Here, the AIGB-Peat generates plausible trajectories, whereas its variant without the KL-Lipschitz constraint produces generations that significantly deviate from the reference and exhibit evident issues.
Table 5: Evaluator accuracy for simulated and real-world experiments.
Results are reported for
training data and OOD data evaluated using 5-fold cross-validatiinn
Simulated Exp
Triining Data
OOD Cross-Validationn
Real-world Exp
Training Data
ODD Data (Cross- Validarion)
MAR L
0.6
0.7 + 0.06
MAR L
1.0
1.2 - 0.03
AUCT
89.9%
85,5% + 0.5%
AUC
77.4%
75.1% + 0.2%
4.4 LIPSCHITZ VALUE EXAMINATION
To answer RQ(3): We report that the Lipschitz value of the trajectory quality U(T) and the con- ditional trajectory distribution pD of the offline dataset are 1.62 and 0.38, respectively. We set Lp = 0.50, which is close to its lower bound estimate of 0.38 F] To calculate the Lipschitz constants of the evaluator and the planner, we sample 8, 000 pairs of trajectories and compute their Lipschitz constants. The results are shown in Fig. and Fig. 5 It can be observed that most sample values satisfy the Lipschitz constraint, and the Lipschitz constants of the evaluator do (7) and planner po(T y) are 2.2 and 0.56, respectively, near 1.62 and 0.50. This indicates that the models' Lipschitz constraints are successfully satisfied.
Figure 4: Examination Figure 5: Examination of Evaluator Lipschitz. of Planner Lipschitz.
4.5
EVALUATOR ACCURACY EXAMINATION
The evaluator's 's accuracy is assessed along two dimensions, including the abso- lute accuracy measured by mean absolute error (MAE) metrics, reflecting how close the predicted scores are to ground truth scores, and the ranking accuracy by AUC metric, reflecting the correctness of relative rankings between trajectory pairs. Note that the MAE of each advertiser's data sample normalized by its budget to ensure comparability across advertisers. A lower MAF, together with higher AUC, indicates better evaluator accuracy.
Accuracy Metrics.
To answer RQ(4): We report the accuracy of the trained evaluator in both simulated and real- world experiments in Table 5 We evaluate the evaluator's accuracy on the training data and its generalization ability using K-fold coss-validation, where K 5. To the best of our knowledge, we are the first to introduce the trajectory evaluator into the generative biddin framework. The reasonableness of our evaluator is evidenced by its pairwise ranking accuracies of 86% AUC and 75% AUC on OOD trajectories in simulated and real-world experiments, respectively, substantially above the 50% random-chance level, despite the high complexity and dynamic nature of the bidding environment. Importantly, with the guidance of the trained evaluator, the planner outperforms state- of-the-art AIGB methods even on OOD data, demonstrated in the Table.
5 CONCLUSIONS
This paper proposes AIGB-Pearl, which enhances AIGB by incorporating reward evaluation and policy optimization. By introducing a trajectory evaluator and provably KL-LLipshittz-conttriin score-maximization objective, our approach ensures safe and efficient generalization beyond the offline dataset, supported by theoretical guarantees. Extensive simulated and real-world experiments validate the sate-of-the-art performance of our approach.
As the data-driven lower hound estimation can be underestimated, we slightly increase its value in practice.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physies. Advances neural information processing systems, 29, 2016.
Anurag Ajay, Yilun Du, Abhi Cupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Santiago Balseiro, Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo. Robust auction design in the auto-bidding world. Advances in Neural Information Processing Systems, 34: 17777-7778788 2021a.
Santiago R Balseiro, Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo. The landscape of auto-bidding auctions: Value versus utility maximization. In Proceedings of the 22nd ACM Conference on Economics and Computation, pp. 132-133, 2021b.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 345, 1952.
Zhe Cao, Tao Qin, Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 136, 2007.
Li Chen, Guanliang Chen, and Feng Wang. Recommender systems based on user reviews: the state of the art. User Modeling and User-Adapted Interuction, 25:99-154, 2015.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Yuan Deng, Jieming Mao. Vahab Mirrokni, and Song Zuo. Towards efficient auctions in auto- bidding world. In Proceedings of the Web Conference 2021, pp. 3965-39777 2021.
Yuan Deng, Negin Golrezaei, Patrick Jaillet, Jason Cheuk Nam Liang, and Vahab Mirrokni. Multi- channel autobidding with budget and roi constraints. International Conference on Machine Learning, pp. 7617-7644. PMLR, 2023.
Zhijian Duan, Yusen Huo, Tianyu Wang, Zhilin Zhang, Yeshu Li, Chuan Yu, Jian Xu, Bo Zheng, and Xiaotie Deng. An adaptable budget planner for enhancing budget-constrained auto-bidding in online advertising. arXiv preprint arXiv:2502.051877 2025.
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052 2062. PMLR, 2019.
Djordje Gligorijevic, Tian Zhou, Bharatbhushan Shetty, Brendan Kitts, Shengjun Pan, Junwei Pan, and Aaron Flores. Bid shading in the brave new world of first-price auctions. In Proceedings of the 29th ACM International Conference on Information Knowledge Management, pp. 2453-2460, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63((1): 2020.
Ziyu Guan, Hongchang Wu, Qingyu Cao, Hao Liu, Wei Zhao, Sheng Li, Cai Xu, Guang Qiu, Jian Xu, and Bo Zheng. Multi-agent cooperative bidding games for objectobeeetivee optimization in e-commercial sponsored search. arXiv preprint arXiv:2 /06.0//56 2021.

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12940 2025.
Jiayan Guo. Yusen Huo, Zhilin Zhang, Tianyu Wang. Chuan Yu. Jian Xu, Bo Zheng, and Yan Zhang. Generative auto-bidding via conditional diffusion modeling. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. 5038-5049, 2024.
Yue He, Xiujun Chen, Di Wu, Junwei Pan, Qing Tan, Chuan Yu, Jian Xu, and Xiaoqiang Zhu. A unified solution to constrained bidding in online display advertising. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, pp. 2993-3001, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Junqi Jin, Chengru Song, Han Li, Kun Gai, Jun Wang, and Weinan Zhang, Real-time bidding
with multi-agent reinforcement learning display advertising. In Proceedings of the 27th ACM International Conference Information and Knowledge Management, pp. 2193-2201 2018.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:67195- 67212, 2023.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Diederik Kingma and Max Welling. 1rXiv:1312.6114, 2022.
Auto-encoding variational bayes. arXiv preprint
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference Learning Representations, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179-1191 2020.
Yu Lei, Wenjie Li, Ziyu Lu, and Miao Zhao. Alternating pointwise-pairvin learning for personal- ized item ranking. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 2155-2158, 2017
Yinchuan Li, Xinyu Shao, Jianping Zhang, Haozhi Wang, Leo Maxime Brunswic, Kaiwen Zhou, Jiqian Dong, Kaiyang Guo, Xiu Li, Zhitang Chen, Jun Wang, and Jianye Hao. Generative models in decision making: A survey. arXiv preprint arXiv:2502.17100, 2025.
Torgny Lindvall. Lectures on the coupling method. Courier Corporation, 2002.
Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regulariza- tion for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 40587-40609, 2023.
Yixiu Mao, Qi Wang, Chen Chen, Yun Qu, and Xiangyang Ji. Offline reinforcement learning with ood state correction and ood action suppression. Advances in Neural Information Processing Systems, 37:93568 -36601, 2024a.
Yixiu Mao, Qi Wang, Yun Qu, Yuhang Jiang, and Xiangyang Ji. Doubly mild generalization for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:51436- 51473, 2024b.
12

---

Published as a conference paper at ICLR 2026
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature, 518(7540):529-555 5339 2015.
Zhiyu Mou, Yusen Huo, Rongquan Bai, Mingzhou Xie, Chuan Yu, Jian Xu, and Bo Zheng. Sustain- able online reinforcement learning for auto-bidding. Advances in Neural Information Processing Systems, 15:2651-26663 2022.
Zhiyu Mou, Miao Xu, Wei Chen, Rongquan Bai, Chuan Yu, and Jian Xu. Permutation equivariant model-based offline reinforcement learning for auto-bidding. arXiv preprint 2tt.::2550..999999 2025.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional conference on machine learning, pp. 8162-87 8171. PMLR, 2021.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit and incomplete trajectories. In International Conference Machine Learning, pp- 36878-26890 PMLR, 2023.
Zhiyong Peng, Yadong Liu, and Zongtan Zhou. Deadly triad matters for offline reinforcement learning. Knowledge-Bact Systems, 284: 1 1341, 2024.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp- 387-395. Pmlr, 2014.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265, pmlr, 2015.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387790519.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz. Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2008
Hao Wang, Bo Tang, Chi Harold Liu, Shangqin Mao, Jiahong Zhou, Zipeng Dai, Yaqi Sun, Qian- long Xie, Xingxing Wang, and Dong Wang. Hibid: A cross-channel constrained bidding system with budget allocation by hierarchical offline deep reinforcement learning. IEEE Transactions on Computers, 73(3):815 82888885888888888888888888888888 2023.
Menghan Wang, Yuchen Guo, Zhenqi Zhao, Guangzheng Hu, Yuming Shen, Mingming Gong, and Philip Torr Mp2: A momentum contrast approach for recommendation with pointwise and pair- wise learning. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2105-2109, 2022a.
13

---

Published as a conference paper at ICLR 2026
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large- scale learning: An efficient and user-friendly scaling library. arXiv preprint arXiv:2506.0612) 2025.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022b.
Chao Wen, Miao Xu, Zhilin Zhang, Zhenzhe Zheng, Yuhui Wang, Xiangyu Liu, Yu Rong, Dong Xie, Xiaoyang Tan, Chuan Yu, et al. A cooperative-compettiv multi-agent framework for auto- bidding online advertising. Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1129-1139, 2022.
Wush Chi-Hsuan Wu, Mi- Yen Yeh, and Ming-Syan Chen. Predicting winning price in real time bidding with censored data. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1305-1314, 2015.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14442, 2020.
Zhengbang Zhu, Hanye Zhao, Haoran Hc, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: A survey, arXiv preprint arXiv:2311.01223, 2023.
14

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Preliminaries
Problem Statement
2.2
Offline RL Methods
2.3
Generative Auto-bidding Methods
3
Method
3.1
Lischiieett score maximization
3.1.1
sub-optimality gap bound
3.2
Practical Algorithm Design
3.2.1
Lipschitz Trajectory Evaluator
6
3.2.2
Lipschitz Planner With Synchronous Coupling
Experiments
Experiment Setup
8
Overall Performance
4.3
Ablation Study
Lipschitz Value Examination
10
4.5
Evaluator Accuracy Examination
10
5
Conclusions
10
Related Works
17
A.I RL-based Auto-bidding Methods
17
A.2 Generative Auto Auto-bidding Methods
17
AIGB Method Details
17
B. Training Stage
18
B.2 Inference Stage
18
Theoretical Proofs
19
Proof of Theorem
19
Proof of Theorem
20
Proof of Eq. 6
21
Proof of Eq.
22
5 Proof of Theorem
23
6
Proof of Score Gradient
24
D
Theoretical Performance Range Tradeoff Discussion
24
15

---

Published as a conference paper at ICLR 2026
AIGB-Pearl Algorithm Summary
25
Additional Designs For Evaluator Accuracy Enhancement
25
E.1.1
LLM Embedding Enhancement
25
Pair-wise Loss
26
E.
13
Overall Evaluator Loss
26
Additional Experiments
26
FT
Simulated Experiment Settings
26
................................
Real-world Experiment Settings
27
Real-world Experiments TargetROAS Bidding Problem
................................
28
Pathological Trajectory Behavior Explanation
28
F.5
Training Stability
29
F6
Evaluator Accuracy
F7
Empirical performance with general offline data distributions
................................
30
F8
Hyper-parameter Tuning
Extending AIGB-Pearl to first-price auctions
32
Extending AIGB-Pearl to Online Setting Discussion
32
LLM Usage
32
16

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
A.1 RL-BASED AUTO- BIDDING METHODS
Auto-bidding plays a critical role in online advertising by automatically placing bids, allowing ad-
vertisers to participate efficiently in real-time auctions (Balseiro et al.] 2021a Deng et al. 2021
Balseiro et al. 2021b). The auto-bidding problem can be modeled as a Markov Decision Process
and addressed using reinforcement learning techniques. USCB (He al. (021) proposes unified
solution to the constrained bidding problem, employing the RL method DDPG (Silver et al.
2014)
to dynamically adjust parameters toward an optimal bidding strategy. Mou et al. (2022) propose
sustainable online reinforcement learning framework that alternates between online exploration and offline training, thereby alleviating the sim2rel problem. few studies explore multi-agent RL for auto-bidding (Jin et al. [0018] Guan et al. [2021; Wen et al.) 2022), while several focus on budget al- scenarios using RL-bbased approaches (Wang al.) 2023 Deng et al. 2023 Duan al. (2025). Importantly, offline RL methods such as BCQ (Fuji- moto et al. 2019). CQL (Kumar et al. 2020), IQL (Kostrikov et al. (2022), and MOPO (Yu et al. 2020) have demonstrated significant potential in this domain. These methods allow policy learn- ing from pre-collected datasets without requiring online interaction. Moreover, offline RL, such as Diffusion-Q1 QL. (Wang al. 2022h), employs generative policy architecture improve expressive capacity.
location and bidding strategies in multi-cham
However, RL-based methods often suffer from training instability arising from bootstrapping and alternating training between critics and actors. Training instability typically deteriorates policy per- formance (Sutton al. 1998). Moreover, training stability is even more critical in auto-bidding, given two domain- -peeiin challenges: the absence of an accurate offline policy evaluation method and the high cost of online policy evaluation in a real-world advertising system (Mou et al., 2022). Therefore, stable convergence to a well nell-performed policy is essential to ensure deployment reliabil- ity and system safety.
A.2 GENERATIVE AUTO-BIDDING BIDDING METHODS
Generative models exhibit strong capabilities for capturing and reproducing underlying data distri- butions across wide range of fields (Kingma & Welling 2022; Goodfellow et al. 2020: Pan al.) 2023 Sihl-Dickstein et al. 2013 Ho et al.) 2020; Vaswani et al.) 2017). They can be effectively incorporated into decision making systems by generating complete trajectories that guide agents toward high-reward behaviors (Zhu et al., [023 Kang et al. 2023 Li et al. 2025). In particular, Decision Transformer (DT) (Chen et al.
2021) reframes RL as a conditional sequence modeling problem and leverages transformer architectures to generate actions conditioned on desired returns, historical states, and actions. AIGB (Guo et al. (0224) extends the generative perspective to the auto- bidding domain by formulating auto-bidding as conditional generative modeling problem. DiffBid generates state trajectory based on the desired return utilizing conditional diffusion model, and then generates actions aligned with the optimized trajectory. These methods achieve superior per- formance in auto- bidd-n and offer distinct advantages over traditional RL methods. They do not rely on the bootstrapping mechanism commonly used in RL, thereby avoiding the instability caused by the deadly triad. Even so, these generative auto-bidding methods still face a performance bottle- neck due to their neglect of fine-gra ain-grrined generation qualitt evaluation and their inability to explore beyond static datasets. In contrast, our method facilitates both reward evaluation and policy search through a learned trajectory evaluator.
B AIGB METHOD DETAILS
AIGB models the sequential decision-making problem via conditional diffusion, enabling effective trajectory generation for auto-bidding scenarios. Specifically, AIGB utilizes the denoising diffusion probabilistic model (DDPM) (Ho et al. 2020) for generation. The forward and reverse processes are modeled as:
(Tr+aal mk), ((T2)7641,8(T)).
(14)
respectively, where g represents the forward noising process while po the reverse denoising process.
17

---

In addition to this standard objective, AIGB also Published as a conference paper at ICLR 2026
Forward Process. In the forward process, the noise is gradually added to the latent variable by Markov chain with pre-defined variance schedule By:
(TTx/T=-1) = NTTs V1 - Tk 11,
(15)
where k E [K] refers to the diffusion step, ти the k-th diffusion step, and To is the original trajectory. A notable property of the forward process is that at arbitrary time-step k can be sampled in closed form as:
1$1: s2,... ; ST)x represents the latent variable in
(TTu/TO)) = (TTi Vāsto, (1 - (x)
(16)
where og = 1 - 3r and Og = II Org. When k + co, approaches standard Gaussian distribution. In particular, AIGB employs cosine noise schedule (Nichol & Dhariwal 202T to control the schedule S3.
Reverse Process. In the reverse process, diffusion models aim remove the added noise on тк and recursively recover To- This process is governed by the conditional model (e(T=-1/Th. ((T),, which parameterized through a noise prediction model (T ((T), k). AIGB adopts a temporal U-Net (Ronneherger et al.] (2055) for the noise prediction model, common choice in diffusion-based decision-making methods (Ajay et al. )023).
B. TRAINING STAGE
The training of the diffusion model is typically formulated as minimizing the mean squared error between the predicted noise co and the true noise applied during the forward diffusion process. Specifically, during each iteration,
randomly sample trajectory from the offline dataset D and pick time step E [Z]. We recursively add the Gaussian noise to the states in T with time steps bigger than & and predict the added noises with (T(Tk, (()), k). where the states between and in Th are set to real history states $1, $2. ...
incorporates a supervised loss that measures the discrepancy between the true actions and the actions predicted by an inverse dynamics model fsr, 8e+1). Overall, the complete training objective of AIGB can be expressed as:
C(0, +) Ex,re |I|e (T v(T), k)]1 + Reo.ol. - for $1+1)113).
(17)
During training, the condition y() is randomly dropped to enhance model robustness. This tech- nique ensures that both the unconditional model CT k) and the conditional model cT(Te, ((T). ks) are effectively trained together.
B.2 INFERENCE STAGE
Starting with Gaussian noise, trajectories are iteratively generated through a series of denoising steps. Specifically, AIGB uses classifier-free guidance strategy (Ho & Salimans 202T) to guide the generation of bidding and extract high-likelihood trajectories in the dataset. During generation, AIGB combines conditional and unconditional score estimates linearly:
E = 60 (Tk, k)) + w (co (Tr. ())),) co (Tk, k)),
(18)
where is the guidance scale that controls the influence of the condition y(T). This formulation effectively steers the trajectory generation toward regions of the data distribution that are most con- sistent with the given condition. The predicted state each step is sampled from po(Tk-) (T,,, (T)):
Th-1 ~ N (T--1/P0 (Tk, (T), A), E. (T))
(19)
Be Ex) and E- = Br. Note with mean and variance defined as ne [T( (Tk; U(T), k) = - (Tx- -í
that the initial noisy trajectory TK ~ (O(0, 1) assigned with history states $ia for the first states to ensure history consistency. This is consistent with the training process. By recursively applying the reverse diffusion process using:
Tke = k)) + V Bez,
(20)
where /O I), we obtain the final denoised trajectory Th from which the next state 6,-1 is derived. Then the action is generated through an inverse dynamics åi so(se, 81+1).
18

---

Published as a conference paper at ICLR 2026
22
4
1
Rx
3
4
5
8
9
10
11
Impression Index
with both al and Figure 6: The impression opportunities within time step and t + 1, where pi is the 1/RO1 of impression i. Without loss of generality, consider two actions a1,t and 02 and let a22 d1,t- The impressions within the shadow area are the impressions won by action 022 but lost by action 01
Impressions only oē
Impressions lost with both of and
C
THEORETICAL PROOFS
C.I PROOF OF THEOREM T
Theorem 1 (Lipschitz Continuous of y(r)... The trajectory quality y(T) is timuous with respect to the Frobenius norm.
Proof. Recall from Section 2.1 that the ot and reward rt under action de between time step
and L + can be written as:
-=
pl and =1
pf
(21)
where pi and of denote the market price and the value of the i-th impression between time step and + Accordingly, the cost ratio é and the normalized reward can be written
& = B Σ
a0
P
1 Σ and F pi
M 1
(22)
Consider two different trajectories and with actions, cost ratios and normalized rewards se- 17" quences {a1,e, Cr,t fi, and {az,e, 02 12 respectively. The trajectory quality gap be- tween 7j and holds that:
(b( v(T2)1
Σ FI,1 - Σ F2. &
Σ FI. - F2.l-
(23)
Consider the reward gap between time step and t + 1, as shown in Fig 6 Without loss of generality, let d2,t 2 01 We have:
Σ
pt
{ 01
F1.
F2.1l
ag
Σ
p
B a2,1
>
2
a1,
Σ
1
02,1
2
a1.
B
21
pt
Rm
<
a221
a1
pz-
(24)
B
19

---

Published as a conference paper at ICLR 2026
Note that the cost ratio gap between time step and t + can be written
1
Σ 1
02
pt
4t 01
if
p
1
Σ
a21
p
2
a1 ppi.
B
(25)
Therefore, combining Eq.
24 and Eq. 25 we have:
Fi.e F2,1
Rm - 22
(26)
We examine the Frobenius norm of the gap between 71 and 72:
01,0
2,0
x
2
č1,
2
x
||7 --2l/ r=
:
T
C1,7-1
x
02
x
Σ (Ōt, - Ō221)
lē1, Ō2.el
(Cauchy-Schwarz Inequality)
(27)
Combining Eq.
23
Eq. 26 and Eq. 27 we can obtain that:
(w(T)) - u(T2)l S Σ FI, - F221l
S Rm Σ 101,1 T2
< TTRm 1
VT Σ l01, o2,el
S VTRm | - T2l1F
(28)
This concludes the proof.
C.2 PROOF OF THEOREM 2
Here, we list two lemmas used in the proof of Theorem 2]
Lemma (Additivity of the Lipschitz). Let fi (x) and fz(2) be two Lipschitz continuous functions with Lipschitz constants L1 > 0 and L2 > 0, respectively. Then (fi(z) + f2(2)) is also Lipschitz continuous function, with Lipschitz constant at most L, + L2.
Proof. Recall the Reverse Triangle Inequality states that Va, b, we have ||a| - |6/l I la b). Then,
dx, y. we have:
1z() (fa + fe(v)ll
s (1)(() + fz(x) fi(e) - fe(v))
$ (x) - (1)) (f((x) f2(v)l
$ (L1 + L2]lz= -
(29)
This concludes the proof.
Lemma 2 Senntorovich-Rubineee Duality Theorem Villani 2021
Let (X, d) be metric space, and let p and q be two probability distributions on X. Let : IR be an fiiiscchtz,uucctinn and W, (p.0) q) denotes the I-Wasserstein distance between p and q. Then we have:
Ba-p (x)
EE Eg f()) < L. Wi(p.g).
(30)
We next give the proof of Theorem 2
20

---

Published as a conference paper at ICLR 2026
Theorem 2 (Evaluator Bias in Planning Performance Bound). Let the upper bound of the evaluator's bias on its training dataset 'D be åD 0. The gap between the planner's score L(0) and its true performance (() can be bounded by:
(J(0) L(0) s 8n + (1 (l
Wi
W (i
Tipschire sensitiviry to
imitaation error on
where W1 denotes the I-Wasserstein distance.
Proof. The evaluator bias in the planner's performance can be written as:
[J(0) L(0) = (Br-no(rio))u((T)) - (T))) s Br-no(riv)) (U(T) - (T(T)
(31)
ff(r)
Let ((T) - [((T) - [T] be the evaluator bias in trajectory т. From Theorem and Lemma we know that f() is (1 + k) /T -Lipschitz continuous function. Then, we have:
(((0) (0)) < (x>po(rlu)))(T))
Ey-ppll)
E Er-polrlu Erupo(r f) + (e-po(rio)((t)
-uy-polo) Er Propotifir
Brano(rior)((T)) Er-po(r())(())
=ôp + + Ey)po(u)
((1+k)TTR, Wi ( (Lemma
+ E (r))p(r)
E E
- ((((7>)))-rrooooo( (r
2 5(I W..tyy/TT....ttctooooooooc. (lomm
<60 + () &y (
y
+ (1 + k),TR, (W
(32)
Therefore, we have:
(J(0) L(O) s 8D + +1++ k).TR (1 ( Wi(pe(Tle"), (..)))//etll)(o-.((....))))).))- PD
(1
(33)
This concludes the proof.
C.3 PROOF OF EQ.
6
We give the proof of Eq. 6 as follows. Denote LiPw, (po(Tly)) as the planner's Lipschitz constant with respect to y regarding the Wasserstein distance W,, we have:
Ey<pp(") (Wa (Poo po((l))) S Lipw; (po (n((Tly) Eypp [((1 + e)gm y)]
Lipw; (po (po(Tl)) ( mm
pD(v)[(1 +e)ym y)dy
Lipw; (po(Tly)) f'h Mm
PD(v)[(1 +e)ynddy
= (1 + e) (Po(Tly)),
(34)
where we leverage the non-negativity property of the condition 2 0, Vy E D. This completes the proof.
21

---

Published as a conference paper at ICLR 2026
C.4 PROOF OF EQ.
Lemma 3 (Pinsker's Inequality (Tsybakov] 2008)). Let P and Q be two probability measures de- fined the same measurable space, and assume that P is absolutely continuous with respect to Q iee, P & Q. Then the total variation distance between P and Q is bounded above by the KL. divergence from P Q as follows:
P QIIrv S V V2DKUPPPI).
(35)
Lemma (Wasserstein- Variation Inequality on Bounded Metric Spaces (Villani al. 2008
Let (Z,d) d) be metric space with diameter diam{ 2) & sup d(21, 22). Let P and be two probability measures on 2. Then the 1- I-Wasserstein distance between and Q satisfies:
W (P(.QQ Q) S diam [2) Qlivv.
(36)
We give the proof of Eq. 7 as follows. Equipped with the above two lemmas, have:
Wi (P(o(T))) < diam(T) (TPo - pn(rlv)
I (diam()) (TV (xL(po(((w)lle((tto))
(37)
where is the trajectory space. Note that due the budget constraint Σ. c I B sum of the cost ratio satisfies Σ, C S 1. The trajectory space can be expressed as:
we have the
T- f-[[1..0 11. Cu. x), 12, 12, E1 ...... [T. "...
àa 2 Vt, and Σ a aass
(38)
We next prove that the diameter of the trajectory space, tiam((), be bounded by constant. Specifically, the diameter only depends on the largest possible distance between the cost ratio se- quences in two trajectories since:
diam(T)
sup
| T2/LF
T.T.T
01,
x
2
x
02,1
sup
T.
T
C1.-1
T
C2.T-1
x.
sup
(ā1. - E2,))
(39)
m.M.T
For convenience, we let e; A ",,00,,,... result, (,, 2, can be written as:
; E [1,2]. Then, the key part in the above
Σ (0 2
Σ (c 20,, + (q.
cc 112 cc 1/ 2(c1, C2)
< | 1
cc 12
(40)
where (CI, C2) 2 As Eie 2 and E, Et. S we have S Ct.t S 1. Therefore, holds that:
||c,112 11
Σ 2 E 1.
(41)
Combining Eq. 40 and Eq. 41 have:
Σ (cr. Ttle-ōzue 22.)3 / ////1113 + ||c2212 S 22.
(42)
As explained in Footnote the budget constraint is guaranteed to be satisfied in real-world advertising systems thanks an automatic suspension mechanism that halts bidding once the budget exhausted.
22

---

Published as a conference paper at ICLR 2026
Therefore, we have diam( v2. According to Eq. 37 we have:
Wi (po(Tll)) v)) S v (Dw.
(43)
Recall that we impose the KL-constraint as:
(y)r (pw(lpp(ty)lll
s 8x,
(44)
Taking the expectation over y ~ pD(y) on both sides of Eq. 43 we have:
By-pp(1) (iooo - Ey-rD())
po
(Jensen Inequality)
$ dx
(KL constraint Eq 44
(45)
This completes the proof.
C.5 PROOF OF THEOREM 3
(rly)) be the expected distance between the optimal trajectory distribution and the trajectory distribution of the offline dataset D. The true performance gap between the optimal parameter O* and the solution to Eq. 8 is bounded by:
Theorem 3 SSub-optimality Gap Bound). Let dyg - Eny-pols)
[VoN + Vox + (1 + e)umbbp
J(0") - ((0) - 26 +
(1 +1
2k) TTR.
(46)
Proof.
The sub-optimality gap can be expressed as follows:
J(0)) - J(0) = (J(0") - L()))) + (L(O*) - L()) + (L(Ō) - (())
< - L(0') [ L(0) L
(47)
evaluator in Pa*
score gop
evaluator bias Ps
We examine the above three terms accordingly.
(1) Evaluator Bias in pæ.. Denote the evaluator bias on trajectory as f(T) & (M((T) - (T)
Following the derivation process Eq. 32 we have:
[J(0") I(0") Ey-op(() E
Er + Ey-po()) E (r(ve) - - (r-
Sō (1 k))kk) R) [W1
(48)
where W1 (pg. (Tly"), ) (T())) denotes the probability distribution distance between the optimal planner and the offline dataset. Based on the derivation in Appendix C.4 we have:
Ey-pp(u) [W (po- (Tly"), (D( lu)) V EEy-polo (po((p(TT))) (1y)))lo (rly*))]
(49)
Let dM A Byy Dxz (pD(T llo- (rly")) be the distance between the optimal trajectory distribution and the offline dataset trajectory distribution. We have:
(J(0)) L(0')) S 6n + (1 +kV+)RRR,,, Von.
(50)
(2) Score Gap. Recall that the trained evaluator (T( is Lipschitz continuous function with the Lipschitz constraint design. With Lemma 2 we have:
L(0") L(θ)
|E~pp- (7 (0 ((rr))-rr-pptee (r
kv T R,, W, (po (711").
Rv TTRRR Ey-py-pp
W Wi (Tly") (o(l) + (et(o((tl))
< kv kVTRR Von +
+1 (1+ c) Tip vox
(51)
23

---

Published as a conference paper at ICLR 2026
Upper Round:
Lower Bound:
Higher by 6
I(6") (2 R)
max Ex-pb(t)]
True Performance
sut,spp())) O2 02.5)
(0) Range
Higher Performance
Figure 7: Performance range of the planner with respect to hyper-paramen parameters őz, Lp and c. There is a trade-off in the selection of nyper-parameters: larger dg. Teps results in smaller lower bound but a higher upper bound.
where leverage the fact that 0 is solution to Eq. straint, and we leverage the results in Eq. 6 Eq.
8
which satisfies the KL and Lipschitz
and Fq. 49
(3) Evaluator Bias in Pg- Since 0 satisfies the KL and Lipschitz constraint in Eq. we can use the results in Theorem 2 Eq. 6 and Eq. to obtain:
(()) L(0) 6D (1 k) (1 + e)ym Lp + oxl:
(52)
Overall, combining the results in Eq. 50 Eq. 51 and Eq. 52 we have:
J(0") J(Ø) < 26D + (1 + 2k) TR V 68 + Vox + (1+ c)
(53)
This concludes the proof.
C.6 PROOF OF SCORE GRADIENT
The probability of the trajectory generated by the Causal Transformer can be decomposed into:
(s = (s (81181111111)
(54)
Then, have:
VoL(O) Vo
Va
(-.) .ds
/ po(sually*)
V Vepe(S11T1V (v"))
30 (o(r)ds ... (o(r)ds)....ds)
po(su:rlys 1x))
V Vo po (ppr=po
(0)
Vo log (1p )o)(()
Vo Vologpdsslssu--1///661
(55)
D THEORETICAL PERFORMANCE RANGE TRADEOFF DISCUSSION
We note that Theorem actually gives the lower bound of the planner's true performance (0),.....
J(Ø") 26 (1 + 2 2 VR Vou + Vox + (1 c) 5 <(())
(56)
Ln.e)
24

---

Published as a conference paper at ICLR 2026
where we denote that gap term f as function of prper-parameter's 5K Lp and that adjust. As f(ok. Lp, <) is monotonically increasing with respect to 6x, Lp and c, smaller values these terms result in a higher lower bound of J(0). However, as illustrated in Fig. 1 these three terms also determine the planner's exploration range, denoted as D. (ōx, Lp. e). Specifically, higher values of 62, lpp indicate larger exploration range, which thereby leads to a higher performance upper bound of the planner, i.e.,
((0) max (=xxp())((())) s.t., supp(p(r)) S (..0k:Lp..) p(r)
(57)
This introduces trade-off in the selection of syper-parametent As illustrated in Fig. higher 5R, Lp. results in smaller lower bound but a higher upper bound. To this end, we conduct hyper- parameter tuning and give hyper-parameter determination in Appendix F.8
E AIGB-PEARI ALGORITHM SUMMARY
Algorithm summarizes the training process of AIGB-Pearl. Specifically, compute the Lipschitz constants y(T) and Pp (Tly) using the offline dataset D and, accordingly, determine the Lipschitz constraints for the evaluator and the planner, respectively. Then, sequentially perform evaluator learning, planner pretraining, and L.L-LLpschhizzccnsttriiee
score maximization for the planner. The
development of AIGB-Pearl is supported by ROLL (Wang al. 2025).
E.1 ADDITIONAL DESIGNS FOR EVALUATOR ACCURACY ENHANCEMENT
To further enhance the reliability of the trajectory evaluator, we design two additional techniques for the evaluator learning. Specifically, as described in the following, (i) integrate LLM embedding into its input feature for better representational capacity; and (ii) employ the pair-wise loss for better score estimation accuracy. Fig. illustrates the complete evaluator learning losses. The effectiveness of these two methods is studied in Appendix F.6
E.1.1 LLM EMBEDDING ENHANCEMENT
Motivated by the success integrating user-specific features into recommendation systems (Chen et al. 2015), incorporate advertiser-speciff features into the trajectory evaluator to enhance its representational capacity and improve scoring accuracy. Note that some advertiser-specific features textual in nature-such incorporate directly into the vectorized trajectory т. To address this, we construct prompl con- taining all such textual attributes and employ a pre-trained large language model (LLM), QWen2.5- I.st-Int with general world knowledge to extract T representation, which we refer to as the LLM embedding. Specifically, we use the output of the last hidden layer as the LLM embedding. This embedding is then used as an additional positional encoding in the evaluator. The prompt template given below:
product titles, categories, and reviews-and
therefore difficult to
LLM Prompt Template. am an [advertising platform] advertiser, operating the [brand name] brand in the category of [category name), classified as a [advertiser tier] tier advertiser. have a product titled [product name] currently running in advertising campaigns. This product belongs to the leaf category of Ileaf category], priced [product price]. and is positioned in the [price range). Its price ranks within the top pprice ranking in the leaf category] % in leaf category.
Historical Average Performance: The product generates an average of [average daily tronsactions] daily transactions, with a GMV of (average daily GMVJ, driven by advertising. It receives an average laverage duily impressions] daily impressions from search and recommendation traffic, [elick num- bers] clicks, [average daily BuyCnt] BuyCnt, and GMV of [GMV]. It ranks within the top [sales volume ranking in leaf categoryl % in sales volume in the leaf category, with average transaction value of (average transaction value].
Historical Time-based Average Performance: This product has undergone continuous exposure to ad- vertising for number of advertising days days. The average hourly advertising spend distribution per day (from 0:00 to 24:00) is historical spend distribution sequence. The average GMV distribution across its calegory during this period is historical GMV distribution sequence. The average daily advertising budget daily advertising hudget.
25

---

Published as a conference paper at ICLR 2026
Pale-wise Polnt-wise loss Lipschitx Penalty
Patr-wise Loss
Trajectory Evaluator
Trajectury Evalustor
Trajectory Bvaluator
1
68
Fv
Trojecttryy S
y 3000
Trajectory
7 LLM Embedding
Lipschitz Penalty
[b66))-00x)) -TRul -ullll sall)
9ptn2
Palta)
Large Language Model (Qwen2.5
Trajectory Evaluator
Trajectory Evaluator
Prompt with Advertiser-seett spectfic Peatures:
+
Taoban advertiser. and selling
Trajectory fz
Trojectory 12
Figure 8: AIGB-Pearl Evaluator Learning. The evaluator loss is composed three parts, in- cluding the point-wise loss, the pair-wise loss, and the Lipschitz penalty. The evaluator's 's input representation is augmented with an LLM embedding to incorporate semantic information from the advertiser's textual features.
E. PAIR-WISS WISE LOSS
Unlike human feedback scores used in LLM post-training (Christiano et al. 2017 which suf- fer from subjective biases in their absolute valuations-the trajectory quality y(T) has real physical meaning and is comparable among different trajectories. Therefore, can adopt hybrid point- wise and wair-wise loss for the score to capture the absolute value of y(T) and their relative prefer- ence, respectively. This approach has demonstrated superior performance recommender systems (Cao et al. 2007 Lei Lei et al. 2017 Wang al. 2022a). Specifically, the pair-wise loss function can be expressed as:
B (rwsn)-D,
log (ū,((T)) (T(())))
(58)
where the pair-wise loss is implemented by the typical Bradley-Terry (BT) function (Bradley & Terry 1952). Here, Dp {(Twr 7, ((TW), denotes pair-wise dataset extracted from the offline dataset D, where denotes the trajectory with higher trajectory quality, i.c., y(Tw) 2 ((7).
E. OVERALL EVALUATOR Loss
Combining the previous techniques, the overall evaluator loss is:
le(0) = E,~D (Mo(T) x())) +B, E (TTssn)-D, [ogo(Go(Tw)
point-wise loss
pair-wisc loss
+ B1 Ery, [(o(T)) - VTRm I| - 7211
(59)
Lipschite penalty
where Bg 0 is hyper-parameter.
F ADDITIONAL EXPERIMENTS
F.1 SIMULATED EXPERIMENT SETTINGS
Specifically, we consider the bidding process in day, where the bidding episode is divided into 96 time steps. Thus, the duration
We include the detailed simulated experiment settings in Table 6
26

---

Published as a conference paper at ICLR 2026
Algorithm 1: AIGB-Pearl (Planning with EvaluAtor via RI)
Input
Offline dataset D, desired condition y", hyper-parameters B1, A8, A3.
Output Optimizze 0 and (b
Initialization:
randomly initialized planner parameter 0, trajectory evaluator parameters & // Determining the Lipschitz Value
Calculate the Lipschitz value of (()) and (D()))) using the offline dataset D.
Set the Lipschitz constraint value Lg for the evaluator and Ly for the planner to be bigger than the Lipschitz value of y(T) and pT(lly), respectively.
// Training the trajectory evaluator
while not converged do Update by minimizing Eq. 11 end
// Training the generative planner
Warm start with pretrained planner po;
while converged do
Generate bidding trajectories 5 ~ po((Ty");;
Score generated trajectories with frozen Φ: is (T);
Update 0 by maximizing Eq. 12
end
Table 6: Settings of the simulated experiments.
Parameters
Values
Number of advertisers
30
Time steps in an episode,
96
Minimum number of impressions within time step
50
Maximum number of impressions within time step
300
Minimum budget
1000 Yuan
Maximum budget
4000 Yuan
Value of impressions
Minimum bid price, min
0 Yuan
Maximum bid price, max av;)
Yuan
Maximum market price, PM
10 Yuan
between two adjacent time steps and L + 1 is 15 minutes. The number impression opportunities between time steps / and / + 1 fluctuates from 100 to 500. The minimum and maximum budgets of advertisers are 1000 Yuan and 4000 Yuan, respectively. The upper bound of the bid price is 10 Yuan, and the values of impressions are positive.
Hardware Resource. The simulated experiments are conducted based on an NVIDIA T4 Tensor Core GPU. We use 10 CPUs and 200G memory.
F.2 REAL-WORLD EXPERIMENT SETTINGS
We include the detailed real-world experiment settings in Table 7 Specifically, we consider the bidding process day, where the bidding episode divided into 96 time steps. Thus, the duration between two adjacent time steps and t + 1 is 15 minutes. The number of impression opportunities between time steps and t + fluctuates from 100 to 2,500. The minimum and maximum budgets of advertisers are 50 Yuan and 10,000 Yuan, respectively. The upper bound of the bid price is 25 Yuan, and the values of impressions are positive.
Hardware Resource. The training process in the real-world experiments is conducted using 10 NVIDIA T4 Tensor Core GPUs in a distributed manner. For each distributional worker, we use 10 CPUs and 200 GB of memory.
27

---

Published as a conference paper at ICLR 2026
Table 7: Settings of the real-world experiments.
Parameters
Values
Number of advertisers
6,000
Time steps in an episode, T
96
Minimum number of impressions within time step
100
Maximum number of impressions within a time step
2,500
Minimum budget
50 Yuan
Maximum budget
10,000 Yuan
Value of impressions
Minimum bid price, min (aez]
0 Yuan
Maximum bid price, max (av;)
25 Yuan
Maximum market price, PM
25 Yuan
95
With
SMAPE
75
-3 4pt
35
40k
80k
120k
160k
Training Steps
Figure 9: The SMAPE training curves of the trajectory evaluator with and without LLM Embed- dings. Incorporating L.M embeddings helps to achieve faster convergence and improved absolute accuracy.
F.3 REAL-WORLD EXPERIMENTS ON TARGETROAS BIDDING PROBLEM
In addition to the budget-constraineed auto-bidding problem, also apply the proposed AIGB-Pear algorithm to a more challenging type of auto-bidding problem, named TargetROAS, with an extra ROI (Return Investment) constraint. We evaluate method in a real-world experiment TaoBao involving 300k advertisers over 22 days. The offline dataset comprises 16 million trajecto- ries of 800k advertisers. The results are given in Table g AIGB-Pearl achieves +5.1 % improve- ment in GMV compared to the SOTA auto-bidding method, DiffBid, demonstrating its effectiveness in managing more complex and realistic constraints.
F.4 PATHOLOGICAL TRAJECTORY BEHAVIOR EXPLANATION
In industrial practice, stable and effective metrics have been developed to evaluate pathological be- haviors. For the case of the budget-constrained auto-bidding problem with bidding cycles structured as 24-hour episodes (T = 96 time steps), the following three key metrics are commonly used to identify pathological behaviors:
Excessive budget consumption: there exists time step such that the cost between time step and f exceeds 10% of the budget B:
Forward- (or Backward-) trending pacing: the cost between time step 1 and 24 (or between time step T 24 and T)) exceeds 40% (or 40%) the budget B;
under-utilization of available budgets: the total cost over the bidding episode is lower than 90% of the budget B.
28

---

Published as a conference paper at ICLR 2026
Table 8: Overall performance of TargeiROAS in real-world A/B test, involving 300k advertisers over 22 days.
Methods
GMV
BuyCnt
ROI
Cost
DiffBid
779,642,891 819,550,812
11,519,082
4.68
166,544,918
AIGB-Pearl (ours)
11,886,501
4.70
174,234,673
+5.1 %
+3.2%
+0.5%
+4.6%
Offline RL with Bootstrapping
AIGB-Pear
Comulative Rowanb
0.3
Cumulative Rewards
0.8
0.7
04
soat??
seed3
seall
9.6-
1000
1500
Steps
Steps
Figure 10: Learning curves of cumulative rewards between offline RL with bootstrapping method and AIGB-Pera under 10 seeds.
F.5 TRAINING STABILITY
We present additional comparisons between the training curves of the offline RL with bootstrapping and those of AIGB-Pearl in Fig. I0 Fig.
Fig. 12. and Fig. concerning:
Cumulative Rewards: the main performance index of the considered auto-bidding prob- lem;
Online Rate: the ratio between the bidding period before the budget runs out and the total bidding period. A larger Online Rate indicates better performance.
Bad Case Rate: the ratio between the number of "bad" trajectories and the total number of generated trajectories. A lower Bad Case Rate indicates a better performance.
Cost Rate: the ratio between the cost and the budget. A larger cost rate indicates a better performance.
It can be observed that the offline RI. method tends to exhibit significant instability throughout training, with high variance across different seeds. In contrast, AIGB-Pest achieves much smoother and more consistent learning progress, demonstrating the improved training stability.
F.6 EVALUATOR ACCURACY
Accuracy Metric. We evaluate the accuracy of the evaluator along two dimensions, including the absolute accuracy, reflecting how close the predicted scores are ground truth qualities, and the order accuracy, reflecting the correctness of relative rankings between trajectory pairs. Specifically, we use the symmetric mean absolute percentage error, SMAPE, as the metric for absolute accuracy and the AUC, defined the ratio of correctly predicted ordinal pairs to the total number of pairs, as the metric for the order accuracy. The SMAPE ranges from 0% to 200%, and the AUC ranges from 0% to 100%. Lower SMAPE and larger AUC indicate better evaluator accuracy.
In the following, we investigate the effectiveness using LLM embeddings and a pairwise loss for evaluator learning.
LLM Embedding Effectiveness. We examine the accuracy of the trajectory evaluator without using LLM embeddings, and the results reported in the lines of "w/o LLM" in Table and Table 10, can observed that LLM embeddings can improve both absolute and order accuracy. Fig. 9 compares the training progress with and without L.M embeddings in terms of the SMAPE
29

---

Published as a conference paper at ICLR 2026
Offline RL with Bootstrapping
AIGB-Pearl
Oeliee Rae
seed:
see2
scad.7
soudh
seats
10at
500
1000
1500
2000
500
1000
1500
2000
Figure 11: Learning curves online rate between offline RL with bootstrapping method and AIGB- Peral under 10 seeds.
Offline RL with Bootstrapping
AIGB-Peart
0.4
seah1
soud7
seed2
seel.7
0,3
Bad Cass Rate
Und Case Rote
sood-10 10
seed:10
0.1
500
1500
1000
Steps
Stops
Figure 12: Learning curves of bad case rate between offline RL with bootstrapping method and AIGB-Peral under 10 seeds.
metric. As illustrated, the evaluator incorporating L.LM embeddings converges faster and achieves lower SMAPE than the one without LLM embeddings. The performance gain stems from LIM embeddings" ability to encode high-level semantic information, thereby facilitating a more nuanced understanding of dependencies among sequential bidding states.
Hybrid Point-wise and Pair-wise Losses Effectiveness. Table Table present the SMAPE and AUC of the trajectory evaluator when trained with point-wise loss only, pair-wise loss only, and a combination of both. can also be seen that using only pairwise loss yields significantly worse SMAPE performance, despite some improvement in AUC. This suggests that while pairwise loss can enhance ranking consistency, it falls short of providing accurate absolute-value predictions. When both point-wise and pair-wise losses used together, the evaluator achieves lower SMAPE and higher AUC. This indicates that combining these two loss types not only improves absolute accuracy but also enhances order accuracy in trajectory evaluation.
F.7 EMPIRICAL PERFORMANCE WITH GENERAL OFFLINE DATA DISTRIBUTIONS
Note that in many real-world auto-bidding systems, including the one considered the paper, due to operational safety constraints, the online-deployed bidding policy is typically single fixed model, and the offline dataset is collected exclusively from this single policy over multiple days, where an advertiser contributes single trajectory per day. For example, in the considered auto-bidding system, the online-deployed baseline policy is a conditional generative model that, given a given ad- vertiser and identical conditions, generates identical trajectory plans each day. The variation across
Table 9: SMAPE results from ablation experi- ments on the trajectory evaluator.
Table 10: AUC results from ablation experi- ments on the trajectory evaluator.
SMAPE
Point-wise
Pair-wise
Both
AUC
Point-wise
Pair-wise
Both
w/o LLM
43.55%
196.38%
35.00%
w/o LLM
61.57% 65.00%
64.91%
70.91% 75.10%
with LLM
38.20%
196.00%
31.60%
with LLM
71.30%
30

---

Published as a conference paper at ICLR 2026
Offline RL with Bootstrapping
AIGB-Peard
1.0
0.5
0.9
0.6
Cont Kate
Celt
0.8
0.4
seats soad. soadi
seed:1
seef
0.2
0.7-
swahs
seald
seat 10
500
1500
2000
500
1000
1500
2000
Figure 13: Learning curves of cost rate between offline RL with bootstrapping method and AIGB- Pearl under 10 seeds.
different trajectories of the same advertiser in the offline dataset is solely due to stochastic environ- mental factors (e.g., traffic fluctuations). Since these exogenous perturbations are typically the sum of many independent impression-level sources of noise, the resulting trajectory deviations can be reasonably approximated as a Gaussian distribution.
To demonstrate the broad applicability the proposed algorithm, we evaluate its performance in settings where multiple policies used for data collection. Specifically, collect trajectories the simulated environment using nine distinct bidding policies, thereby constructing an offline dataset with multi-mode distribution that violates the Gaussian distribution. The empirical results are presented Table II
Table 11: Empirical performance with multiple data-collection policies.
Methods
GMV
ROI
Cost
DiffBid
548.5
5.00
109.4
AIGB-Pearl (ours)
575.7
5.04
114.6
A
+4.9%
+0.7%
+4.2%
It can be observed that AIGB-Pearl still outperforms AIGB by 4.9%, indicating that its performance robust the offline dataset's specific distribution.
F.8 HYPER-PARAMETER TUNING
We conduct sensitivity experiments with respect the hyper-parameter dg. For hyper- parameee Lp, we leverage the lower bound given in Eq. 10 in the main experiments. For the hyper-parameter the same empirical value of 5% AIGB in the main experiments, which is typically set based on operational requirements.
Specifically, the planner loss given in Eq. 12 32 is the penalty factor corresponding to the KL constraint. Actually, we control the KL divergence dR by tuning 32. Table. 12 gives the hyper- parameter tuning results. It be observed that, as long ôş is neither too large (in which the algorithm degenerates into AIGB with pure demonstration likelihood maximization term) nor too small (c.g., 82 O, which completely removes the demonstration likelihood maximization), the proposed method consistently outperforms the original AIGB. This also validates the discussion on performance bounds in Appendix [D] which demonstrates that a moderate &ş balances the lower and upper bounds, thereby yielding optimal performance.
Table 12: Hyper-parameter tuning with respect to KI. constraint dg.
8g Setting
0 (no KL)
0.1
0.5
1.0
5
10
(AIGB, only KL)
& values
0.85
0.51
0.50
0.49
0.43
0.39
0.36
0.32
GMV
548.5
563.8
571.3
574.2
573.9
570.7
567.6
556.3
31

---

Published as a conference paper at ICLR 2026
Table 13: Empirical performance in the simulated experiments with first-price auction.
Methods
GMV
ROI
Cost
DiffBid
1,546
5.13
301
AIGB-Pearl (ours)
1,611
5.18
311
A
+4.2%
+1.0%
+3.3%
G EXTENDING AIGB-PEARL TO FIRST-PRICE AUCTIONS
We note that the proposed method remains effective in first-price auctions with proper adaptation. Specifically, unlike second-price auctions where the optimal bid for impression takes the form bid; = av,, in first-price auctions, the optimal bid for impression is given by bid; = min (av;, Pt which typically involves an extra bid shading method to predict the winning price (Gligorijevic al., 2020 Wu et al.) 2013). Equipped with an off- aff-the-shell bid shading method (whose design is beyond the scope of this work), the auto-bidding problem in first-price auction remains an offline sequential decision problem, i.e., making decisions over an a-sequence, to which the proposed method applies directly.
To validate the effectiveness of the proposed method under first-price auctions, we additionally conduct a simulated first-price auction experiment against the state-of-the-art AIGB method, where all methods are equipped with the same bid shading method. The results are presented in Table I3 demonstrating the effectiveness of our proposed method.
H EXTENDING AIGB-PEARL. TO ONLINE SETTING DISCUSSION
We note that AIGB-Peard can be extended to online settings when equipped with a safe online exploration policy. Specifically, due safety constraints, the auto auto-bidding policy during training cannot interact directly with the live advertising system; only safe exploration policies are permitted collect data online (Mou al. 2022 Consequently, online auto-bidding framework typically involves two parts:
a safe online exploration policy, which is a well-established component in existing work (Mou et al. 2022) and beyond the scope of this paper;
an offline policy training method that leverages the data collected online.
AIGB-Pearl can be directly applied as the offline policy training method within the online framework without modification. In practice, due the safety and stability concerns, many industrial auto- bidding systems adopt an offline optimization paradigm. For this practical reason, we focus on the offline setting in this work.
I
LLM USAGE
The authors have used Large Language Models (LLMs) exclusively for grammar checking and lex- ical refinement during the writing process. No LLM-generated content, data analysis, or substantive contributions the research methodology. results, or conclusions are involved in this work.
ACKNOWLEDGMENTS
This work supported by Alibaba Group through the Alibaba Innovative Research Program.
32


Reviews:
[{"review_id": "5pHHIVrFr7", "reviewer": "Reviewer_3d4u", "summary": "The paper introduces AIGB-Pearl, a novel method designed to enhance AI-Generated Bidding (AIGB) by overcoming its limitation of imitating trajectories from a static offline dataset. AIGB-Pearl integrates generative planning with policy optimization by first constructing a trajectory evaluator through supervised learning to provides the necessary reward signal to guide the generative model. To ensure reliable utilization of this evaluator and to mitigate the Out-of-Distribution (OOD) issue, the method designs and enforces a provably sound KL-Lipschitz-constrained score maximization objective for the generative planner. AIGB-Pearl achieves greater training stability and demonstrates SOTA performance in extensive simulated and real-world advertising systems, consistently achieving improvement in GMV and other metrics compared to prior SOTA AIGB method", "questions": "1. The effectiveness of AIGB-Pearl hinges on the reliability of the trajectory evaluator. The theoretical analysis bounds the gap between the planner's score and its true performance, where one of the core terms is the evaluator's bias on its training dataset. Could the authors provide the measured value of the evaluator's bias in both the simulated and real-world experiments?\n\n2. Related to weakness 2, Regarding “this assumption could be satisfied through strategic adjustment of the trajectory sampling probability per condition y in the offline dataset D”, Could the authors detail the specific nature and extent of this strategic adjustment applied to the offline dataset (e.g., which trajectories or conditions were adjusted)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty. AIGB-Pearl is the first work that integrates generative planning and policy optimization to address the performance bottleneck of existing AIGB methods. The integration of offline RL methods into the AIGB is important to guide the policy training and exploration of new trajectories\n2. Technical Contribution. The paper provides provable theoretical guarantees to ensure the reliability and safety of the policy optimization, which is critical when exploring beyond the static offline dataset. The core innovation is the design of a provably sound KL-Lipschitz-constrained score maximization objective that help mitigates the OOD problem\n3. Evaluation. In both the simulation and the real-world online A/B test, the methods achieve SOTA performance in the auto-bidding problem, out-performing the strong benchmark AIGB.", "weaknesses": "1. The paper provides details on the overall dataset sizes: the simulated experiment uses 5k trajectories generated by 20 advertisers, while the real-world experiment uses 200k trajectories from 10k advertisers. However, it doesn’t explicitly state how many of these trajectories are used to train the evaluator model, and 5k/200k appears relatively small for training a stable evaluator capable of mitigating OOD issues.\n\n2. The paper states and assumes that the offline data distribution must satisfy the Logarithmic Sobolev Inequality (LSI) with a positive constant ρ, and this assumption could be satisfied through strategic adjustment of the trajectory sampling probability per condition y in the offline dataset D, the need to potentially alter the raw offline dataset distribution to ensure the theoretical bounds hold represents a limiting constraint or assumption on the data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3nRGDG4n2", "reviewer": "Reviewer_wJct", "summary": "This paper proposes a novel method (AIGB-Pearl) to enhance generative auto-bidding (AIGB) by integrating\nreinforcement learning (RL) into a trajectory-based generative modeling framework. The key idea is to\ntrain a trajectory evaluator to score the quality of generated trajectories and use this score to guide the generative\nplanner to find higher-performance trajectories beyond the offline dataset, while ensuring safety and\ngeneralization via KL and Lipschitz constraints. The method successfully leads to more stable training than\ntraditional offline RL, and is supported by theoretical guarantees on sub-optimality and generalization. The\nauthors conduct extensive experiments in simulated and real-world advertising environments to demonstrate\nstate-of-the-art performance.", "questions": "1. The paper considers the second-price bidding problem. How would the core formulation need to be adapted\nfor a first-price auction environment? Will the method still be effective under first-price settings?\n\n2. The theoretical analysis assumes the offline dataset satisfies LSI. What is the performance of AIGBPearl\nif the assumption does not hold? Are there any ways to relax the assumption?\n\n3. The paper considers offline RL baselines. Can the framework be extended to online cases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality: The integration of RL into a generative auto-bidding framework is innovative and addresses\nthe inability to explore beyond the offline dataset of existing AIGB methods. The introduction of a\ntrajectory evaluator and the use of KL-Lipschitz constraints to ensure safe exploration are also novel\ncontributions.\n\n2. Theoretical Rigor: The paper provides a thorough theoretical analysis, including bounds on evaluator\nbias, sub-optimality and generalization. The use of Wasserstein distance and synchronous coupling for\nLipschitz regularization is well-motivated and technically sound.\n\n3. Significance: The method is deployed in a real-world advertising platform and shows significant improvements\ncompared to previous methods, indicating high practical relevance.\n\n4. Ablation Analysis: Ablation studies clearly demonstrate the importance of both KL and Lipschitz\nconstraints. The analysis enhances the rigor and completeness of the article’s theoretical framework.\n\n5. Experimental Validation: Comprehensive and detailed experiments are conducted in both simulated\nand real-world environments with large-scale datasets and multiple budget levels, demonstrating the\nsuperior performance of the algorithm.", "weaknesses": "Theoretical Assumptions: Some theoretical results in this paper rely on strong Assumption 1, which can be hard to verify on real-world dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e1euXzRFWQ", "reviewer": "Reviewer_6mYT", "summary": "The paper proposes AIGB-Pearl (Planning with EvaluAtor via RL), a method that augments generative auto-bidding (AIGB) with a learned trajectory evaluator and a KL–Lipschitz constrained score-maximization procedure to enable safe exploration beyond an offline dataset. The work combines theoretical analysis (evaluator bias bounds and a sub-optimality gap under KL/Lipschitz constraints), a practical planner/evaluator architecture (synchronous coupling, Lipschitz regularization), and extensive simulated and real-world A/B experiments that report consistent GMV/ROI improvements over strong baselines.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Integrating an evaluator into a generative planner and constraining planning by KL + Lipschitz criteria to control evaluator bias is a principled and timely idea for offline auto-bidding.\n2. The paper gives provable bounds (evaluator bias → performance gap; sub-optimality bound under the proposed constraints) that make the method more convincing.\n3. Results include both simulated and large real-world A/B tests (TaoBao), showing consistent gains over state-of-the-art baselines and sensible ablations.\n4. The synchronous coupling and the proposed surrogate for the Wasserstein term are well-motivated and seem implementable in practice.", "weaknesses": "1. In the problem statement the intrinsic impression value is introduced as v_i>0 (Section 2 / preliminaries). Later, in the experimental settings (Tables 5/6) the “value of impressions” is reported as ranging from 0 to 1. This inconsistency leaves readers unclear whether the theory (which uses v_i>0 and constants like R_m = max_i v_i/p_i) assumes arbitrary positive values or implicitly assumes normalized values in [0,1]. It also affects interpretability of constants such as R_m` Lipschitz constants that depend on reward magnitudes, and the hyperparameter choices.\n2. In the simulated-experiment paragraph the manuscript text says: “The maximum and minimum budgets of advertisers are 1000 Yuan and 4000 Yuan, respectively.” This phrasing is inverted relative to Table 5, which lists Minimum budget = 1000 and Maximum budget = 4000. In the real-world experiment paragraph a similar inversion appears: “The maximum and minimum budgets of advertisers are 50 Yuan and 10,000 Yuan, respectively.” This contradicts Table 6 (Minimum = 50, Maximum = 10,000). Such directional swaps are minor editorial errors but can confuse reproducibility and readers’ understanding of budget regimes used in experiments. Given that the budget ranges are central to the task formulation, they should be unambiguous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jm40pB29b", "reviewer": "Reviewer_29sV", "summary": "This paper studies how to integrate an offline reinforcement learning (offline RL) approach and conditional generative planning for the auto-bidding task of commercial advertisement. The proposed approach is akin to a conservative model-based RL method, where the evaluator is first trained in a pessimistic way with a Lipschitz constraint and then, controller is optimized to maximize the evaluator's score under the KL and Lipschitz constraints. The theoretical analysis is provided to bound the sub-optimality, and the experiment demonstrates that the proposed method works better than existing offline RL baselines and generative planning baselines.", "questions": "- What is the sensitivity of the model like regarding the constraints (i.e., $\\delta_K$ and $L_p$)? How are these values determined in the experiment, and how much effort is needed to adjust these hyperparameters?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The motivation for combining conditional generative planning and optimization (score maximization) is well-explained.\n\n- The proposed approach appears reasonable given the motivation of maximizing the score as an RL problem. In experiments, the proposed method is compared to offline RL baselines and works better than the compared method. \n\n- A good ablation study is provided. Especially, qualitative analysis showing how the trajectory of the proposed method differs between w/ and w/o KL explains why Lipschitz constraints are not sufficient even under the Lipschitz condition, which answers one of the questions I had.", "weaknesses": "- I wonder where the conditional generative planning component is in the proposed method. While the proposed method works better than both traditional offline RL and generative planning-based baselines, given that the proposed method does not try to maximize the likelihood of demonstration, this looks more like a conservative model-based RL rather than a combination of planning and RL. The phrasing should be reconsidered, and in this sense, I wonder how impactful this paper is to the ICLR community (I acknowledge the contributions for the auto-bidding application, while in the offline RL context, Lipschitz constraints are another simple way to introduce conservativeness differently).\n\n- Theorem 2 seems to lack a necessary assumption. In my understanding, the evaluator's bias cannot be measured without having an assumption of the evaluator's accuracy or how the evaluator is trained (e.g., the evaluator is trained to satisfy the Lipschitz condition). It seems that he necessary assumption is not provided in advance.\n\n- Similarly, the paper claims that Theorem 1 is satisfied by \"the evaluator's design\" (i.e., by having $\\root{T} R_m$ constraints in the regression). However, if this is the reason, the theorem should be about the Lipschitz continuity of $\\hat{y}(\\tau)$, not that of $y(\\tau)$. There is no evidence presented regarding whether $y(\\tau)$ satisfies the Lipschitz continuity. I believe this should be an \"assumption\", instead of a \"theorem\". Overall, the theoretical analysis does not seem rigorous, and these details should be carefully reviewed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)