Steerable Adversarial Scenario Generation through Test-Time Preference Alignment
Authors: Tong Nie
Yuewen Mei
Yihong Tang
Junlin He
Jie Sun
Haotian Shi
Wei Ma
Jian Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Adversarial Scenario Generation
Autonomous Driving
Traffic Modeling
Test-time Alignment
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce a new paradigm for adversarial scenario generation with test-time steerability.

Abstract: Adversarial scenario generation is a cost-effective approach for safety assessment of autonomous driving systems. 
However, existing methods are often constrained to a single, fixed trade-off between competing objectives such as adversariality and realism. This yields behavior-specific models that cannot be steered at inference time, lacking the efficiency and flexibility to generate tailored scenarios for diverse training and testing requirements.
In view of this, we reframe the task of adversarial scenario generation as a multi-objective preference alignment problem and introduce a new framework named Steerable Adversarial scenario GEnerator (SAGE). SAGE enables fine-grained test-time control over the trade-off between adversariality and realism without any retraining. We first propose hierarchical group-based preference optimization, a data-efficient offline alignment method that learns to balance competing objectives by decoupling hard feasibility constraints from soft preferences. Instead of training a fixed model, SAGE fine-tunes two experts on opposing preferences and constructs a continuous spectrum of policies at inference time by linearly interpolating their weights. We provide theoretical justification for this framework through the lens of linear mode connectivity. Extensive experiments demonstrate that SAGE not only generates scenarios with a superior balance of adversariality and realism but also enables more effective closed-loop training of driving policies. Project page: https://tongnie.github.io/SAGE/.

Published as a conference paper at ICLR 2026
STEERABLE ADVERSARIAL SCENARIO GENERATION THROUGH TEST-TIME PREFERENCE ALIGNMENT
Tong Nie Yuewen Mei?; Yihong Tang", Junlin He', Jie Sun", Haotian Shi?,
Wei Malt Jian Sun
'The Hong Kong Polytechnic University, Hong Kong SAR, China Tonggi University, Shanghai, China
McGill University, Montreal, QC, Canada
ABSTRACT
Adversarial scenario generation is cost-effective approach for safety assessment autonomous driving systems. However, existing methods are often constrained single, fixed trade-off between competing objectives such as adversariality and realism. This yields behavior-specific models that cannot be steered at in- ference time, lacking the efficiency and flexibility generate tailored scenarios for diverse training and testing requirements. In view of this, we reframe the task of adversarial scenario generation as multi-objective preference alignment problem and introduce a new framework named Steerable Adversarial scenario GEnerator (SAGE). SAGE enables fine-grained test-time control over the trade- off between adversariality and realism without any retraining. We first propose hierarchical group-based preference optimization, data-efficient offline align- ment method that learns to balance competing objectives by decoupling hard feasibility constraints from soft preferences. Instead of training a fixed model, SAGE fine-tunes two experts opposing preferences and constructs continu- ous spectrum of policies at inference time by linearly interpolating their weights. We provide theoretical justification for this framework through the lens of lin- ear mode connectivity. Extensive experiments demonstrate that SAGE not only generates scenarios with a superior balance of adversariality and realism but also enables more effective closed-loop training of driving policies. Project page: https: attps://tengnie github..hh....g LSCEE
INTRODUCTION
Limitations existtin paradigms
b. SAGE: steerable generation by testttime allgnment
&. Glosed-loop training
1. Single objective: adversariallty
From discrete experts to continuous spectrum of behaviors
Dual curriculum learning
Odfline: learn experts with trade-offs Precondition
Test-time preference control
Rao
Multi objective linear scalarization
Hleearchical group
e(1) - (1-A) Brer A
800
ENY
ROV
Confiating
policy optimization
Co-evolution
Solt ps
Hard cumtriint
Learning
Adverumnal
Realitiic
peafonedrefeeecce landscape
Mixed model Is
HGPO
Debanior Spnce
Pretrained .0n
Static& Intiexible
Adrersantafrregeeny
Contleaiio
One flee
Meunte CyerHE
Mare
Waighrepaoe
gradiwv
Ros
Realistic IIGPO
Leas
Figure! Limitation of existing adversarial generation methods, our solution, and its application.
Safety assessment of autonomous driving (AD) systems is a prerequisite for their large-scale deploy- ment. A cornerstone of verifying and enhancing their capabilities is simulated testing, particularly through scenario-based methods (Xu et al., 2022; Ding et al., 2023). To improve testing efficiency and alleviate the issue of "curse of rarity" (Liu & Feng, 2024), adversarial scenario generation has emerged as cost-effective technique. Built on this method, practitioners can move beyond simple
"quual contributions. Corresponding authors.

---

Published as a conference paper at ICLR 2026
testing and enable closed-loop adversarial training, where an AD policy is continuously improved by the very challenges it fails (Ma et al., 2018; Wachi, 2019). This is achieved by systematically creat- ing challenging. taig-tailed corner cases that efficiently probe the system's operational boundaries, identify its blind spots, and thereby accelerate the development of more robust driving policies.
Instead of creating scenarios from scratch, many modern generation paradigms operate by perturbing a learned naturalistic driving prior and converting it into an adversarial posterior. Viewed through
this lens, whether the adversarial model is built using Reinforcement Learning (RL) (Feng et al., 2023; Ransiek et al., 2024a), guided diffusion (Xie et al., 2024; Chang et al., 2024; Xu et al., 2025), or direct optimization (Wang et al., 2021; Hanselmann al., 2022; Zhang al.. 2023), the task can be framed as an eejective-oriented adversarial optimization problem. The core challenge lies in con- trollably navigating the inherent trade-off between adversariality (capacity to induce a failure) and realism (physical and behavioral plausibility). This controllability is crucial for serving two distinct purposes: targeted stress testing may require highly extreme scenarios, while data augmentation for closed-loop training demands challenging yet realistic behaviors to ensure effective improvement. Fulfilling both needs requires a generation framework that is exceptionally flexible and steerable.
However, existing paradigms that directly solve the adversarial optimization problem fall short in providing this necessary level of controllability, especially at test time (Fig. ((a). On the one hand, naively optimizing for single objective, such as maximizing adversariality, often leads to collapse in realism. For example, methods based on aggressive search for collisions can generate physically implausible trajectories where vehicle spins in place or executes kinematically impossible turn to intercept the ego agent (Zhang et al., 2022; 2023). While adversarial, such scenarios offer limited value as they do not reflect behaviors that could occur in reality. On the other hand, while framing as multi-objective problem is a natural solution, the prevailing approach linearly scalarizes multiple managing set of
loss terms (Chang et al., 2024; Xu et al., 2025), which reduces the problem delicate weighting hyperparameters. This is highly heuristic and prone to instability, particularly when objectives induce conflicting gradients and involve a mixture of soft and hard constraints (Deb et al., 201 6). More critically, even a well-tuned model remains tied to a single fixed preference in the parameter space, mapping to a single point in the behavior space of possible trade-offs. This limits their test-time steerability, as adapting
new preference usually requires expensive retraining.
In this paper, we propose the first paradigm that reframes the solution of the adversarial optimiza- tion problem through the lens of preference alignment (Fig. 1(b)). Drawing inspiration from multi- objective alignment (Wortsman et al., 2022; Rame et al., 2023), we tackle intricate relationships between competing objectives on the Pareto front directly from preference data, balancing adver- sariality, realism, and feasibility, analogous to the "3H" (helpful, honest, and harmless) principles in Large Language Models (LLMs) (Bai et al., 2022). Crucially, our method enables test-time con- trol in both weight and behavior spaces. Instead of producing a single model for a fixed trade-off, we derive a -exture-of-prefernnce model that generates a continuous spectrum of policies. With this posteriori selection, user can adjust preferences at inference to generate better-informed trade- offs on the Pareto front tailored to specific needs without retraining. This framework is positioned as generalizable, applicable to various methods that employ adversarial optimization, well as to different driving policies under test. In essence, we shift the paradigm from manually designing weighted objectives to learning a controllable preference landscape. Our contributions are fourfold:
We introduce SAGE (Stcerable Adversarial scenario GEnerator), the first policy- and backbone- agnostic paradigm that reframes the task as multi-objective preference alignment problem.
We propose hierarchical group-based preference optimization, a sample-efficient offline alignment fine-tuning technique that decouples hard feasibility constraints from soft preference trade-offs.
SAGE uniquely provides a principled way for test-time preference control, enabling users nav- igate the entire Pareto front to generate tailored scenarios without retraining. Furthermore, we theoretically and empirically ground this paradigm the hypothesis of linear mode connectivity.
Comprehensive experiments demonstrate that SAGE establishes a superior trade-off, delivering competitive adversariality alongside the highest realism, and thereby enabling more effective closed-loop training of driving policies (Fig. I()) than state-of-the-an methods.
2 RELATED WORK
Adversarial Scenario Generation. This problem has been explored through several distinct paradigms. RL methods are well-suited for closed-loop training but suffer from limited transferabil-
2

---

Published as a conference paper at ICLR 2026
ity and tendency toward unrealistic reward hacking without meticulous engineering design (Kuutti et al., 2020; Feng et al., 2023; Ransick et al., 2024a; Qiu et al., 2025). Conversely, diffusion- bbssed models excel in generation with controllability (Zhong et al., 2023; Xie et al., 2024; Xu et al., 2025), but their high computational expense currently renders them impractical for the on-the-fly genera- tion required in closed-loop settings. Optimization and sampling- based methods (Wang al., 2021; Hanselmann et al., 2022; Zhang et al., 2022; 2023) offer computationally efficient and transferable alternative suitable for closed-loop training. However, they often struggle to balance adversariality with realism, frequently producing physically implausible scenarios without principled control.
Preference Alignment. Our SAGE is inspired by recent advances in steering LLMs. This field aims to align model behavior with complex and often conflicting human values, exemplified by the "3H" principles (Bai et al., 2022). Representative methods such as Reinforcement Learning from Human Feedback (RLHF) (Ouyang al., 2022) and Direct Preference Optimization (DPO) (Rafailov et al., 2023) offer powerful tools for aligning models into an aggregated preference signal. However, key limitation that they only produce a single model for a fixed trade-off. To over- come this, recent multi-objective alignment research has shifted towards enabling test-time control (Rame et al., 2023; Shi al., 2024; Xie et al., 2025). They typically train a set of experts, each specializing in different objective, and then interpolate parameters or predictions at test time to dy- namically navigate the Pareto front of trade-offs without retraining. While preference-based meth- ods are emerging in scenario generation (Cao et al., 2024; Qiu et al., 2025; Yu et al., 2025; Chen et al., 2025), they primarily focus on aligning with a single objective, such as realism or safety, by using standard DPO or RLHF during training. Our work is the first ground the multi-objective test-time alignment paradigm in adversarial scenario generation. By learning the trade-offs directly from preference data, provide a principled framework for steerable generation without retraining.
3 SAGE: STEERABLE ADVERSARIAL SCENARIO GENERATOR
3.1 PROBLEM FORMULATION
The goal of adversarial scenario generation is to discover or synthesize situations that reveal the vulnerabilities of a given driving policy Tego (Ding et al., 2023). Instead of creating scenarios from scratch, a common and effective paradigm is to perturb existing, typically naturalistic scenarios Stog logged from the real world. This approach, which we refer to as adversarial optimization (Zhang et al., 2022; Xu et al., 2025), can be abstracted into general multi-ohjective optimization problem.
Definition (Perturbation- based Adversarial Optimization). Given a naturalistic scenario Stng con- taining an initial trajectory Tly for an opponent agent, the goal is to find an optimal perturbation d that, when applied to Tlgg maximizes the objective Johj while adhering to set of constraints Tvalid:
8 arg.aaa Jany (Sugg 8; Teyo) s.t. Tlog + ô E Tralids
(I)
where Joby rypically involves multi-objective trade-offs, and Tralid enforces validity constraints.
Eq. provides unifying lens through which various existing methods can be viewed. Trajectory optimization (Hanselmann et al., 2022; Zhang et al., 2022) solves Eq. I directly using gradient- based techniques. Sampling- baase methods (Zhang al., 2023) approximate the solution by sampling from weightttt candidates based on Jatj. Diffusion- based generators (Xie et al., 2024; Xu et al., 2025) use Juyy as guidance signal to steer the reverse denoising process toward adversariality. The trade-off between competing objectives gives rise to a Pareto front: a set of optimal solutions where no single objective can be improved without degrading another. However, directly solving Eq. for fixed Jabj yields only single strategy corresponding to one point on the Pareto front. Our goal is to shift from optimizing for a single to generating a continuous spectrum of adversarial policies (Te, that can trace the entire Pareto front at test time. This paradigm is generalizable to various methods grounded in the adversarial optimization problem, enabling steerable scenario generation.
3.2 SREPERENCE-ALINNERR ADVERSARIAL TRAJECTORY OPTIMIZATION
Existing adversarial generation methods are often optimized for aggressiveness without sufficient constraints, leading to physically unrealistic or infeasible trajectories. Therefore, we fine-tune a pretrained motion generation model to generate trajectories that are simultaneously adversarial, realistic, and map-compliant. We frame this as a preference alignment problem, developing a new hierarchical preference optimization method that strategically handles the multi-objective nature.
3

---

Published as a conference paper at ICLR 2026
Multi-Objective Formulation. Let Tee be a pretrained, probabilistic motion generation model that maps a scenario context с to a distribution over future trajectories for a specific agent. The context includes the road map geometry . and the historical states of all agents. Our objective is to learn a new policy To. fine-tuned from Tee. that generates trajectories T ~ xo x=(-lc) biased towards our desired attributes. The quality of generated trajectory т is multifaceted. A naive approach would be to define a scalar preference score function Rooa(-) that holistically evaluates a trajectory:
(T; Wady Rad (T, Tego c) Wreal P Preat (7)
(2)
where Rudy measures the adversarial potency against the future trajectory of the ego vehicle Tego: should maximize the probability of causing a failure or a near-miss event to expose vulnerabilities of Tego- Preat ensures to adhere to the patterns of naturalistic driving; It should be statistically plausible and avoid behaviors that are physically possible but extremely unlikely, which could lead to an overly conservative ego policy. l'map penalizes violations of map constraints (e.g., crossing solid lines or driving off-road). The weights wi() balance these competing objectives. A trajectory 7" is preferred over r', denoted as 7 > r', if and only if (T") > (ro)a
However, this linear scalarization formulation presents a structural issue: it conflates soft preference objectives (trade-off' between adversariality and realism) with hard feasibility constraints (map com- pliance). A trajectory that passes through a building is totally invalid, not just less preferable. The binary nature is not well captured by continuous penalty. This complicates the optimization land- scape, which can result in unreliable outcomes where the model only partially satisfies constraints. Hierarchical Group-based Preference Optimization, To address this issue, we propose Hierar- chical Group-based Preference Optimization (HGPO), a new framework that decouples hard con- straints from soft preferences and improves the data efficiency of the alignment process. First, we treat map compliance not as a component of the reward, but as a precondition. We define a binary feasibility function F(T, MM {0, 1), which returns 1 if the trajectory T is fully compliant with M and 0 otherwise. The soft preference is then judged by a separate preference reward function Rper:
Rp c) Wady Rady (T, Tego: c) Wseat Wreat (T).
(3)
Second, a direct way would be to sample many feasible trajectories, identify the single best (r") and worst (7') according to R, and apply the standard DPO loss. However, this approach is data- inefficient as it discards fine-grained preference information contained within the rest of the samples. Inspired by the efficiency of group-wise learning (Shao et al., 2024), but adapted for an offline set- ting. we move beyond the single winner-losed pair. For each context c, we sample a group of N can- didates 9. = [ ~ [-cc).). We then partition Ge into a feasible set Geens = {r E G. F= 1] and an infeasible set Ginleas = {T E Go F = 0}. From these sets, we construct a dataset of pref- erence pairs D = {(r", +)} based on the following two principles, which hierarchically inject the desired inductive bias: (1) Feasibility First: Any feasible trajectory is strictly preferred over -') (r",-')
where 7" E Gfeus and 7 E Ginleas (2) Preference within Feasibility: Among feasible trajectories, we prefer those with higher preference reward. We generate pairs (r",7') -²) where both are in Gfeas but satisfy Rpre((T") (T"":) c) > Rper(r';c) + 5,, The margin 6m prevents learning from noisy preferences where the reward difference is negligible.
any infeasible one. We generate pairs (r
Fine-tuning Objective. Hierarchical principles enable the model to learn from a wider distribution of suboptimal and near-optimal examples. The preference fine-tuning process optimizes the policy Ty to explain the preference data better while being regularized by the reference policy Tref- Our proposed HGPO extends this to maximize the likelihood of the entire set of group- sampled pairs:
log a (3 (8(10s (T(7" log ग Teen (T'Ic)
To
LHGPO (8) E
eND Geovtol-le) A
Trer (T
(4)
where 3 controls the strength of the alignment. This formulation offers several advantages. First, by decoupling hard constraints, it simplifies the optimization landscape and directly enforces absolute feasibility. Second, by using group-wise sampling, it significantly improves data efficiency and leams more robust model from a richer set of comparisons. Last, the entire process remains offline, ensuring stable and efficient fine-tuning without the complexities of online reward modeling.
3.3 TEST-TIME STEERABLE GENERATION BY MIXTURE OF PREFERENCES
While HGPO provides an efficient way to manage hard constraints, it still requires fine-tuning a policy for fixed preference trade-off. This static approach has notable limitation: model fine-
4

---

Published as a conference paper at ICLR 2026
tuned on a fixed set of linearly scalarized rewards produces solutions corresponding to only a single point on the Pareto front of competing objectives. Adapting to different testing requirements needs costly retraining. Drawing inspiration from LLM alignment (Rame et al., 2023), we introduce test- time alignment method that enables continuous control over the properties of generated trajectories by creating mixed preference model, allowing users navigate the entire Pareto front at inference.
Training Expert Preference Models. Instead of training a single policy, first train set of expert models, each specialized towards a different trade-off between core goals. In our primary case with two objectives (i.e., udversariality and realism), we train two expert policies, Tea and Te Crucially, these are not trained on single objectives in isolation. Instead, they are fine-tuned from the same pretrained model Fee using HGPO described in Section 3.2, but with opposing rewards:
Radv--prer( (T) = wn RaddRad (T) (1 - w²) PP)PPe)(T) for training Tow.: Breat- prer( = (1 1 R (T) w= (T) for training Ti
where w' (0.5, 1] is fixed hyperparameter that pushes the experts towards the extremes of the preference space. This preconditions the models to understand the trade-off space, a key difference from methods that train on purely orthogonal rewards. These two expert models effectively anchor the endpoints of the achievable Pareto front for further preference interpolation in the entire space.
Steerable Generation via Test-Time Weight Interpolation. At test time, given a user-specified preference weight E [0, 1] defining the desired trade-off between adversariality and realism, we do not need to retrain. Instead, inspired by Wortsman et al. (2022); Rame et al. (2023), we directly construct novel, preference-mixee mixed model TT()) by linearly interpolating the weights of two experts:
(()) = (1 - re0real + A0
(6)
where X is coefficient linked the user preference μ. The mixed model TT(A) then generates set of K candidates {Tx K The final output r" is selected by ranking these candidates according to the user's 's real-time reward -" arg c(r.) Ry (Tk) with (r) puRaa (T) - (1-x)) (ral trace continuous path in weight spaces, which in turn generates
By varying À E [0, 1]. continuous Pareto front of behaviors, from naturalistic to aggressive, without any retraining. This defines set of (near) Pareto-optimal solutions {TT())) xx replacing costly multi-policy strategies. Furthermore, we can extrapolate beyond the interpolation range using preference vectors to generate more extreme and out-of-distribution cases for rigorous stress testing, as detailed in Appendix B.3.
SAGE in Closed-loop Adversarial Training. The steerability of SAGE is particularly useful when integrated into closed-loop adversarial training pipeline to improve an ego policy Tego- This process follows a min-max structure where, at each iteration, SAGE generates a new adversary tai- lored to the latest ego policy's weaknesses using the adversarial policy TØ(A). The ego agent is then updated to handle these new challenges. To stabilize training and prevent the agent from overfitting to adversarial cases while failing in normal driving, we employ a dual-axis curriculum. This curricu- lum progressively increases the challenge by gradually annealing two parameters: (1) the intensity of the scenarios, controlled by the interpolation weight X, which shifts from realistic to more aggres- sive behaviors. and (2) the frequency of adversarial encounters. This structured approach ensures the ego agent develops robust, generalizable skills. Further details are provided in Appendix B.
3.4 LINEAR MODE CONNECTIVITY IN FINE-TUNED MOTION GENERATION MODELS
We now provide a theoretical justification for our test-time weight interpolation scheme (Bq. 6). grounding its effectiveness in the phenomenon of Linear Mode Connectivity (LMC) (Frankle al., 2020; Ainsworth et al., 2022). LMC posits that when models are fine-tuned from the same pretrained initialization related tasks, the resulting solutions often lie within a single, wide, low-loss basin and can be connected by a linear path in the parameter space along which performance remains high. SAGE leverages this property, hypothesizing that this path effectively traces the Pareto front of the underlying multi-objective problem. This geometric property of the reward landscape has two key consequences that we formalize below, with full derivations provided in Appendix C.
Bounded Suboptimality of Weight Interpolation. First, LMC implies that the path between 6mn and 0reat likely traverses region of high reward. This suggests that an interpolation along it, (()) = (1 - 1)0 + ABadv. can serve as a high-quality approximation for the true optima 0,, corresponding to any user preference l E [0, 1). We formalize this by bounding the suboptimality gap.
5

---

Published as a conference paper at ICLR 2026
Theorem 1 (Suboptimality of Weight Interpolation). Let the base reward functions Rudi(() and R (0) be L-smooth and m-strongly concave in the local region of the fine-tuned optima. Let 81 and 0g be the optimal parameters for the two expert models (e.g.. Eq. 5) trained with mixing weight E (0.5,1) 1), and let 8y be the true optimum for user preference R,, (RRadt(0) + (1
(1) Rrea())) (0), E [0, 11. Then the suboptimality gap of our interpolated model is bounded by:
Ry((0)) max (,((11- X302 + X01) S C(u, B, L, L. | 01113 XE(O.1]
where Ci,, B, I,, m. is a constant dependent on the user preference j,, the expert weight 3, and the geometry of the reward landscape (smoothness I, and concavity m). See Appendix C.2 for details.
Theorem provides a quantitative guarantee the performance of our method. The LMC hypoth- esis suggests that because both experts are fine-tuned from the same powerful pretrained model oe on closely related objectives, they likely reside in the same basin of the reward landscape, mak- ing their distance small. This controlled suboptimality ensures that the weight interpolation can closely approximate the true Pareto front. This result generalizes the intuition from simplified quadratic reward setting (Appendix C.I), where we show that there exists range of user prefer- ences ja E [1 3, (] defined by the experts' training weights for which exactly optimal.
Superiority of Weight-Space over Output-Space Mixing. Second, LMC provides principled reason to prefer mixing model weights over mixing their outputs (i.e., ensembling trajectories). While ensembling is common alternative, the concavity of the reward landscape between solutions makes weight-space interpolation particularly effective. We analyze the difference in expected loss (e.g., mean square error (MSE) against an ideal trajectory) between the two strategies.
Proposition 1 (Advantage of Weight Mixing over Output Ensembling). Consider the expected loss t (A) of a weight-mixed model TE(A]) and an oulput-mixed model, (X). The difference in their performance is approximated by:
X(1 A) dLL
L weight
A(1 11-A) XL- 2
L weightt (A) L (A)=
(8)
dx2
Term 1: Benefit from Reward Concavity
Term 2: Benofit from Output Diversity
where (r(c)= T0, (-1c) TG, (-lc) is the difference in the mean predicted trajectories for given context c. The proof is detailed in Appendix C.3.
Proposition reveals a crucial trade-off. Term 2 always non-negative and represents the benefit of ensembling diverse outputs. However, Term 1 depends on the geometry of the loss landscape along the linear path connecting 01 and 0z. For weight mixing to be superior, Term must be negative and dominate Term 2. This requires the loss function to be convex along the interpolation path, a
i.e. dx= > 0. This convexity in the loss landscape corresponds reward landscape. Intuitively, linear path in the weight space traces a curved and concave path in the reward space. This means an interpolated model ((A) can achieve a reward greater than the linear average of the experts' rewards. While LMC posits that both experts reside in common low-loss basin, this does not imply zero curvature. Instead, the path connecting them acts as a high-reward ridge (Fig. 6(c)). The concavity of this ridge provides the necessary positive curvature in the loss, making Term 1 negative and dominant. This phenomenon is empirically validated by the concave reward curves shown in Fig. 6(d), which lie strictly above the linear interpolation line. By operating in the weight space of a well-structured model, our method exploits this geometric property generate high-quality behaviors learned by a coherent model.
concave geometry in the
Together, these results suggest that the geometric properties of the fine-tuning landscape make linear weight interpolation not merely heuristic but principled method for controllable generation.
4 EXPERIMENTS
We conduct comprehensive experiments to validate SAGE. Our evaluation is designed to answer three research questions: (1) In an open-loop setting, how does it compare against SOTA baselines in generating challenging yet realistic scenarios? (2) In closed-loop setting, does training with these scenarios translate into superior driving policies? (3) How can the steerable generation capability be comprehended in principled way? (4) Do the key designs, such HGPO, have positive impacts

---

Published as a conference paper at ICLR 2026
on the performance? All experiments conducted in the MetaDrive (Li et al., 2022) simulator using real-world scenarios from the Waymo Open Motion Dataset (Ettinger et al., 2021). Detailed setups and supplementary results are provided in Appendix B and D.
4.1 BENCHMARKING SAFETY-CRITICA SCENARIO GENERATION METHODS
Table 1:
Evaluation of adversarial generation methods against the Replay policy. Higher is better
for Attack Success Rate and Adversarial Reward (1), while lower is better for penalty metrics (L). WD denotes Wasserstein distance. SAGE is trained using separate adversarial weights Wadv-
Attack Suce.
Adv.
Real. Pen. +
Map Comp.
Dist. Dif. (WD)
Methods
Rate
Reward
Behav. Kine, Crash Obj. Cross Line Accel.
Vel. Yaw
Rule
100.00%
5.048
2.798 5.614
1.734
7.724
2.080 8,546 0.204
CAT (Zhang al,, 2023)
94.85%
3,961
8,941 3.143
2.466
9.078
1,556 7,233 0.225
KING (Hanselmann al., 2022)
40.85%
2.243
5.883
3.434
3.126
6.056
0.972 255.5 0.098
Adv AdvTrajOpt (Zhang et at., 2022)
70.46%
2.652
4,500
2.775
2.547
10.16
1.754
6,177 0,268
SEAL (Stoler et al., 2025)
63.93%
1.269
3.017
2.423
2.732
11.612
1,544 6,959 0,202
GOOSE (Ransick ct al., 2024a)
36.07%
2.378
4.718
21.32
4.426
14.48
6,368 8.286 0,154
SAGE (Waa 0.0)
16.26%
1.065
0.332
1.998
0.677
0.948
1.459
9.313 0.054
SAGE (Mayy 0.5)
50.41% %
2.523
0.483
2.064
0.755
0.949
1.521
8,471 0.079
SAGE (Wal 1.0)
76.15%
4.121
1.499 2.479
0.731
1.084
2.098 8,088 08184
Table 2: Evaluation of adversarial generation methods against the RL policy.
Attack Suce.
Adv.
Real. Pen. J
Map Comp.
Dist. Diff. (WD)
Methods
Rate
Reward
Behav. Kine. Crash Obj. Cross Line Accel.
Vel.
Yaw
Rule
65.57%
2.761
2.180 113.7
1.803
6.148
10.85 13.47 0.336
CAT (Zhang et al., 2023)
30.33%
1.319
8,191
3.039
2.623
6.967
1.539
8.877 0.187
KING (Hanselmann al., 2022)
19.14%
1.148
2.041
2.596
3.114
5.857
0.983 259.1 0.097
AdvTrujOpt (Zhang al., 2022)
19.40%
0.992
4.542
2.779
2.459
9.973
1.749 6.187 0.269
SEAL (Stoler et al. 2025)
31.40%
0.752
5.871
2.684
3.030
11.98
1.563 8.267 0,267
GOOSE (Ransiek et al., 2024a)
12.46%
0.667
4.369
15.47
3,507
11:45
4.662 8:070 0.152
SAGE (1aad 0.0)
11.20%
0.722
0.332
2,000
0.738
0.956
1,456 9.344 0.055
SAGE (Wal 0.5)
13.66%
0.819
0.496
2.066
0.820
0.820
1.515
8.475 0.080
SAGE (Maayy = 1.0)
28.42%
1.400
1.468
2.496
0.792
1.366
2.098 1.4 18188
We first benchmark SAGE in an open-loop setting against SOTA adversarial baselines. Tabs. 1 and 2 suggest that SAGE shows a superior balance between adversariality and realism. When configured for maximum adversariality. SAGE achieves a high attack success rate and adversarial reward, com- petitive with the strongest baselines. More importantly, it maintains significantly lower realism and map compliance penalties across the board. For instance, against the Replay policy, SAGE reduces map violations (c.g., Cross Line penalty of 1.084) by over 85% compared Rule-based (7.724) and CAT (9.078). while also achieving the highest behavioral realism. This highlights the effectiveness of treating map compliance as a hard constraint rather than soft penalty. A noteworthy observa- tion arises from the distributional distance metrics. While SAGE excels at maintaining low realism penalties for individual trajectories, it can produce a higher distributional distance than other meth- ods, though within reasonable range. This suggests that it achieves ad adversariality not by generating physically implausible actions, but by discovering coherent and statistically rare long- tail scenarios that challenge the ego agent precisely because they underrepresented in the training distribution.
RAW
CAT
Rule
SAGE
Qualitative results in Fig. (more examples in section D.5.2) confirm these findings. While baselines often produce physically awkward or rule-violating trajectories, SAGE generates challenging yet plausible maneuvers. This abil- ity is crucial for meaningful safety validation and robust agent training. Similar trends are observed when testing against a stronger RI. policy (Tab. 2), confirming the robustness of SAGE. Quantitative results against other reac- tive policies are provided in Appendix D. and visualized in Fig. 12. Fig. 3 shows that SAGE
AJJ
0g
Advensarial Vehicle
Ego Vehicle
O Other Vehicle
Figure 2: Behavioral realism comparison. Adver- sarial generation against the Replay policy. generates diverse types of plausible adversarial behaviors for effective stress tests and training.
7

---

Published as a conference paper at ICLR 2026
Cut-in
Rear-end
Sudden-breat
Left-tur
Opposite conflict
Turning overtake
Powe
Adversarial Vehicle
Ego Vehicle
Other Vehidle
Figure 3: SAGE generates diverse types of meaningful adversarial behaviors (Replay policy).
4.2 TEST-TIME STEERABLE GENERATION VIA PREFERENCE ALIGNMENT
A key contribution of SAGE is the ability to steer scenarios test time without retraining. Fig. 4(a) illustrates this capability by comparing the Pareto fronts of different mixing strategies. Our proposed weight mixing traces superior Pareto front, achieving a better realism udversariality compared to logit-space trajectory-space mixing (see section B.3), which empiri- cally validates Proposition The continuous and monotonic curves Fig. 4(b) further indicate the fine-graines controllability: as Wadv increases, the collision rate smoothly rises, with corresponding increase in the distributional distance from naturalistic data, indicating a controlled trade-off. The performance of SAGE at different weight configurations, shown in Tabs. 1 and 2, further quantifies this smooth transition. This controlability visualized Fig. 5: the generated trajectory transitions
for any given level of
smoothly from a compliant lane-following maneuver highly aggressive cut-in or sudden brake.
(a) Pareto Front Comparison of Adversarial Methods
(b) Transition Collision Distributional Dist.
Distribuu tomel Distanne (Yew Rales ND)
Co Rota
0.25
Distibutionl Distance
Adhinaaaal Nilggt (Maud
-0.20
Colfaion Rats
0.6
<he
0.15
-48-
0.4
0.3
-0.10
0:06
Monn
0.1
0.0
0.00
0.6
Adversarial Weight (Waou)
Figure 4: Pareto front and continuous performance transition at test time. (a) We compare the trade- off curves for different model merging strategies in terms of their adversariality and realism. (b) SAGE achieves smooth and continuous outcome control by varying the adversarial weight.
Raw
0.0
Wadv
Wade 1.0
Adverssrial Vehicle
Ego Vehicle
Adversarial Weight
Figure 5: More aggressive behaviors are generated from SAGE by increasing Wadv from 0 to (see Appendix D.5.3 for more examples). Adversarial generation against the Replay policy.
4.3 DECIPHERING THE PREFERENCE ALIGNED SPACES OF FINE-TUNED MOTION MODELS
We empirically validate the theoretical understanding of weight interpolation and LMC. Since we fine-tune well-pretrained model, it is plausible that the learned experts remain in local region of
8

---

Published as a conference paper at ICLR 2026
the weight space with favorable geometry. Fig. 6 provides strong evidence. Fig. 6(a) visualizes the preference vectors in a PCA-projected space, showing they are distinct and non-collinear, captur- ing unique aspects of the desired behaviors. The interpolated points lie between the line segment connecting two vectors, while the extrapolation points are scattered in space, which aligns with our understanding. In addition, the angle between the directions of two vectors lies within (0", 90"), which is reasonable to cover wide Pareto front. As we interpolate between Pady and Oeat,, Fig. 6(b) shows a smooth, monotonic transform in both the L2 norm of the preference vector and its cosine similarity to the target adversarial vector, indicating well-behaved and controllable path.
(a) Behavior (Task) Vector Geometry in PCA Space
(b) Smooth Linear Transformation of Task Vec. (e) Preference vector extrapolation.
Reation Peealy it Jdvet. dj
Comporeet
20
Veate
Coatre Centarty
70.67
Princical Component
Hopobdiin W
54
(c) Reward Landscape of Preferance-aligned Models
(d] Linear Mode Connectivity Motion Models
Adwesaridl Reward P pLadeed
Orteagres Dreckin
02
0.
Pucee
02-
Ob
Figure 6: Empirical evidence for LMC and preference vector manipulation in the weight space. (a) PCA plot shows the preference vectors (Andv. Arcal, see Eq. 14) forming a well-defined space for interpolation. (b) The interpolated vector's properties transform smoothly, indicating controllable path. (c) The reward landscape reveals a high-reward ridge connecting two experts, which is a key property of LMC. (d) The rewards exhibit concavity along the path, justifying the superiority of weight mixing. (e) Preference vector extrapolation improves performance beyond the convex hull.
Crucially, Fig. 6(c) shows the reward landscape along a 2D plane defined by the two experts. The linear path connecting them traverses high-reward plateau, consistent with the LMC prediction of a flat loss basin. This flatness and curvature are key to the success of weight interpolation, making the suboptimality gap well-bounded. Fig. 6(d) provides direct evidence: the measured rewards of interpolated models closely track or even exceed the rewards predicted by linear interpolation of the rewards of endpoints. This confirms that interpolating in weight space is more effective than simply expecting a linear combination of outcomes, supporting Proposition 1. Furthermore, Fig. 6(e) demonstrates that preference vector extrapolation (see Appendix B.3) successfully extends generation beyond the convex hull of the experts, yielding a path that achieves even higher rewards and lower penalties than the solution on the original Pareto front, thereby enabling the creation of duiti--distribution scenarios for rigorous testing. More results are provided in Appendix D.4.
4.4 CLOSED-LOOP ADVERSARIAL TRAINING FOR IMPROVED DRIVING POLICY
Table 3: Evaluation of trained RL policies in the generated (adversarial, Wady = 1.0) environments.
Methods
Reward f
Cost |
Compl.
Coll. $
Ave. Speed Ave. Jerk $
SAGE
45.14 +: 3.27 0.61 + 0.04 0.69 + 0.03 0.31 + 0.02
8.98 +: 0.02 28.85 + 0.68
CAT
37.70 + 1.53
0.70 + 0.04
0.58 + 0.02
0.37 + 0.04
6.85 + 0.03
31.83 + 1.10
Replay (No Adv)
41.32 3.21
0.68 + 0.04
0.62
0.04
0.44
0.06
8.77 + 0.01
30.42 + 1.12
Rule-bused Adv
32.99 4.89
0.72 = 0.03 0.00
0.33 + 0.02
5.99 0.04 30.51 + 0.99
To assess the downstream utility of SAGE, we integrate it into closed-loop RL training pipeline. As shown in Fig. 9 (in Appendix D.2) and Tab. 3, an ego agent trained with SAGE consistently outper- forms baselines across all metries: it achieves higher rewards, lower costs, greater route completion, and a lower crash rate. Beyond the safety metrics, it also achieves better efficiency and comfort performances. Moreover, SAGE alleviates catastrophic forgetting as evidenced by Tab. 10 (in Ap-

---

Published as a conference paper at ICLR 2026
pendix D.2). The policy trained with SAGE also achieves the best overall performance in log-replay scenarios. This justifies the effectiveness of our dual-axis curriculum (see Appendix B.
To further ensure fair evaluation and assess the risk of overfitting, we conducted a cross-evaluatiin of the trained RL policies. In this setup, we benchmarked every trained agent on held-out test sets generated by the CAT and Rule-based methods. The results in Tab. 4 provide several key insights. First, we observe that an agent tends to achieve the best performance on the adversarial metric when evaluated on scenarios created by its own training generator. For example, the CAT-trained agent has the lowest collision rate in CAT-generated scenarios. This validates the importance of this evaluaavalutiion. Second, and more importantly, the tAGEetrained agent demonstrates superior generalizability across all environments. Its safety performance remains highly competitive, ranking a close second. Notably, the traied agent achieves the highest route completion rate across all adversarial test sets, demonstrating that it learns to handle challenges more effectively without compromising its primary driving objective. This robustness across diverse adversarial distributions suggests that the curriculum-bus basedddd--dddd-d--- training with SAGE, which exposes the agent to wide spectrum of adversarial intensities, prevents overfitting to a specific attack pattern and enables a more general- izable and practical driving policy. In contrast, agents trained more static adversarial distributions (CAT, based) show a more significant performance drop when tested out-of of-distribution.
Table 4: Evaluation of Trained RL Policies Different Adversarial Environments.
CAT-generated scenarios
Rule-generated scenarios
Training Methods
Compl. f
Coll. +
Compl.
Coll.
SAGE
0.678 + 0.034
0.307 0.037
0.655 + 0.037
0.334 + 0.038
CAT
0.660 1 0.039
0.298 + 0.050
0.633 1 0.043
0.389 1 0.044
Replay (No Adv)
0.665 + 0.042
0.454 + 0.051
0.642 + 0.027
0.461 + 0.046
Rule-based Adv
0.569 + 0.050
0.348 = 0.028
0.556 = 0.059
0.293 + 0.043
4.5 ABLATION STUDY
We conduct ablation studies to validate key de- sign choices. Fig- 7(b) highlights the superior sample efficiency of HGPO over DPO. By con- structing multiple preference pairs from group of samples, HGPO utilizes much more infor- mation per scenario, leading to a more stable training loss and faster convergence (Fig. 7())). We also validate our approach of treating map compliance a precondition. Figs. 7(c.d) show that this maintains a high map feasibility rate of nearly 90% and a higher reward throughout training. In contrast, removing the map con- straint leads to a collapse feasibility, as the model learns to exploit off-road shortcuts. Us- ing weighted penalty improves feasibility but remains suboptimal compared to our HGPO. This confirms that decoupling hard constraints from soft preferences is critical for generating valid trajectories. See more results in Appendix D.3.
Training Loss DPO)
(b) Average Sample
DPO) HOPO (oan))
6.905
Avq (1a3) Toeeto
0.00
DFO Sasiind
Daps
Trahing Strpe
[c] Winner Reward (Mop Pun. Ablutions)
[d) Foasibillty Rate (Map Pon. Ablations)
Agg See The Reans
Hppo shamn
HOPD (oun)
vay)
HSFO be up (wz)
Figure 7: Ablation studies.
CONCLUSION
This work introduced SAGE, a new paradigm that reframes adversarial scenario generation as a test-time preference alignment problem. By fine-tuming experts on opposing preferences and inter- polating their weights at inference, SAGE enables fine-grained, steerable control over the trade-off between adversariality and realism. SAGE not only yields superior Pareto front in open-loop eval- uations but also results in improved driving policies in closed-loop training. Our work establishes principled foundation for steerable generation and opens several future avenues. First, the cur- rent framework could be extended to incorporate a richer set of objectives, such as scenario novelty or complexity. Second, more advanced model merging techniques could potentially uncover better paths in the parameter space. Finally, an automated curriculum where the model is dynamically adapted based on the agent's learning progress could lead more intelligent adversarial training.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This research was in part supported by the National Natural Science Foundation of China (Project No. 524B2164, 52232015, and 52125208); the Research Grants Council of the Hong Kong Spe- cial Administrative Region, China (Project No. PolyU/15206322 and PolyU/15227424); and Otto Poon Charitable Foundation Smart Cities Research Institute, The Hong Kong Polytechnie University (CD06).
REFERENCES
Samuel К Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint urXiv:2209.04836 2022.
Luca Anzalone, Paola Barra, Silvio Barra, Aniello Castiglione, and Michele Nappi. An end-to-end curriculum learning approach for autonomous driving scenarios. IEEE Transactions on Intelligent 2022.
Transportation Systems, 23(10):19117-1988
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Yulong Cao, Chaowei Xiao, Anima Anandkumar, Danfei Xu, and Marco Pavone. Advdo: Realistic adversarial attacks for trajectory prediction. In European Conference on Computer Vision, pp- 36-52. Springer, 2022.
Yulong Cao, Boris Ivanovic, Chaowei Xiao, and Marco Pavone. Reinforcement learning with human feedback for realistic traffic simulation. In 2024 IEEE international conference on robotics and automation (ICRA), pp. 14428-14434 IEEE, 2024.
Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka. Wei Zhan, and Manmohan Chandraker. adversaries. In
Safe-sim: Safety-critical closed-loop traffic simulation with European Conference on Computer Vision, pp. 242-258. Springer, 2024.
Keyu Chen, Yuheng Lei, Hao Cheng, Haoran Wu, Wenchao Sun, and Sifa Zheng. Frea: Feasibility- guided generation of safety-critical scenarios with reasonable adversariality. arXiv preprint arXiv:2406.000 02983, 2024.
Keyu Chen, Wenchao Sun, Hao Cheng, and Sifa Zheng. Rift: Closed-loop rl fine-tuning for realistic and controllable traffic simulation. arXiv preprint arXiv:2505.033444 2025.
Kalyanmoy Deb, Karthik Sindhya, and Jussi Hakanen. Multi-objective optimization. In Decision sciences, pp. 161-200. CRC Press, 2016.
Wenhao Ding, Baiming Chen, Minjun Xu, and Ding Zhao. Learning to collide: An adaptive safety- critical scenarios generating method. In 2020 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 2243-2250. IEEE, 2020.
Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao. A survey on safety-critical driving scenario generation methodological perspective. IEEE Transactions Intelligent Transportation Systems, 24(7):6971-6988. 2023.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. Proceedings of the IEEE/CVF interna- fional conference computer vision, pp. 9710- 2021.
Shuo Feng, Xintao Yan, Haowei Sun, Yiheng Feng, and Henry X Liu. Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment. Nature communica- tions, 12(1) :2(1):748, 2021.
Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. Dense reinforcement learning for safety validation autonomous vehicles. Nature, 615(7953): 020-627,40 2023.
11

---

Published as a conference paper at ICLR 2026
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp- 3259-3269 PMLR, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine leurning, pp. 1587-1596. PMLR, 2018.
Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 15303- 15312, 2021.
Niklas Hanselmann, Katrin Renz, Kashyap Chitta, Apratim Bhattacharyya, and Andreas Geiger. King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. In European Conference on Computer Vision, pp. 335-352. Springer, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089 2022.
Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information pro- cessing systems, 29, 2016.
Sampo Kuutti, Saber Fallah, and Richard Bowden. Training adversarial agents to exploit weaknesses in deep control policies. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 114 IEEE, 2020.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3):3461-3477 2022.
Henry Liu and Shuo Feng. Curse of rarity for autonomous vehicles. nature communications, 15 (1):4808, 2024.
Yuxuan Liu. Peo: Improving bi-factorial preference alignment with post-training policy extrapola- tion. arXiv preprint arXiv:25033./2233 2025.
Xiaobai Ma, Katherine Criggs-Campbell, and Mykel J Kochenderfer. Improved robustness and safety for autonomous vehicle control with adversarial reinforcement learning. In 2018 IEEE Intelligent Vehicles Symposium (/V), pp. 1665-1671. IEEE, 2018.
Yuewen Mei, Tong Nie, Jian Sun, and Ye Tian. Bayesian fault injection safety testing for highly automated vehicles with uncertainty. IEEE Transactions Intelligent Vehicles, 2024.
Yuewen Mei, Tong Nie, Jian Sun, and Ye Tian. Lim-attacker: Enhancing closed-loop adversarial scenario generation for autonomous driving with large language models. IEEE Transactions on Intelligent Transportation Systems, 2025a.
Yuewen Mei, Tong Nie, Jian Sun, and Ye Tian. Seeking to collide: Online safety-critical sce- nario generation for autonomous driving with retrieval augmented large language models. arXiv preprint arXiv:2505.00972 2025b.
Tong Nie, Jian Sun, and Wei Ma. Exploring the roles of large language models in reshaping trans- portation systems: survey, framework, and roadmap. Artificial Intelligence for Trunsportation, 1:100003, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Le Qiu, Zelai Xu, Qixin Tan. Wenhao Tang, Chao Yu, and Yu Wang. Aed: Automatic discovery of effective and diverse vulnerabilities for autonomous driving policy with large language models. arXiv preprint 2503::2000.0004 2025.
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53744 2023.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, lean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by in- terpolating weights fine-tuned diverse rewards. Advances in Neural Information Processing Systems, 36:71 (6671095-71133 134, 2023.
Joshua Ransiek, Johannes Plaum, Jacob Langner, and Eric Sax. Goose: Goal-conditioned reinforce- ment learning for safety-critical scenario generation. In 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), pp. 2651 IEEE, 2024a.
Joshua Ransiek, Philipp Reis, and Eric Sax. Adversarial and reactive traffic agents for realistic driving simulation. arXiv e-prints, pp. arXiv-2409, 2024b.
Davis Rempe, Jonah Philion, Leonidas Guibas, Sanja Fidler, and Or Litany. Generating useful accident-prone driving scenarios via a learned traffic prior. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 17305-17315, 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hanna Hajishirzi, Noah A Smith, and Simon Du. Decoding-time language model aligmment with multiple objectives. Advances in Neural Information Processing Systems, 37:48875 48920,, 2024.
Benjamin Stoler, Ingrid Navarro, Jonathan Francis, and Jean Oh. Seal: Towards safe autonomous driving via skill-enabled adversary learning for closed-loop scenario generation. IEEE Robotics and Automation Letters, (99):1-8, 2025.
Akifumi Wachi. Failure-scenarió maker for rule-based agent using multi-agent adversarial rein- forcement learning and its application to autonomous driving, arXiv preprint arXiv: 10654, 2019.
Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 9909-9918, 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23993 PMLR, 2022.
Guofu Xie, Xiao Zhang, Ting Yao, and Yunsheng Shi. Bone soups: A seek-and-soup model merging approach for controllable multi-objective generation. arXiv preprint arXiv:2502.10765 2025.
Yuting Xie, Xianda Guo, Cong Wang, Kunhua Liu, and Long Chen. Adydiffuser: Generating adversarial safety-critical driving scenarios via guided diffusion. 2024 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 9983-9988 IEEE, 2024.
Chejian Xu, Wenhao Ding, Weijie Lyu, Zuxin Liu. Shuai Wang, Yihan He, Hanjiang Hu, Ding Zhao, and Bo Li. Safebench: A benchmarking platform for safety evaluation of autonomous vehicles. Advances in Neural Information Processing Systems, 35:25667 17756677-5682,, 2022.
Chejian Xu, Aleksandr Petiushko, Ding Zhao, and Bo Li. Diffscene: Diffusion- safety-critich scenario generation for autonomous vehicles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 8797-8805, 2025.
Seungjun Yu, Kisung Kim, Daejung Kim, Haewook Han, and Jinhan Lee. Direct preference optimizationneehancee multi-guiden diffusion model for traffic scenario generation. arXiv preprint 22x 2025.
13

---

Published as a conference paper at ICLR 2026
Linrui Zhang, Zhenghao Peng, Quanyi Li, and Bolei Zhou. Cat: Closed-loop adversarial training for safe end-to-end driving. In Conference on Robot Learning, pp. 2357 2372-2372. PMLR, 2023.
Qingzhao Zhang, Shengtuo Hu, Jiachen Sun, Qi Alfred Chen, and Z Morley Mao. On adversarial robustness of trajectory prediction for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15159-15168, 2022.
Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guide traffic simulation via scene-level diffusion. Conference robot learning, pp. 144-177. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
DECLARATION OF THE USE OF LLMS IN THE WRITING PROCESS
During the preparation of this work, the authors used LLMs to improve language clarity and read- ability. The authors reviewed the content as needed and take full responsibility for the content.
APPENDIX
This technical appendix provides comprehensive supplementary materials to the main paper. It is structured as follows: Section A offers detailed review of essential background concepts. Section B outlines the full implementation details of our experiments. Section C presents a rigorous theoretical analysis. Finally, Section D provides extensive supplementary experimental results.
APPENDIX CONTENTS
A Background and Preliminary
16
Preliminary About Adversarial Scenario Generation
16
A.1.1 Traffic Scenario Representation
16
A.1.2 Closed-loop Adversarial Training for Safe Driving
16
A.2 Detailed Related Work on Adversarial Scenario Generation
16
Detailed Implementation
18
B.1
Experimental Setups
18
B.2 Detailed Formulation Reward and Penalty Functions
22
B.3 Weight Extrapolation via Preference Vectors
24
B.4 Pseudocode for Offline Preference Optimization and Online RL Training
25
C
Theoretical Analysis
25
C.1
Suboptimality Gap with Quadratic Reward Approximation
27
C.1.1 Problem Formulation
27
C.1.2 Derivation of the Exact Suboptimality Gap
28
C.2 Suboptimality Gap with Non-quadratic Reward
29
C.2.1 Problem Formulation
29
C.2.2 Derivation of the Suboptimality Gap Bound
29
C.3 Analytical Comparison of Weight Mixing and Output Mixing Schemes
D Supplementary Results
35
D.1 Adversarial Generation Evaluation against IDM and Rule-based Policies
35
D.2 Full Results of Closed-loop RL Training
36
D.3 Full Results of Ablation Studies
36
D.4 Additional Results of Weight Extrapolation
37
D.5 Additional Case Studies .uddies.........................
38
D.5.1 Examples of Adversarial Generation against Different Ego Policies
38
D.5.2 More Examples of Generated Scenarios
38
D.5.3 More Cases on Preference Controllable Generation
39
15

---

Published as a conference paper at ICLR 2026
A BACKGROUND AND PRELIMINARY
A.1 PRELIMINARY ABOUT ADVERSARIAL SCENARIO GENERATION
This section first introduces concepts and notation for representing traffic scenarios and agent be- haviors. We then formalize the closed-loop adversarial training framework discussed in section 3.3.
A.1.1 TRAFFIC SCENARIO REPRESENTATION
We model traffic scenario S as collection of interacting agents operating within a static envi- ronment. The environment is defined by a high-definition map M. which contains geometric and semantic information. The set of agents in the scenario is denoted by where dg is the AD agent being tested (ego), and {a.);=1 are the other traffic participants, referred to as other agents. The state of an agent a, at timestep is represented by si = (x), yi. 0j, 2f). which includes its 2D position, heading, and speed. The trajectory of an agent a over time horizon T is a sequence of its states, 7² {sh, .f..... 8. The collective behavior of all agents in the scenario described by the set of all trajectories T = 1 Thus, a complete scenario can be defined as In this sense, the scenario context e refers to the historical scenario the initial scenario be predicted. Here, scenarios are sourced from real- worldwrlld driving logs. In this work, we consider the ego agent ag is controlled by an end-to-end driving policy Tego which directly maps a sequence of observations of future plan or low-level action.
{ao, 01,...,. .........e........ -N-1),
a tuple S (M. {7080 ,7, .... n N ))).
A.1.2 CLOSED-LOOP ADVERSARIAL TRAINING FOR SAFE DRIVING
To enhance the robustness of an ego agent's 's driving policy, we leverage the paradigm of adversarial training (Ma et al., 2018; Wachi, 2019). This approach goes beyond training on a fixed dataset of benign scenarios by dynamically generating challenging scenarios that are specifically tailored the policy's 'ss current weaknesses. In the context of reinforcement learning (RL), the standard objective for the ego agent is to learn a policy Tego that maximizes the expected cumulative reward J within given environment, characterized by a transition function f:
Σ R(se, R(st,at) an)
max I (Tego: 5) max Eep Wagg
Maag
where R is the environment reward function and the trajectory T sampled by executing policy Tego in the environment f. Adversarial training extends this into min-max optimization problem. The adversary actively modifies the environment's 's transition function to fadv to minimize the ego agent's 's reward, while the ego agent simultancously learns to maximize its reward in this worst-case environment:
max min I (TTgoo fadv), Kego fute EJ"
(10)
where F is the space of feasible adversarial environments. The inner loop, min corresponds to the adversarial scenario generation process, and the outer loop max is the policy optimization step, where Tego is updated using an RL. algorithm to overcome the challenges posed by faáv- This formulation can lead to closed-loop and iterative training pipeline (Zhang et al., 2023). The policy is continuously improved by training on adversarial scenarios generated on-the on-the-fly against its most recent version. This co-evolution of the agent and environment forces the policy to learn general- izable and robust driving skills, rather than overfitting to a static set of scenarios (Anzalone et al., 2022). The efficiency and quality of the generator are therefore critical to the success of the pipeline.
A.2 DETAILED RELATED WORK ON ADVERSARIAL SCENARIO GENERATION
The generation of safety-critical scenarios crucial for testing and enhancing the robustness of au- tonomous driving (AD) systems (Xu et al., 2022; Ding et al., 2023; Nie et al., 2025). In recent years, several distinct paradigms have emerged to automate this process. each with its own strengths and limitations. In this section, provide comprehensive review of these paradigms, focusing Re- inforcement Learning- baase. Diffusion Diffusion-based, and Optimizzation/Smmpingg based methods. We po- sition work within this landscape, addressing key unresolved challenges of existing paradigms.
16

---

Published as a conference paper at ICLR 2026
RL-based Generation. RL has been widely adopted to generate adversarial scenarios by training an adversary agent to maximize a reward function correlated with the failure of the ego policy (Wachi, 2019; Kuutti et al., 2020; Feng et al., 2021; 2023; Chen et al., 2024; Ransick et al., 2024a; Qiu et al., 2025). In this paradigm, the ego agent is treated as part of the environment, and the adversary learns complex, non-intuitive behaviors to exploit its weaknesses. A significant advantage of RL--aaed methods is their inherent suitability for closed-loop training, enabling the ego agent to be hardened against an adapting adversary (Kuutti al., 2020; Ransiek al., 2024a). However, this approach suffers from two major drawbacks. First, limited transferability: the adversarial policy tightly coupled with the specific ego agent it was trained against. A scenario generated for one AD stack may not be challenging for another, necessitating costly retraining for each new system under test (Ransiek et al., 2024b; Ding et al., 2020). This makes it ill-suited for large-scale, general- purpose testing. Second, unrealistic behaviors: RI. agents are prone to reward hacking, where they discover unrealistic or physically implausible strategies to maximize the reward, such as erratic movements or violating basic traffic rules, unless the reward function is meticulously engineered (Kuutti et al., 2020; Qiu et al., 2025).
Diffusion-based Generation.
Leveraging their success in high-fidelity image and video synthe- sis, diffusion models have recently been applied to traffic scenario generation (Zhong et al., 2023; Xie et al., 2024; Xu et al., 2025). These methods learn the distribution of real-world traffic data and can generate realistic, diverse, and controllable scenarios. By incorporating controllability during the reverse diffusion process, these models can be steered to produce safety-critical events, such as collisions or near-misses, based on textual descriptions (Zhong et al., 2023) or predefined objective functions (Xie et al., 2024; Chang et al., 2024; Xu et al., 2025). This controllability is primarily achieved with guidance and conditioning. Diffusion guidance is strong in control but slow, and the other is statistically true but weak in adjustability. This paradigm excels in generating realistic and highly controllable scenarios. Moreover, these methods are generally policy-agnostic, as the gener- ative model is trained on real-world data rather than against a specific ego agent, leading to better transferability. The primary limitation of diffusion- overhead. The iterative denoising process is computationally expensive and slow, which makes it currently unsuitable for closed-loop adversarial training, where scenarios need to be generated on-the-fly to continuously challenge and improve the ego agent's policy.
methods is their significant computational
Optimization and Sampling-based Generation. This category of methods focuses on perturbing existing real-world scenarios or sampling from a learned prior to generate adversarial variants (see section I). The search for adversarial parameters often performed in different spaces, such as the parameter space of vehicle kinematics (Hanselmann et al., 2022; Cao et al., 2022; Mei et al., 2024), the latent space of generative model such as a VAE (Rempe et al., 2022). or directly in the trajectory space ( (Wang et al., 2021; Zhang et al., 2023; Mei et al., 2025a;b). Especially, CAT (Zhang et al., 2023) falls into this category by modeling adversarial traffic as a probabilistic factorization and sampling trajectories with the highest collision probability. Their main advantage lies in their computational efficiency, which makes them viable for closed- adversarial training frameworks. They are also typically often struggle with controllability and realism. The optimization or sampling process, if not properly constrained, can push trajectories into unrealistic regimes. As noted in our motivation, methods like CAT can produce overly aggressive yet physically implausible trajectories, such as vehicles spinning in place, which limits their utility for training robust and realistic AD policies.
and thus highly transferable. Nevertheless, these methods
Summary. Existing paradigms for adversarial scenario generation present a fundamental dilemma between generalizability, efficiency and controllability. Reinforcement Learning (RL) methods, for instance, can discover complex adversarial strategies but are often coupled to specific AD policy and prone to reward hacking, leading to unrealistic behaviors (Feng et al., 2023; Qiu et al., 2025). Conversely, fiffuusionbbseed generative models offer superior generalizability and fine-graines con- trol (Zhong et al., 2023; Xie et al., 2024; Xu et al., 2025), yet their significant computational over- head renders them impractical for on-the-fly generation required in closed-loop settings. A promis- ing avenue lies in optimization or sampling-bos based methods, which operate on perturbing a learned naturalistic driving prior (Wang et al., 2021; Hanselmann et al., 2022; Zhang et al., 2023). They are efficient and generalizable for closed-loop training, but achieving meaningful control over the gen- eration process remains challenge. A new paradigm that can inherit the merits of both is desired.
17

---

Published as a conference paper at ICLR 2026
B DETAILED IMPLEMENTATION
B.I EXPERIMENTAL SETUPS
This section provides detailed description of our experimental setups, including the dataset, simu- lation environment, backbone model, baseline methods, and our fine-tuning and RL training proce- dures, to ensure the reproducibility of our results.
Datasets. Our experiments are conducted using the Waymo Open Motion Dataset (WOMD) (Et- tinger et al., 2021), which is consistent with prior works (Zhang et al., 2023; Stoler et al., 2025). WOMD provides large collection of real-world, complex, and interactive traffic scenarios. From the standard set of 500 scenarios used in Zhang et al. (2023), we select a filtered subset of 370 scenarios for experiments. This subset is curated by excluding scenarios where the adversar- ial agent's initial trajectory is either too short for meaningful interaction or already violates map boundaries, ensuring high-quality and challenging cases for adversarial generation. This dataset utilized for both generating adversarial scenarios and for fine-tuning our generative models.
Environment. All our experiments are performed in the MetaDrive vve (Li et al., 2022) simula- tor, a lightweight and efficient simulation platform for AD research. Following the setup in Zhang et al. (2023), we use MetaDrive to reconstruct scenarios from the WOMD dataset. This allows for closed-loop evaluation, where the ego agent can react dynamically to the behavior of the ad- versarially controlled traffic participants. The simulation runs at frequency of 10Hz. To ensure fair and direct comparison with prior work, we adopt the specific environmental and agent training parameters from Zhang et al. (2023). The key hyperparameters for the scenario generation process used for evaluation are detailed in Tab. 5.
Table 5: Hyperparameters for the open-loop scenario generation evaluation.
Table 6: Hyperparameters for training the TD3 RL agent.
Hyper-parameter
Value
Hyper-parameter
Value
Scenario Horizon T
9s
Discounted Factor y
0.99
History Horizon t
1s
Train Batch Size
256
of OV candidates M
32
Critic Learning Rate
10-4
# of EV candidates N
Actor Learning Rate
3 x 10-4
Penalty Factor &
0.99
Policy Delay
Policy Training Steps
1 x 106
Target Network
0.005
Pretrained Backbone Model. The foundation of our scenario generation framework is prob- abilistic motion generation model. To ensure fair comparison with the baselines, we employ DenseTNT (Gu et al., 2021), the same backbone model used in CAT (Zhang et al., 2023). DenseTNT is one of the state-of-the-art goal-conditioned motion prediction model that generates a diverse set of future trajectory proposals along with their corresponding confidence scores. The pretraining task is to train the model to predict future trajectories using historical observations and scene context, supervised by human experts. This follows the standard imitation learning (Gu et al., 2021). This pretrained model serves as the learned traffic prior that SAGE fine-tunes and that all baselines (except for the rule-based one) leverage to generate adversarial behaviors. To enable end- to-end preference optimization, we modify the decoding procedure of DenseTNT by outputting the top-k candidates directly, bypassing the non-differentiable non-maximum suppression (NMS) sampling process. Furthermore, our tockbone-agnostee framework is generalizable and applicable to any probabilistic motion forecasting (generation) backbone that can generate multimodal trajec- tories with their probabilities (predicted logits). We take DenseTNY as an example in the experi- ments. We adopt the publicly released pretrained model checkpoint from the CAT's repository for experiments to ensure a fair comparison.
We compare SAGE against a comprehensive set of strong baselines from recent litera- ture on safeta-critical (adversarial) scenario generation.
Baselines.
18

---

Published as a conference paper at ICLR 2026
CAT (Zhang al., 2023): A of method that generates adversarial scenar- ios by resampling trajectories from a learned traffic prior (DenseTNT) to maximize a collision-based objective in closed-loop setting. Following the original implementation, CAT employs a heuristic objective. It generates a set of candidate trajectories T from the DenseTNT prior and selects the adversarial trajectory Tady that maximizes the collision probability with the ego vehicle's ' planned path Tegoi Tadv = arg m (T Tego)).
SEAL (Stoler et al., 2025): A recent approach that employs a skill-enabled adversary. It combines learned objective function with a reactive, skill-based policy to generate more realistic and human-like adversarial behaviors than CAT. We utilize the official open- sourced implementation of SEAL. Its objective is based on learned scoring network, Tscore, which is trained to predict two criticality metrics: collision closeness (feon) and ego behavior deviation (farr) The final adversarial objective is select trajectories that maximize the sum of these predicted scores: max R [Jaon (Tadv; Tego) + saur( (Tat; Tego)] This learned objective, combined with a skill-based policy, aims to produce more nuanced and realistic adversarial behaviors.
KING (Hanselmann et al., 2022): A gradient-based approach that perturbs adversarial tra- jectories by backpropagating through a differentiable kinematic bicycle model to induce collisions. For a fair comparison, we re-implemented its core mechanism within our exper- imental environment, as detailed in our supplementary code. The optimization is performed for 100 steps with learning rate of 0.005 using an Adam optimizer.
AdyTrajor (Zhang et al., 2022): This method models adversarial scenario generation as a trajectory optimization problem and solves it using Projected Gradient Descent (PGD). We adapted its WGD-based trajectory perturbation logic into environment, ensuring operates on the same initial conditions and interacts with the same ego agent. Our imple- mentation uses 50 PGD steps with learning rate of 0.2. For both KING and AdyTrajOpt, we use the same multi-objective reward function as SAGE, defined in Eq. (2) to ensure a fair comparison. To align with their goal of generating highly challenging scenarios, we set the weights to a fixed point on the Pareto front that heavily prioritizes adversariality over realism, with Wady = 0.9 and Wheat 0.1. This provides a direct comparison of the generation quality under a shared, aggressive objective.
GOOSE (Ransiek et al., 2024a): A condiiondi RL framework for safety-critich sce- nario generation. GOOSE models the entire trajectory using Non-Uniform Rational B- Splines (NURBS), and the RL agent learns to manipulate the NURBS control points to achieve predefined adversarial goals. For our experiments, we use the pretrained model checkpoint released by SEAL's repository for the WOMD dataset to ensure a fair compar- ison.
Rule-based: heuristic baseline consistent with the one described in Appendix of Zhang et al. (2023). This method generates adversarial path by selecting waypoints from the ego vehicle's future path, mixing them with the adversary's original waypoints, and fitting a smooth Bezier curve through them to create an aggressive cut-in or blocking maneuver.
To justify the superiority of the test-time alignment scheme proposed in section 3.3, we further compare two different alignment methods as described below:
Trajectory mixing: The output is generated by directly mixing trajectories from the two expert models: (A)) = (1 (1- + ATalv- This is an instantiation of the output ensembling method discussed in Proposition 1.
Multi-objective decoding (Shi al., 2024): Inspired by Shi et al. (2024), mix the predicted logits of two experts. Following the architecture of DenseTNT (Gu et al., 2021), we use the logits of all candidate goal points as the approximation of trajectory probability. This gives a mixed goal score g(c; 2) = (1 A)g(c; Oreat) + >g(c; Andv). ). Then, the final trajectory is generated by decoding g(c; A) with the highest score value using the pretrained trajectory decoder. This scheme mimics the token prediction stage of LLMs.
Compared to our weight-spaat mixing, both of them are action-space space mixing strategies, thus serving as direct counterparts to justify Proposition 1.
19

---

Published as a conference paper at ICLR 2026
HGPO Fine-tuning. Our preference fine-tuning approach has several hyperparameters. For the number of trajectory candidates, we let the policy model generate a group of M = 32 trajectories used for reward computing. A reward difference exceeding margin of 6,,. 0.2 is regarded as a valid signal. We sample up to 8 pairs per scenario for the group-based learning process. The temperature parameter is set to 8 = 0.05 to regularize the alignment. We use the AdamW optimizer with a learning rate of 1 x 10-5. We train two specialized models by adjusting the weights in the preference reward calculation:
Adversarial Model: We set the adversarial weight Wadv = 0.9 and the realism weight Wreal = 0.1. This configuration strongly encourages the model to prioritize generating trajectories that result in safety-critical interactions.
We set Wadv 0.1 and What 0.9. This configuration prioritizes kine- matic and behavioral plausibility, encouraging the model to generate more human-like and realistic, albeit potentially less aggressive trajectories.
Realism Model:
The models trained for 200 epochs the 370 selected training scenarios. Hyperparameters for HGPO are provided in Tab. 7.
Details of the System Under Test. To thoroughly evaluate the generated adversarial scenarios, we test them against diverse set of ego agent policies, representing different levels of driving capability and design philosophies of AD systems.
This is the most basic agent, which deterministically replays the ground- truth trajectory of the ego vehicle recorded in the WOMD dataset. It serves non- reactive baseline to measure the raw difficulty of a scenario.
Replay Policy:
Intelligent Driver Model (IDM)) A standard, rule-based car-following model widely used in traffic simulation. The IDM agent maintains safe following distance and adjusts its speed based on the vehicle directly ahead. It is also equipped with rule-based lane change and overtake maneuvers to enable lateral control.
Trained RL Agent: A standard TD3 agent (Fujimoto et al., 2018) trained via RL on the original, non-adversarial WOMD scenarios. The observation input of the RL agent contains ego states, navigation information and surrounding information provided by 2D LiDAR sensor. This agent represents a competent end-to-end searning-based policy but is naive to adversarial behaviors, allowing us to measure the impact of adversarial training.
Rule-based Expert: A sophisticated and safety-oriented rule-based policy. The rule-based policy operates on perfect state observations, which directly provide the ground-trun po- sitions of surrounding vehicles. This expert agent employs a hierarchical control structure. At the low level, it uses separate PID controllers for lateral and longitudinal control. Crit- ically, it features a proactive safety layer that uses a kinematic bicycle model to forecast its own trajectory and the future states of surrounding vehicles. It continuously checks for potential future collisions by performing oriented bounding box intersection tests. high-risk situation is predicted, it activates the collision avoidance strategy, which is to reduce the vehicle speed. This expert serves as a challenging SUT, representing a robust, well-engineered AD system.
Open-loop Evaluation. The open-loop evaluation protocol is designed to benchmark the effee- tiveness and characteristics of different scenario generation methods against a target ego agent be- havior, ensuring a fair comparison. The process follows a two-stage procedure (Zhang et al., 2023) for each scenario and each SUT:
1. Ego Trajectory Collection: First, the SUT (e.g., the Rule-based Expert or the Trained RL Agent) is executed in the original, unmodified WOMD scenario. Its complete trajectory over the 9-second horizon is recorded. This step establishes a consistent and deterministic behavioral target for all adversarial generation methods.
2. Adversarial Attack and Re-simulation: The environment is then reset to the initial state of the same scenario. The adversarial generation method under evaluation uses the ego trajectory recorded in the first stage target to generate optimal adversarial trajectory
20

---

Published as a conference paper at ICLR 2026
for designated background vehicle. Finally, the scenario is re-simulated with the SUT and the newly generated adversarial trajectory.
During the re-simulation, we measure key metrics, including the success rate of the attack (i.e., col- lision rate), the adversarial reward (measuring the severity of the interaction), and various realism metrics that penalize kinematically implausible behaviors and map violations. In addition, we caleu- late the Wasserstein distance (WD) using the kinematic profiles (speed, yaw rate, and acceleration) of the generated and logged scenarios, serving as distributional metric. This open-loop setup iso- lates the performance of the generator by holding the ego agent's ' reactive behavior constant across all compared methods.
Note that the measure of adversarial reward is different for the replay policy and others (Tables and 2). For interactive policies, this reward is calculated using the resulting ego trajectory after interacting with the adversarial agent, i.e., after the re-simulation step.
Closed-loop Adversarial Training with Dual Curriculum Learning. The ultimate goal is to integrate SAGE into closed-loop training pipeline to progressively improve the capabilities of the policy Tege= as discussed in section 3.3 and A.1.2. In this setup, the scenario generator and the RL agent (a TD3 agent with hyperparameters detailed in Tab. 6) are trained concurrently. A dual curriculum is designed to alleviate catastrophic forgetting, where Tego is overly optimized for corner cases and fails in normal conditions.
SAGE in RL Loops. The adversarial training follows the min-max structure in section A.1.2. At each iteration, we solve the inner-loop optimization by generating a new adver- (t)
sary tailored to the current policy Tego- Instead of having the adversary find a single worst case, we define the adversarial environment Sudv from the mixed policy ma(A())- This pro-
cess samples from a controllable adversarial priors: ((+11) Geeerate(TTo(x60)) Tego: c((),
where e(*) is the current context and (+) is the interpolation weight at iteration i. The ego is then trained for several steps, forming the outer loop: Tego
(i+1) UpdateRr (Fego adv ((++1))
At the end of each training episode, the RL agent's trajectory is recorded and stored in buffer. When an adversarial episode is initiated, the scenario generator uses the most recent trajectories from this buffer to generate a new adversarial scenario specifically tailored to the agent's current policy and its emergent weaknesses. This iterative co-evolution forces the ego agent to continuously adapt. Unlike training on fixed set of scenarios, SAGE gen- crates a dynamically shifting distribution of scenarios. This enhances the generalizability of Tego- as it learns to handle wide range of adversarial behaviors rather than overfitting to a narrow set of hard but unrealistic cases.
Dual Curriculum. As discussed in section 3.3, SAGE enables natural implementation of a scenario curriculum along two axes: intensity and frequency. Exposing a nascent policy to frequent aggressive scenarios from the start can destabilize RL training. We mitigate this by smoothly increasing the challenge the agent becomes more competent. (I) Intensity: The adversarial intensity is controlled by annealing À over the course of training. Let Tioal be the total number of training steps. At timestep t, ((t is given by:
(+)) Avaant (Aend Astan) min ......0). This ensures the ego agent first lears to handle common deviations before being confronted with safety-critical maneuvers. (2) Frequency: Simultaneously, we control how often the ego agent encounters an adversarial scenario. At the end of each episode, we decide whether to generate an adversary with a (t)
probability Pndv. which is given by: Pad- = Pstant + (Pend - Pstant -iin
Timn ( 1.0) where
Pstnt and Pend are initial and final values. Specifically, it starts from a low probability of 0.1 and achieves 0.9 by the halfway point. Second, the weight of the adversarial objective Wady
is annealed from an initial value of 0.5 final value of 1.0.
After several training steps, the RL agent is evaluated both normal and adversarial scenarios (with a fixed Wadv = 1.0 for the latter). The performance is measured by collision rate, route completion rate, total environment reward, and total cost. The environment reward is the cumulative sum of rewards throughout an episode, reflecting overall task performance. It includes a dense reward for progress along the planned route, positive reward for successfully reaching the destination, and
21

---

Published as a conference paper at ICLR 2026
significant penalties for failures like crashing going off the road. The total cost is pure safety metric, representing the cumulative sum of costs from safety violations during an episode. A cost of 1.0 is incurred for each instance of crash cost considered perfectly safe, making this a direct measure of the agent's safety failures. this experiment, use 300 randomly selected scenarios for RL training, and 70 scenarios for evaluation. To evaluate the superiority of SAGE in closed-loop training, we compare it with several different scenario generators in the same training environment, including Replay, CAT, and a rule-based generator.
out-of-road event. An episode with zero total
used
B.2 DETAILED FORMULATION OF REWARD AND PENALTY FUNCTIONS
This section provides the detailed mathematical formulations for the functions used to evaluate and filter generated trajectories. These include the adversarial reward (Rady), the realism penalty (Preat), and the binary map compliance function (F(T, M)). For full reproducibility, we provide list of all hyperparameters used in reward functions in Table 7.
Table 7: Hyperparameters for reward functions and HGPO.
Category
Hyperparameter
Value
Optimizer
AdamW
Learning Rate
1.0 10-5
Training Epochs
200
Batch Size
(scenario)
HGPO Fine-tuning
Group Size (N)
32
Max Pairs per Group (K)
Preference Margin (6m)
0.2
Beta (3)
0.05
Adversarial Expert (TTa.)
Preference Reward
Ugdy Wheal Realism Expert (#
0.9
Wadv / Wheal
0.1 0.9
Collision Reward Scale
10.0
Radv Components
Proximity Reward Scale
1.0
Proximity Decay Rate
0.2
Whum
5.0
Wstop-tum
3.0
Kinematic Penalty Factor (Accel)
5.0
Kinematic Penalty Factor (Ang. Vel.)
5.0
Preat Components
Accel. Comfort Zone
7.0 m/s²
Lat. Accel. Comfort Zone
6.0 m/s?
Ang. Vel. Comfort Zone
0.8 rad/s
Max Reasonable Turn
w radians
Cross Solid Line Penalty
50.0
Map Feasibility
Crash Object Penalty
10.0
Adversarial Reward (Raav) The adversarial reward Rad (Tadv, Tego) is designed to quantify the threat level of a generated adversarial trajectory Tadv with respect to the ego vehicle's recorded (re- simulated) future trajectory Tegu- A high reward should correspond to high-risk scenario. The function provides dense signal by rewarding both direct collisions and near-miss events.
Let the adversarial and ego trajectories be represented sequences of 2D positions over T future timesteps:
{p. ego
T' Jt=
Pr odv E R2
Tago
{p. Jt
T D E R2
We define ((p, vb I, w) as the oriented bounding box (polygon) of a vehicle at position p with yaw vh length 1, and width w. A collision occurs timestep if the bounding boxes of the vehicles
22

---

Published as a conference paper at ICLR 2026
intersect: (Faddv,) O B (Tego.t) # 0. Let fooll be the first timestep at which a collision occurs:
tcoll = minft E [1, [] B(Tadv,)) n ((Tego.:) + 0}
If no collision occurs, we define ccll oc.
The adversarial reward is then defined as a piecewise function:
Cooll (1
full T
if fooll S T if bcoll >> T
Radv (Tadv: Tego) =1c Cprox exp( -Aprox dmin)
(11)
where Coou is large constant reward for achieving collision (e.g., 10.0). The term (1 tcoll/T) (T) incentivizes earlier collisions, as they are typically more critical and harder to avoid. dmin min;e(1.T] |ppddv Pt pt ||2 is the minimum Euclidean distance between the vehicle centers through- out the horizon. Cpexx is a scaling factor for the proximity reward (e.g., 1.0), and > is a decay rate (e.g.. 0.2). This term provides smooth, dense reward signal for near-misses, encouraging the agent to generate trajectories that are spatially close to the ego vehicle even if they do not result in collision.
Realism Penalty (Prea) The realism penalty, (T), ), is designed to discourage trajectories that are physically implausible or exhibit unnatural driving behavior. It is composed of two sub-penalties: a kinematic penalty Pain and behavioral penalty Pheh-
Let trajectory t be sequence of states (Pr, ( where the is the yaw angle at timestep t. We first compute the primary kinematic quantities with a time interval of At 0.1s:
Speed: s, ||p: Pt-1ll2/At
Longitudinal Accelcration: Glong,t = (st -s-1)/Att
Angular Velocity: Wt (V'r - 1/-11/At,, where the heading is unwrapped handle angle discontinuities.
Lateral Acceleration: Ojat = 81
We define a smooth penalty function S(x, Tihresh) = log(1 + exp(1æ] Zzhresh)) which penalizes
values of [x] that exceed threshold Zthresh-
Kinematic Penalty (Biin). This penalty discourages violations of physical limits.
T
Pai(())
Σ
Wa [S (Stong,, Omax) + S(Car,, Qlat.max) ]
T
+ Wyy Slwee Wmax)
(12)
where wa and ww are weight factors, and {Amax, aaa,max, ] are comfort thresholds for longitu- dinal acceleration (e.g., 7.0 m/s?), lateral acceleration (e.g.. 6.0 m/s?), and angular velocity (e.g.. 0.8 rad/s), respectively.
Behavioral Penalty (Phet). This penalty targets unnatural maneuvers, such as spinning in place or executing excessively sharp turns over the trajectory horizon.
Pbel(T) (T) "Wam S(A Ae'max)
Wstop-turn
(13)
T
St
where: Avhoaal 1w V/l is the total change in heading over the trajectory. The penalty dis- courages total turns exceeding a reasonable maximum A (e.g., я radians). The second term penalizes high angular velocity at low speeds, characteristic of unrealistic spinning maneuvers. Whum and tso are weight factors, and small constant to prevent division by zero
The total realism penalty is the sum of these components: Preal(7 (7) Pkin(T) (7) + (bet(T).
23

---

Published as a conference paper at ICLR 2026
Map Compliance Function (F M)) Unlike the continuous- rewards and penalties, map compliance is treated as a hard, binary constraint. The function FTT.M) returns the trajectory is feasible with respect to the map M, and 0 otherwise. A trajectory is deemed infeasible if it violates any of the following conditions:
1. Road Boundary Violation: The trajectory must remain within the drivable area. Let Cimpessable C M be the set impassable map polylines (e.g., road edge boundaries). The trajectory is infeasible if the vehicle's bounding box B (Tr) intersects with any of these lines at any time.
EE E [1 E Limpussable s.t.
B(T) = F(T, =0
2. Static Object Collision: The trajectory must not collide with other static or near-static background vehicles. Let Ostatic be the set of non-player vehicles in the scenario, each with their own future trajectory of bounding boxes (Tob)..)-
38 E [1, 7J, 3royj E Ostati
s.t. ((T)) 0 (Toa,,) + = F(T,M)= =0
If none of these violation conditions are met, the trajectory is considered feasible, and F(T,M) 1. This strict separation of feasibility from preference allows the optimization to focus on learning meaningful trade-offs within the space of valid behaviors. Computationally. these checks are imple- mented efficiently using spatial data structures like STR-trees and broad-phasee phase//arrow-phhaeee colli- sion detection.
B.3 WEIGHT EXTRAPOLATION VIA PREFERENCE VECTORS
While weight interpolation effectively traces the Pareto front between the two expert models, cannot generate scenarios that are more extreme than what the experts themselves can produce. To overcome this, we introduce a post-hoc extrapolation technique inspired by the model editing (Ilharco et al., 2022; Liu, 2025). This section provides more detailed discussion on this weight extrapolation technique introduced in section 3.3. The core idea is reframe the model merging process in terms of preference (task) vectors, which enables principled extrapolation beyond the space spanned by the expert models.
Preference Vectors as Learned Task Representations. We begin by formally defining the prefer- ence vectors (Ilharco et al., 2022). These vectors capture the specific parameter updates that involve each preference. Given a pretrained reference model 7ett with parameters O and two expert mod- cls, Tea., and Th.... trained to specialize in adversariality and realism respectively, define their corresponding preference vectors as:
Audy Oadv Bget-
(14)
A Oreat
Oxe--
Each vector A represents the directional update in the high-dimensional parameter space that adapts the general-purpose reference model to satisfy specific preference objective (c.g., maximizing Radv-peet). contains the knowledge required for that particular task.
Equivalence of Model Interpolation and Vector Interpolation. The linear interpolation of model weights, as described in Eq. 6, can be shown to be equivalent to applying an interpolated preference vector to the reference model. This is clear by rewriting interpolated parameters (()):
((A) = (1 2)0real + A0mtv
(1 2) + Areal) + A(Oref + Annv),
(15)
= (1 1)0ret + (1 - A) + 20 + >
0.t + ADadr + A)
This shows that creating mixing of expert models is mathematically identical to starting with the reference model and adding a weighted combination of the preference vectors. This perspective shifts the focus from merging final models to combining the underlying learned skills.
24

---

Published as a conference paper at ICLR 2026
Extrapolation for Pareto Front Improvement. The interpolated model (T())) is confined to the line segment connecting Peea and Bady in the parameter space. This traces out a corresponding path on the Pareto front of objectives, but this path is not guaranteed to be the globally optimal one (will be discussed below). The optimization landscape for the combined objectives may have better solutions that lie outside this segment. Extrapolation allows 2025). Instead of being limited to convex combinations of the experts, we can form an extrapolated policy The by moving beyond the interpolation segment. We construct a new policy by adding linear combination of preference vectors to an existing point:
search for these superior solutions (Liu,
ext
Brase +
Σ O.
(16)
'E (adv, real)
where Ornse starting point (e.g., Ojet. one of the experts, or an interpolated (())) and o; are scalar coefficients. For instance, setting == 1 and Φndv 0 pushes the model to become even more adversarial than the original TT... This allows for the generation of truly extreme, -itto--distrbbttion scenarios, providing a more rigorous stress test for the ego agent without any additional training.
Intuition from Gradient Optimization. To understand why this works, we can draw an analogy to gradient-based optimization, following the insights from Liu (2025). Let Radv and (e) be the expected preference rewards that the expert models Te.., and Ti... were optimized for, re- approximation of
spectively. The fine-tuning process that produced Bud, from 6eer can be seen moving along the gradient of Rav- Therefore, the resulting preference vector Andy is closely related to the integrated gradient of the objective function. A first-order approximation suggests:
Aniv
fadv Oze B V Rad (0)),
(17)
where 7) represents an effective learning rate over the entire training process.
Next, the extrapolation step Σ A; is approximately equivalent taking new optimization step in direction defined by a weighted combination of the original objective gradients:
V E EoUR10))
Σ -. E
(18)
Recall that the interpolated model TO(A), might have converged a point in the parameter space that is a local optimum or a saddle point with respect to the true, combined multi-objective landscape. The extrapolation, by adding Σ Φ); A,, provides a new gradient like momentum. This momentum, constructed from the gradients of diverse objectives, can help the policy skip out of the local op- the parameter space that yields a better trade-off, which is a
timum and move towards a region point superior Pareto front. This allows us to generate out-of-distribution scenarios that can be crucial for the safety assessment of AD systems.
B.4 PSEUDOCODE FOR OFFLINE PREFERENCE OPTIMIZATION AND ONLINE RL TRAINING
The complete procedures for offline preference optimization and closed-loop RL training are sum- marized in Algorithms and 2.
C THEORETICAL ANALYSIS
In this section, we provide detailed theoretical analysis to bound the suboptimality gap between the true optimal solution for user's 's preference and the best solution achievable by linearly interpolating two mixed-reward expert motion models. We aim to show that for a given user preference, the optimal parameters for the mixed reward are well approximated by interpolation of expert model weights, as justification for Theorem 1. We first consider a simplified version and then extend the analysis to a more general case. Finally, we provide detailed analysis of Proposition
General Assumptions. Before presenting the detailed derivations, first clarify general assump- tions that bridge the theoretical analysis with the practical implementation of our method. Recall that the fine-tuning is performed using the HGPO loss (Eq. 4), which is preference-basee objective.
25

---

Published as a conference paper at ICLR 2026
Algorithm Offline Fine-tuning of Expert Models with HGPO
1: Input: Pre-trained motion model Tret scenarios dataset D scenarius expert rewards Reduppels Ryeat-pee (from Eq. 5), group size N, pairs per group K, margin 6m, DPO parameter (3.
2: function FINEPUNEEXPERT Rexper)
3: Initialize policy me with weights from mie-- 0.
Initialize an optimizer 5: for each training epoch do for each scenario context E Duceurios do
6:
7:
Sample group of trajectories using the latest policy 9e hr mo(-le). Partition G. into feasible Ge and infeasible G sets using M).
8:
Initialize preference pairs De 0.
Rule Feasibility First
11:
Sample pairs (7" (7",,") with T E E , E Ga and add to De & Rule 2: Preference within Feasibility Sample pairs (7 ("), with fr" (r",")) g s.t. Respen (7") ) Respen(T)) 6m. add to De
12:
13:
14:
Limit DDea to K.
Compute HGPO loss: L B (y log (3(log N log ma(6"1e) medt? )}}
15:
16:
Update 0 using the gradient of L.
end for
18: end for
19: return fine-tuned parameters
20: end function
21: Pate Expeettn (aaeeExxeettRaaapeee Radr-pett)
22: Oual FineTuneExpert( Reeat-peet)
23: Output: Expert model parameters Oade Oreal-
Algorithm 2 Closed-Loop Adversarial Training with Dual-Axis Curriculum
1: Input: Fine-tuned expert models nea. Th.2 RL algorithm; Total timesteps Tewt Warm-up steps Taut; Schedules X Play Environment 'env'.
2: Initialize ego policy Tez and replay buffer B.
3: f- env.reset()
4: for to T do
b Ego agent interaction b Initial exploration
6: if < Taan then a ~ Uniform(env.action space)
7: else (s)
8:
end if
s'.r, info <- env.step(a)
10:
Store as,,,r,s', ,'d) in B; s +-
11:
Train Tego using samples from B.
12:
if is True then
13:
env.reset(); Get context c. Update curriculum À +- (t)) Pady +- Paay (Sec. B. if random() < Pady then
Episode ends, and generate
14:
(6)
15:
Generate an adversarial scenario
16:
8(A) + (1 X)0rea - ABpo
17:
~ (Teer Rego, c) (Sec. 3.3 and Sec. B.1).
18:
env.sett.alversary(aaa
19:
else
Generate benign scenario
20:
env.set adversary(oeeeee)
21:
end
22:
end if
23:
end for
24:
Output: Robust ego policy Rego.
26

---

Published as a conference paper at ICLR 2026
However, our theoretical analysis is based on the direct optimization of expected reward functions R(O) and the geometry of their corresponding loss landscapes. To bridge this gap. we introduce the following assumption, which is well-grounder in the literature, to make the analysis tractable while still capturing the core principles that enable our approach.
Assumption (Implicit Reward Function for HGPO). We assume that the process of optimizing the HGPO objective is equivalent to implicitly maximizing an underlying reward function, regularized by the KL-divergence from the reference policy Tyf. This connection is formally established for DPO under the Maudeey-Terry framework (Rafailov et al., 2023), which shows that minimizing the DPO loss is equivalent solving a reward-man maximization problem. Therefore, we posit that the continuous and differentiable functions, the expected adversarial reward Rade (0) = Er-za (Rad(T)) and the expected realism reward (f)
I-Prall in our analysis represent these implicitly learned reward proxies for the optimization landscape shaped by the HIPO objective.
This assumption allows us to analyze the behavior and geometry of the HGPO-optimized solution using reward landscape analysis, even though the practical algorithm operates on preference pairs.
C.1
SUBOPTIMALITY GAP WITH QUADRATIC REWARD APPROXIMATION
Before presenting the general suboptimality analysis for non-quadratic functions, we first provide an intuitive derivation under the assumption that the reward landscapes are quadratic. This simplified setting, inspired by the analysis in Rame et al. (2023), allows for a closed-form solution for the suboptimality gap and offers clear insights into how mixed-expert interpolation method behaves. This analysis serves as a foundation for understanding the more general results in section C.2.
C.1.1 PROBLEM FORMULATION
We model the reward functions as quadratic forms, which can be seen as a second-order Taylor approximation of the true reward landscape in the vicinity of the optima.
Assumption (Simplified Quadratic Rewards). The expected base rewards (0d and Reaa( are quadratic functions of the model parameters 0 E Rdd Specifically, their Hessians are proportional to the identity matrix:
Rady Cadv
Plady
||0
6t 112
2
(19)
Preat Craat
Thral
110
where 0i and Tr are the unique global optima for the pure adversarial and realism rewards, respectively. The constants Nadvr Treat
determine the curvature of the reward landscapes, and
C()) are the maximum reward values.
Under this assumption, we can analytically find the optima for the expert and user reward functions.
Expert Optima. Two expert models are trained on mixed rewards, defined by 8 E (0.5, 1]:
Rí (0) 3 Raa. (0) + (1 (3) Rreal (0), R2(0) = (1 - R)aa + B (O).
(20)
The global optimal parameters 01 and 0g that maximize these rewards are found by setting their gradients to zero. The gradient of Rı (0) is:
RR (0) - -yaav (0- (Msv)) (1 B)) (0 Frea).
(21)
Setting VRi (01) and solving for 01 yields:
0 Brpao Oz (1 33rhraa
(22)
Biaad + (1 Preal
By symmetry, the optimum for the second expert, 02. is:
0₂ = (1 3a Bi 3.
(23)
(1 B) + 3Treal
This shows that the expert optima, 0, and 02. are themselves weighted averages of the pure optima 0% and o and thus lie the line segment connecting them.
27

---

Published as a conference paper at ICLR 2026
True User Optimum. A user's preference is defined by R2(() (B) + +1 μi) Rreal (0), with μ E [0, 1]. Following the same procedure, the true optimal parameters 0r. for the user are:
02 1illdd udv + (1 O
(24)
Hayad (1 hreal
C.1.2 DERIVATION THE EXACT SUBOPTIMALITY GAP
Our method approximates 0, by interpolating the expert weights: ((A) = =1 À [0, 1]. Since 01, 02. and 0z are all collinear (as they are all different weighted averages of fadv and re can perfectly represent Ó,, if it falls within the line segment [02, 01].
(1- A)02 + A01 for
To simplify the analysis and gain clearer insight, we consider the case where the reward curvatures equal, i.e., Iaav 'heal 1/. The optima then become simple linear interpolations:
= 30 + (1 3ea
02 (1 3)0 + 30
(25)
0, = pllaav + (1 1i)afel
Optimal approximation. Our interpolated solution is 0(A) = (1 Xj02 + A0). We seek À [0, such that (()) 0n. Substituting the expressions for 01 and 02:
((A) 1)((1 - 0 + 3 + A(B0 + (1 - 3)
(26)
[[1 A)(1 -3 A + [[1 Δ)B X(1 B)]
Then we have the following equality of the coefficients:
1 8-1
(1 - A) - 3) AB
(27)
23 -
Since 3 E (0.5, 1]. the denominator 2/3 - 1 is positive. The solution is achievable via interpolation if E [0, 1]. This condition holds and only if μ E B, B]
Suboptimality Gap. the user's preference μ falls outside the range [1 B, we cannot perfectly match the true optimum 0,.- The best achievable solution is found at the boundary of the interpolation range, i.e., at X (giving 02) or ity gap, AR, = Ry (0,)) - R2 (Obea), can then be calculated. Consider linear combination of quadratic functions with the same quadratic coefficient, the user reward function is given by R,)(0) C₁ / | 0₂ 12. Therefore, the gap is:
(giving 01). The suboptimal-
17 2 10best 11.11
ARG
(28)
Let /Acippee = clip(/4, 1 3). The optimal point achievable through interpolation corresponds to the user preference Melipped- The distance between the true optimum and the best achievable one is:
8 Abest (μb + (1 11)8 - (Heliipeed@@dv + (1 (Allippod)
(29)
= (μ Melipped) (0 - ("rat)
This leads the final expression for the suboptimality gap.
Theorem 2 (Suboptimality Gap for Quadratic Rewards). Under Assumption 2 with equal curvatures
(Nady Theat n), the suboptimality gap of the mixed-expert model for user preference μ E [0, 1] given by:
Ra
(Q.)
max (O(A)) = "(x ( XE[O,1]
(30)
where 3 E (0.5, 1] is the mixing coefficient used to train the expert models.
28

---

Published as a conference paper at ICLR 2026
Interprefation. This result provides several key insights. (1) Region of Optimality: Our method is completely optimal when the user's 's preference i lies within the range [1 - B, 3]. The width of this range is controlled by the expert mixing coefficient 3. A value closer to 0.5 creates less specialized experts but covers a wider range of user preferences optimally. A value closer to creates more spe- cialized experts but narrows the region of perfect optimality. (2) Error Growth: Outside the optimal region, the suboptimality gap grows quadratically with the distance of from the boundary of the region. (3) Task Dissimilarity: The gap is scaled by |10 0iaal 1|22, the squared distance between the pure optima. This confirms the intuition that the approximation is worse when the underlying objectives are fundamentally more conflicting. requiring very different model parameters. Since we fine-tune pretrained model Tref it is plausible that the learned parameters remain in local region of the weight space with similar geometry.
This simplified analysis demonstrates the effectiveness and predictable behavior of our method. We now proceed to the more general analysis for non-quadratic reward functions, which builds upon these core ideas.
C.2 SUBOPTIMALITY GAP WITH SON-QUADRATIC REWARD
In the following, we generalize the above discussion to non-quadratic rewards by leveraging standard assumptions from convex optimization theory. This provides a complete proof of Theorem 1.
C.2.1 PROBLEM FORMULATION
Following the above ideas, we have similar components but with the consideration of a more class of reward functions: the expected adversarial reward Radv (f) = Eyww, [Rav (T)] and the expected [- Preal (r)]- The definition of other ingredients such as expert rewards, user reward, expert optima, and true optimum keep the same. Our goal is to derive an upper bound for the suboptimality gap: AR, = R²)(0)) maxxejo.11 (0x())).
realism reward (0) = E,-=,
To analyze the behavior of our non-quadratic reward functions, we introduce two standard assump- tions from optimization theory.
Assumption (L-Smoothness). The base reward functions Ras (f) and Reeat are differentiable and their gradients are Lipschitz continuous with constants Ladv > and Limat 0, respectively. That is, for any 0a, Og E Rdd
I Reado (fa) V Radv (06)l S Teady 1 0a Abll.
(31)
V Rmea((.) V Rmmaa (0b) S L 0zl1.
Assumption 4 (m-Strong Concavity). The base reward functions Radr (0) and (rea(()) are strongly concave with constants madd and mimal > respectively. That is. for any fa. 0g Rd:
Rud (O.) S (f6) + V (0,) ((B. 0.)
aad
110,
0b.1
(32)
2
A similar inequality holds for (f) with constant Mreal-
Notes. While the global reward landscape in deep learning is highly non-concave, these assump- tions are reasonable in our case for analyzing the behavior of models in the vicinity of local opti- mum found during fine-tuning. (1) L-Smoothhess: The reward functions Radv and Preal described in section B.2 are constructed from compositions of smooth operations (e.g., exponential, logarithm, norms) and the output of neural network, which is itself smooth function of its weights A. The expectation over trajectories further smooths the landscape, making the L-smoothness assumption plausible. (2) m-Strong Concavity: This is stronger assumption. However, it is common in theoret- ical analysis to assume that the reward landscape is locally strongly concave around a good solution (Kawaguchi, 2016). Furthermore, the use of regularization techniques like L2 weight decay during training explicitly adds quadratic term to the objective. which helps enforce strong concavity. Our analysis thus characterizes the suboptimality within such a well-behaved local region.
C.2.2 DERIVATION OF THE SUBOPTIMALITY GAP BOUND
The derivation proceeds in four steps.
29

---

Published as a conference paper at ICLR 2026
Step 1. We begin by relating the approximation gap the gradient norm. A standard result for an m-strongly concave function f()) with the maximum at x" that f(x*) f(z) S pm [V The user reward R (0) is linear combination of strongly concave functions and is thus itself strongly concave with constant my = (1 pymadr + pur Applying this result, we can bound the suboptimality gap by:
1 (0
R = (pp) (0))
(33)
2m
Our task now to bound the squared norm of the gradient the best interpolated point Øx.
Step 2. We decompose the user's preference gradient in the expert basis. The user gradient V R expressed as a linear combination of the expert gradients V Ri and R22. The expert rewards are defined by the linear system:
V R1 BV Rady + (1 V) R
(34)
R (1 Ralr + VV Rreal-
Solving this system for the base gradients V Radv and V Rreal yields:
V R1
(1 - (1-3)V1 V)VR Rr RVR1 - (1 B3V
V Radv
8² (1 3)2
23 1
(35)
V -(1 8) VR + BVR2
-(1 (1-3) R4 + BVR2
B2 (1- B)2
233 - 1
Substituting these into the definition of the user gradient VR = (1 11) V Rudv + P Rreat
(0)
2/ - (3VR: - (11-8)VRg)
11
23 (- 100--11 B)VR + RV))),
1
133 - n11 - 3) (O) +
[1. (1 n1)(1 B)
(36)
V (2),),
28 - R, (,,3))) = c1,
28 -
+ c2(14, 3) V R2))),
where we define the coefficients e₁ and 02 for notational simplicity.
Step Using the decomposition from Step and the triangle inequality, we can bound the gradient norm at the interpolated point:
R2 (0x)l & Vall (0:)1 + |cy (2)))))
(37)
We now bound each expert gradient term using Assumption 3. The expert reward R₁ is Lm-ohhh
with L1 = LLad + (1 - Lreal Since 01 is the maximizer of R₁, we have R (0,) = 0. Thus:
(+)) = R (0x) ) (R,( - L 1 - 01ll.
(38)
= =L, -116 + λ02- - 8111 = XL 1|0 01ll-
Similarly, Rg is Smoshh with L3 = (1 - Lodv + 3. and (22(02) 0. Thus:
(+)) (+x) (e2)))) S L2| 821l.
L2ll(1 X)61 + Aτ2- - 82ll L2 2)( - -2)1l = (1 L2
6111.
(39)
Substituting these bounds back into Fq. 37:
(x (lca Joy L2 XJ) 1|02 0 ||-
(40)
Step 4. The bound in Eq. 40 depends on the unknown optimal interpolation coefficient X. To
obtain a general bound, we find the maximum value of the term in the parenthesis over all possible [0, 1]. Let (() = L + Jc2 L2(1 - A). Since f()) a linear function of X, its maximum over the interval [0, 1] must occur one of the endpoints:
max S(A) = max (O), = max(1calT2,, TT). XE
(41)
30

---

Published as a conference paper at ICLR 2026
This gives us general upper bound on the gradient norm:
R (0x)l S max([ci]Li,,es (c2)222)110 01ll-
(42)
Finally, we substitute this into our initial inequality from Eq. 33 to arrive at the final bound on the suboptimality gap:
1 (1nx((11/41,110)2221100 0, 1111)²
R S
(43)
2mp
The final result is given by:
max (( (1--)) ( (10-(1-a)1-0)) A
2 L3,
L3
28-1
Rpp
110 01 0111
(44)
2((1 11) Mady + jemrral))
where L1 = Ladv + (1 Lraal L2 = (1 103)))ao Lav + B This concludes our derivation.
Interpretation. The final expression shows that the suboptimality of our method is quadratically dependent on the distance between the expert models in parameter space, and is modulated by the conditioning of the reward functions (ratio of smoothness to strong concavity) and the alignment between the user's preference μ and the expert design B.
C.3 ANALYTICAL COMPARISON OF WEIGHT MIXING AND OUTPUT MIXING SCHEMES
In this section, we provide a theoretical analysis comparing the performance weight mixing (ours) and trajectory mixing (output ensemble) for trajectory generation tasks. Our goal is understand the conditions under which one method is favored over the other and provide theoretical justification on the Linear Mode Connectivity (LMC) discussed in sections 3.3 and 4.3. The analysis is conducted in generic loss landscape. In line with Assumption 1, we interpret this loss function L(0) as the negative of the implicitly learned expected reward, i.e., L(0)
R(0). Therefore, the properties of the loss landscape, such as its curvature (convexity), are directly related to the geometry of the underlying reward landscape (concavity), making this analysis applicable to our problem setting.
Assumption (Loss as an Analytical Proxy for Preference Objective). The HGPO loss implicitly encourages the model's output distribution to shift towards trajectories that better align with the desired preferences. In this context, the ideal trajectory can be conceptualized not as a single ground truth, but representative point in the figh-preference region of the output space that the model is being steered towards.
Under this assumption, the conclusions drawn from the MSE-bar baseb analysis are argued to hold in principle for preference-driven optimization setting.
Lemma 1. Let [ar;0) E R be model that predicts a future trajectory for input x, parameterized by weights E Rd Consider two parameter [0, we define:
- fine-tuned models with parameters 02 and 0z. For an interpolation
1. The weight-mixed model with parameters Oa =(1 aa)1 a02, producing output [weigh((;; a) f(x;0a)
2. The output-mixed (ensemble) model producing output fen (x; a1= + (x; (x:02).
Let the per-sample loss be the Mean Squared Error (MSE), I[f. .)) expected ground-truth trajectory. The expected losses over the data distribution are denoted by Ieeight (ax) Ez,g. (weightt u)] and (n) = E (____ss,))) ,y)]. Then, the difference in expected loss
211 -1f-- where is the v12.
approximated by:
I weightt (a) II (a) a(1 - a) ( d²
w.iightt (ai) (a)- - E.r.y [+))
(45)
2
da
where Af(x) 1 (s: 02) (x; 02) is the difference in predictions between the endpoint models.
31

---

Published as a conference paper at ICLR 2026
Proof. The proof follows Wortsman et al. (2022) and is adapted under the regression framework, which is standard for motion generation (forecasting) tasks. We first establish an exact expression for the loss difference, then approximate the key term related to output difference, connect it to the curvature of the loss landscape, and finally synthesize the result. For clarity, we treat the model output
scalar; the extension to the vector case (where ||
| is used) is straightforward.
Step 1.
We begin by deriving an exact, non-approximated expression for the per-sample loss dif- ference. Using the difference of squares formula, a² b² (a b)(a + b), we have:
pweight
jeus
pwoight 8)² 2 (fons - y)²
(fm-- ((pwoight 8) ----() v)) ((f ((fweight y) + (fmm y))
(46)
(moiight ( (froiiht + fons - 22y),
(fwiight - ff y) + ( fwoightt - fans)2
This expression is exact and serves as our starting point. To obtain a simpler, more interpretable approximation, we will focus on the first-order term, (freightt fons) pont) (fens y), and approximate fons y fweight y.
Step 2. The core of the analysis lies in understanding the difference between weight-mixing and ensemble, which relates to the function's 's convexity. Let = 0g - 01 and 0, = 0 + Tô. By applying
the fundamental theorem of calculus twice, we can write an exact integral expression for the output difference (Wortsman et al., 2022):
feus (x) - fweight (z) = -f[ 8'V3f(x;00)) wa (T)dT,
(47)
where wa((T) min((1 a)T, a(1 r)) is triangular weight function and V is the Hessian of the model output with respect to its parameters 6.
Following Wortsman et al. (2022), assume that the Hessian V2f 0,)) is approximately constant
along the linear path between 0, and 01, i.e., (z)(2; 0,) ~ (x; 0a). This leads to:
fem - eweight (x) ~ (6T v (z; )6) / wa(T)dr.
(48)
a(1 - a) (x;(x;00)))
2
Therefore, the output difference we need is /weight (xx) fam (x) - (62 (=;
Our goal is to express the term involving the output Hessian, 8 V'V f5, in terms of the curvature of the loss function itself. The loss curvature along the parameter path is defined by the second derivative of the loss with respect to the interpolation parameter a, i.c., Af (f()),)))
Step 3.
d
First, we compute the first derivative using the chain rule:
Øl (f.)) da ((((0.)))) of da
d
(49)
The derivative of the MSE loss with respect to its input f is at (f The derivative of the of
model output with respect to is found using the multivariate chain rule:
df(O)) (Vof((a))
dl, = (Vof(ea))
(50)
da
da
since 0,, = 0, + aδ, so
de
5. Combining these gives the first derivative of the loss:
do
d
((f(8)).)) = (f(Ow) - 3) ())))
(51)
da
32

---

Published as a conference paper at ICLR 2026
Next, compute the second derivative. Let (f(0a) v) and v ((Vo) Then the derivatives and are:
da = da (f(0a) - -1) df((a) = (V (Vof(Oa))
du
da
(52)
du
da da ((VoI(0.))) = V2](0.)
Substituting these back into the product rule:
d²
da² (U((a))) = ( ((Vo ((Vo[(())) + u) (3'
(53)
= ((V))((.))"") + (f(0,) - v) (ā.()0))
By rearranging, we can isolate the term containing the output Hessian that appeared in Step 2:
& (f(0.) - v) - (6" Vor = de ((f(O)))))) - ((Vof(e.))*
(54)
This result forms the link between the model's output geometry and the loss landscape's 's geometry.
Step 4. We combine the results from the previous steps. Starting from the dominant first-order term of the loss difference from Step 1, and substituting the approximations from Steps 2 and 3:
tweight
Jens se (f (reiight (f y),
a(1 aa) (6' (6'V)((1;00))) (f(x;0a)- v).
2
a(1 a))
(55)
[U[(0)) v) 10'
a(1 - a)
de
da (f ((VoS(.))))
Taking the expectation over the data distribution (z, v) gives the difference in expected loss:
I, weightt (a) - I, a(1 a)
de
I. weight (a)
E.r,y
((Vo
(56)
2 da²
Finally, we make the first-order approximation (Vef(z; 0 ~ f(x; f(x; = Ax() This holds if the gradient Volf roughly constant between Og and 0z. This yields the final expression as stated in the lemma
Interpretation.
The approximation in Eq. 45 reveals a trade-off between two key factors:
a - a) d weight
2 al1-a) E [[AA(())
a(1 -
I weight
I. ens
(57)
2
da²
Term
Loss Curvature
Term 2: Prediction Difference
(1) Curvature: This term is proportional to the negative second derivative of the loss the mixed model along the linear path connecting the parameters. For weight mixing to be advantageous, this term needs be negative, which requires the loss function be convex along this path d gaigh
da² 0). This directly corresponds to the underlying reward landscape being concave. (2) Prediction
Difference: This term is proportional to the mean squared difference between the predictions of the two endpoint models. Since it is always non-negative, this term favors the ensemble. It captures the benefit of mixing diverse outputs, basic principle of ensembling. When two models produce highly dissimilar predictions (high | fl))) output mixing is more likely outperform weight mixing.
LMC suggests that solutions fine-tuned from a common pretrained model lie within a wide and low-loss basin. However, the path connecting two distinct experts is not perfectly zero-curvature. Instead, it can be conceptualized as a high-reward ridge. As the model traverses this path from expert to another, the combined reward often has arc-like shape, which signifies reward
33

---

Published as a conference paper at ICLR 2026
concavity. The reward landscape in Fig. 6(c) illustrates this high-reward plateau, and the measured performance in Fig. 6(d) provides direct empirical evidence of this concavity, showing that the rewards of interpolated models consistently higher than linear combination of the endpoint rewards. This inherent concavity of the reward path ensures the loss convex, making Term negative and significant, thereby providing a principled reason for the superiority of weight-space mixing in the fine-tuning regime.
This analysis provides a theoretical basis for why weight mixing is effective, particularly in the fine-tuning regime, where the loss landscape's curvature can make the first term dominant. By training on combined rewards, we ensure our expert models Pad and Greal exist in space where linear interpolation is meaningful for generating intermediate behaviors and desired trade-offs. To help better understand Eq. 57, the following, we provide an intuitive illustration to formalize the exact condition where Term 1 can dominate.
Remark (Remark the Dominance of Loss Curvature Benefit). Based on Eq. 57, weight mixing is superior E (x d' Livats des? To make this condition more intuitive, we formalize it under a local quadratic approximation of the loss landscape. Similar to section C.I, we assume that in the local vicinity of the expert solutions 0 and Oz, the loss function L(0) can be approximated by quadratic form with L(0)= Rs 0110- - 0²1 + C, where 0* is the minimum of the loss basin, 'I > is scalur representing the curvature of the basin, and C is a constant.
The right-hand hand side represents the curvature of the loss along the linear path 0g = (1 (1-a))f a:)0 + +082.
We can compute its second derivative with respect to
dlLweight
d8a
(((0a)
n10a -0) -01),
do
do
d Lweight
n702 - 01) (02 01) = =1|102- -0111
da²
The left-hand side is the expected squared difference in outputs. We can approximate this difference using a first-order Taylor expansion of the model function f around 0.:
ff fax; 02) fx; 01) se (z; (12 01).
where O(x;0) = 81 is the Jacobian of the model's output with respect to its parameters. The Jg
expected squared difference is then:
E [[AA(()) = [[] (02 - 01)
=(2 01) [E[ (xr; 01) Jzr 01)
= -02-0,) TO F(2 - 01).
where F is the Fisher information matrix, which measures the sensitivity of the model's 's output to changes in its parameters. By substituting these results back, we obtain the final condition:
(02 -F) - 01) n/ 02 01112.
(02 01) F F(2 - 01)
(58)
002 1
This derivation yields an intuitive condition: weight mixing is superior whenever the intrinsic cur- vature of the loss landscape exceeds the sensitivity of the model's output in the direction connecting the experts. This condition is readily satisfied in our fine-tuning setting, On one hand, LMC implies that fine-tuned models develop robust representations, making their outputs relatively insensitive to parameter changes along the solution path. On the other hand, because the experts are trained on conflicting objectives, the path connecting them has a non-zero curvature (n > 0) reflect the trade-off in rewards, which provides the necessary signal for alignment.
This theoretical insight is directly justified by our empirical findings in Fig. 6. The superiority of weight mixing requires the reward function to be concave along the interpolation path. Fig. 6(d) pro- vides direct visual proof, showing the measured reward curves of interpolated models lying strictly above the linear interpolation line. This confirms that the reward landscape possesses the required curvature. This view also reconciles with LMC: the path connecting experts is not a completely flat plateau but high-reward ridge (Fig. 6(c)). While LMC ensures the solution remains on this low-loss ridge, our analysis shows that it is the ridge's 's concave curvature that allows weight-space interpolation discover superior trade-offs.
34

---

Published as a conference paper at ICLR 2026
D SUPPLEMENTARY RESULTS
D.1 ADVERSARIAL GENERATION EVALUATION AGAINST IDM AND RULE-BASED POLICIES
In addition to the Replay and RL policies presented in the main paper, we provide comprehensive benchmark results against two other common types of policies: rule-based autopilot and the Intel- ligent Driver Model (IDM). Tab. 8 and Tab. 9 show that the trends observed previously hold. SAGE consistently achieves the best trade-off, delivering competitive adversarial performance while main- taining unparalleled realism and map compliance. Specifically, the realism and map compliance penalties of our generated scenarios are an order of magnitude lower than most baselines, under- scoring the general applicability and effectiveness of our HGPO framework across variety of ego agent behaviors. Without the elaborated design of reward functions, some baselines can generate unexpected and unstable behaviors, which can violate basic traffic rules (see Fig. 8).
Table 8: Evaluation of adversarial generation methods against the Rule-basedd/Autopilo policy,
Attack Succ.
Adv.
Real. Pen. +
Map Comp. +
Dist. Diff. (WD)
Methods
Rate
Reward
Behav.
Kine.
Crash Ohj. Cross Line Accel.
Vel.
Yaw
Rule
24. 18%
0,724
2.623
111.184
1.658
6,386
7.631
10.572
0.248
CAT (Zhang ul., 2023)
18.21%
0.659
8.346
3,071
2.799
9.647
1,565
7.781
0.201
KING (Hanselmann al., 2022)
16.43%
0.558
2.445
2.565
3.116
5.807
0.956
256.921
0.097
Adv TrajOptt (Zhang al., 2022)
19.00%
0.543
4.445
2.818
2.567
10.833
1.753
6.207
0.271
SEAL (Stoler et al., 2025)
19.45%
1,584
4,902
2.584
3.260
9.178
1.494
6.756
0,248
GOOSE (Ransiek et al., 2024a)
5.10%
0.325
2.387
14.56
4.363
14.31
5.322
8,107
0.167
SAGE (Wach 0.0)
4.89%
0.397
0.332
2.000
0.707
0.951
1.460
9.332
SAGE (Wad 0.5)
11.14%
0.466
0.497
2.068
0.815
0,951
1.523
8,482
0.080
SAGE (Wad 1.0)
19.84%
0.749
1.430
2.480
0.788
1.087
2.100
8.100
0.185
Table 9: Evaluation of adversarial generation methods against the IDM policy.
Attack Succ.
Adv.
Real, Pen. +
Map Comp. +
Dist. Diff. (WD)
Methods
Rate
Reward
Behav.
Kine.
Crash Obj. Cross Line Accel.
Vel.
Yaw
Rule
60.54%
2.763
4.833
36.947
1.973
6.856
4.466
8.447
0.309
CAT (Zhang al., 2023)
43.48%
1.746
10.275
3.207
3.345
8.696
1.595
7.550
0.231
KING (Hansclmann al., 2022)
16.48%
1.146
2.451
2.566
3.097
5.824
0.955
257.619
0.097
Adv AdvTrujOpt (Zhang et al., 2022)
20.44%
0.963
4.528
2.797
2.561
10.354
1.750
6.173
0.270
SEAL (Stoler et al., 2025)
33.70%
0.942
5.785
2.719
3.562
11,233
1.695
7.368
0.276
GOOSE (Ransick et al., 2024a)
16.39%
0.439
5.092
11.25
3.722
12.64
3.716
8.223
0.157
SAGE (1 0.0)
13.90%
0.864
0.332
2.000
0.708
0.954
1.461
9.318
0.055
SAGE (1asv 0.5)
20.98%
1.183
0.498
2.068
0.817
0.954
1.522
8.458
0.080
SAGE (1aa 1.0)
33.51%
1.779
1.433
2.480
0.790
1.090
2.098
8.070
0.585
Rule
KING
Ed Pl
T,
T
Adversarial Vehicle
Ego Vehicle
Vehicle
Figure 8: Unexpected case of KING and Rule baselines.
35

---

Published as a conference paper at ICLR 2026
D.2 FULL RESULTS OF CLOSED-LOOP RL TRAINING
Tab. 10 provides the full evaluation results for the trained RL policies in normal driving environ- ments, sourced from the WOMD log-replay data. We also display the training trajectories of differ- ent methods on all metrics in Fig. 9. As can be seen, the agent trained with our adversarial generator not only demonstrates superior robustness in adversarial settings (as shown in Tab. in the main paper and Fig. 9) but also achieves the highest reward, route completion, and average speed in these standard scenarios. This result is critical as it shows our curriculum- based adversarial training en- hances the agent's s general driving capabilities, rather than causing it to overfit to rare edge cases at the expense of normal performance. This alleviates catastrophic forgetting, which is a key advantage of our closed-loop training framework.
Reward (Norra)
Cost (Hamma
Route Completon (Noraa)
Crath Rate (iemaa)
New
548-
0/0
co
645
600
OM
cra-
Reward (Adrersatta))
Cost (Adrerantt)
Route Completio (Adreratt)
Crash Taln (Acheraaraa)
Beaty Joino)
300
65
80
40
40%
Figure 9: Training performances of the agent under different scenario generation methods.
Table 10: Evaluation of Trained RL Policies in the Log-replay (Normal, WOMD) Environments.
Methods
Reward
Cost
Compl.
Coll. 4
Ave. Speed f
Ave. Jerk 4
SAGE
51.99 + 1.22
0.48 0.05 0.77 + 0.02
0.16 + 0.05
9.27 + 0.03
24.97 + 0.53
CAT
46.81 + 4.33
0.50 0.05
0.67 + 0.02
0.18 0.05
7.21 = 0.05
28.15 + 1.06
Replay (No Adv)
50.16 = 5.32
0.50
0.07
0.72 0.04
0.23 0.02
9.03
0.08
27.53 0.98
Rule-based Adv
44.61 1 3.88
0.52 0.05
0.63 1 0.04
0.13 + 0.00
6.00 1 0.10
28.22 1 1.44
D.3 FULL RESULTS OF ABLATION STUDIES
To evaluate the effectiveness of our design, we evaluate the performance of the following variations:
1. w/o map penalty: The hierarchical map feasibility preconditioning procedure in section 3.2 is removed;
2. w/o realism penalty: The realism penalty function is removed;
3. w/ weighted map penalty: The hierarchical map fcasibility preconditioning procedure is replaced by direct weighted map penalty, i.e., Eq. 2;
4. w/ DPO: We replace the proposed HGPO with the standard DPO for fine-tuning:
5. The impact of reward margin 8m in HGPO: we report the training performances under different 6m values;
6. We also report the performance of separately trained models with different expert weights.
Tab. 11 presents quantitative results for ablation studies. The "w/o Map Pen." variant shows dra- matic increase in map compliance penalties (Crash Obj. and Cross Line), confirming the necessity of explicitly handling map constraints. The "w/ Weighted Map Pen." variant improves over having no penalty but is still inferior to our hierarchical conditioning approach, yielding lower adversarial rewards and higher map violations. This reinforces the core idea of HGPO that separating hard
36

---

Published as a conference paper at ICLR 2026
(a) Impact of 8 on Sample Pairs
(b) Impact of on Trajectory Reward
Delta (8)
Aug. Parrs Sceeaaioo
10
8e0
6m0,2
Best Tral, Total Reward
40-
0-
8-0.5
6-
30-
Della (8)
7-
20-
8-0.2
8-0.5
a=1
10-
6m2
20
80
80
100
0
20
40
50
100
Training Steps
Training Steps
Figure 10: Ablation studies on reward margin 8m-
feasibility constraints from soft preferences simplifies the learning problem and leads to better out- comes. Comparing our steerable model to separately trained models with fixed scalarized weights ("Scalar. Weight" rows) shows that interpolation approach effectively span the performance range of multiple individually trained models, highlighting its great flexibility.
Furthermore, Fig. 10 investigates the impact of the reward margin 8m Fig. 10(a) shows that as dn increases, the number of generated preference pairs per scenario decreases, as a larger reward differ- ence is required to form valid pair. More importantly, Fig. 10(b) illustrates the impact on learning performance. The setting with om = O results in the poorest performance. confirming that learning from pairs with insignificant reward differences introduces noise and impedes the training process. Conversely, setting moderate margin (8., in [0.2, 1.0I) significantly improves performance by focusing the model on more distinct and informative preferences. The model shows robust perfor- mance within this range, indicating that it is not overly sensitive to this hyperparameter. We selected 6m = 0.2 for our main experiments as empirically provided the best balance between sample efficiency and final performance. An overly large margin, such 6, 2.0, can slightly degrade performance by filtering out too many useful training samples.
Table 11: Ablation study (Wadv 1.0, Wreal = 0.0).
Attack Suec.
Adv.
Real. Pen. +
Map Comp.
Methods
Rate
Reward
Behav. Kine. Crash Obj. Cross Line
w/o Map Pen.
66.40%
3.841
1.580
2.665
3.848
11.65
w/o Real. Pen.
77.78%
4.308
1.966
3.077
0.705
1.762
Weighted Map. Pen.
78.05%
4.115
1.438
2.411
1.165
1.491
Scalar. Weight (Watv 3, Whenl 7)
50.68%
2.592
0.444
2.071
0.785
0.952
Scalar. Weight (Wadv 5, Whent = 5)
57.99%
3.047
0.553
2.167
0.650
1.084
Scalar. Weight (Wady 7, Mpeal 3)
66.67%
3.508
0.786
2.396
0.678
1.491
SAGE (Wady 1.0)
76.15%
4.121
1.429
2.479
0.731
1.084
D.4 ADDITIONAL RESULTS OF WEIGHT EXTRAPOLATION
We compare the task vector- vector-based extrapolation scheme (sections B.3 and 3.3) with a more naive approach of directly extrapolating the model weights, i.e., dcu = x for X & [0. 1]. As shown in Fig. 1 1(b),, this direct weight extrapolation quickly leads to unstable models and degraded performance, as evidenced by the sharp drop in preference rewards outside the [0, 1] interpolation range. In contrast, Fig. (a) shows that the task vector-based extrapolation provides a much more stable and effective means of pushing the model's behavior beyond its trained distribution. By adding multiples of the learned preference vectors, can controllably increase both adversarial reward and realism penalty in their respective directions, demonstrating that this method correctly identifies and amplifies the underlying skill vectors within the model's parameter space.
37

---

Published as a conference paper at ICLR 2026
(a) Task Vector-based Weight Extrapolation
Besiam Faenaly advvoj
Advemana Remerd Meveed
(b) Direct Weight Extrapolation
Reward Landscape Interpolated Models
Poultan Waggt Miool
03 6s a1 d+
Moan PTO Profere ROWOO
Fataly
Adherann Reoond MOUO
Figure 11: Comparison of vector-based extrapolation and direct weight extrapolation.
D.5 ADDITIONAL CASE STUDIES
D.5.1 EXAMPLES OF ADVERSARIAL GENERATION AGAINST DIFFERENT EGO POLICIES
Fig. 12 demonstrates the ability of SAGE generate effective adversarial behaviors against reactive policies. In addition to the fixed Replay policy, SAGE can also be adopted for discovering long- tailed events of reactive and well-deeeloped driving policies. This indicates its practical value in developing learning- or rulc-based ADS.
Replay
Autopilot
IDM
RL
T
T2
Adversarial Vehicle
Ego Vehicle
Other Vehicle
Figure 12: Examples of SAGE-generater adversarial behaviors against different ego policies.
D.5.2 MORE EXAMPLES OF GENERATED SCENARIOS
Figs. 13 and 14 provide additional qualitative comparisons between SAGE and SOTA baselines across variety of scenarios, including intersections and lane changes. These examples further illustrate the superior quality of scenarios generated by SAGE. While baselines frequently produce unrealistic behaviors, such as unnatural swerves (CAT), physically impossible braking (Rule), or trajectories that disregard lane boundaries, SAGE consistently generates adversarial behaviors that challenging, contextually appropriate, and physically plausible. For instance, in Scenario 5, SAGE generates a subtle but dangerous squeeze maneuver, whereas baselines resort to more chaotic actions.
38

---

Published as a conference paper at ICLR 2026
D.5.3 MORE CASES ON PREFERENCE CONTROLLABLE GENERATION
Fig. 15 shows more examples of the test-time controllability of SAGE. By simply increasing the ad- versarial weight from 0 to I, the generated scenarios transition from naturalistic to highly aggressive, enabling efficient larget-conditione testing of AD systems.
RAW
CAT
Rule
SAGE
T
Scenario
T2
T,
Scenario 2
T2
Scenario 2
T2
80
Scenario
lb
l
Tz
Adversarial Vehicle
Vehicle
Other Vehicle
Figure 13: Examples of generated scenarios by different methods.
39

---

Published as a conference paper at ICLR 2026
RAW
CAT
Rule
SAGE
Scenario
T2
Scenario
T2
Adversarial Véhicle
Ego Vehicle
Other Vehicle
Figure 14: Examples of generated scenarios by different methods.
Raw
Wadv 0.0
Wacv 0.5
Wadv 1.0
Adversorial Vehicle
Ego Véhicle
Adversarial Weight
Figure 15: Examples of generated scenarios (SAGE) under different adversarial weights.
40


Reviews:
[{"review_id": "fWlSCUf5d4", "reviewer": "Reviewer_mqnz", "summary": "The authors propose a method for generating adversarial scenarios for driving by interpolating weights between two experts trained on opposing preferences. In contrast to previous works, SAGE’s approach balances adversarial behavior with realism by taking advantage of Linear Mode Connectivity, which enables adversarial behavior to be explored at test-time rather via re-training. Overall, I find this approach to be novel, useful, and interesting.", "questions": "1. Eq 2. Not sure if I missed something, but I’m a bit confused at the -P_{real} term in the reward definition. “P_{real} ensures tau to adhere to the patterns of naturalistic driving; it should be statistically plausible.. “. To me, it would make sense that the objective should reward both adversarial and realistic behavior. Is P_{real} supposed to denote the probability of the trajectory being drawn from the distribution of realistic trajectories? In which case, shouldn’t the term’s sign be flipped (unless it denotes NLL)?\n2. Table 1: Are the baselines (other than IDM) trained with a similar reward objective, even if it is the naive objective corresponding to a single point on the Pareto frontier as mentioned before? \n3. (Appendix) Table 6/7: How is KING producing a distributional difference of ~256? It would be nice if authors discussed this / provided examples to demonstrate this failure mode. \n4. Figure 2: What policy is governing the ego vehicle? Tables 1 and 2 present results against two different ego vehicle policies, so I am confused about which results Figure 2 correspond to.\n5. At what point (if any) of increasing W_{adv} result in violation of map rules? Does the enforced “feasibility first” inductive bias ensure that all adversarial agents avoid violating map constraints, even if W_{adv} increases? \n6. A very common adversarial scenario where the adversarial vehicle is at fault is rear end collisions (the ego vehicle is in front). Does the adversarial policy demonstrate this behavior when the log begins with the adversarial agent behind the ego vehicle in the same lane? I noticed that a lot of the qualitative examples (Figure 11) demonstrate cut-in behavior. I was curious if the adversarial policy was mostly limited to cut-ins, or if there are other types of adversarial behavior types.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The motivation and methodology are well-written and well-justified. \n- Additional results in supplementary is helpful. \n- Authors provide very extensive details on experiment setup. Without the  code provided, I feel as though the work is reproducible based on the text.\n- Authors show theoretical and empirical applicability of LMC to traffic simulation problems.\n- Results are comprehensive and provide actionable insight to the simulation community.", "weaknesses": "- Even though a lot of information is referenced and provided in the appendices, I found it difficult to interpret results off first glance in the main body. I suggest the authors make captions of Figures and Tables more self-contained and avoid having the reader look for references. It would be nice to know off first glance 1) what policies govern adv and ego vehicles, and 2) if certain qualitative results correspond to any tabular results. For example, I found Figure 2 difficult to interpret. Even though the images show that SAGE produces more realistically adversarial behavior, I’m not exactly sure what policy the ego vehicle is following. The main body also does not seem to specify. \n- Some notation definitions are missing (see question 1 below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3l9VwzkBgy", "reviewer": "Reviewer_ZEJW", "summary": "The paper reframes adversarial scenario generation for autonomous driving as a multi-objective preference alignment problem and proposes SAGE. Contributions are: (i) Hierarchical Group-based Preference Optimization (HGPO), which decouples hard feasibility (map compliance) from soft preferences (adversariality vs. realism) by gating infeasible samples with a binary feasibility function $F(\\tau,\\mathcal M)$ and learning from group-wise, multi-pair preferences within the feasible set to improve sample efficiency; (ii) training two expert policies from a shared pretrained model (an adversarial-leaning expert and a realism-leaning expert) and interpolating their weights at test time to steer along the trade-off; and (iii) integrating SAGE into an existing closed-loop RL framework.", "questions": "1.  Baselines include Rule, CAT, KING, AdvTrajOpt, and SEAL, but omit GOOSE (2024) from related work; Is there a reason why it doesn't appear in the experiments as a baseline method?\n\n2. I wonder how diverse generated adversarial behaviors are. Could authors give visualizations or quantitative results on this perspective?\n\n3. please address issues in point 3. in weakness.\n\n4. What is the expert reward/penalty design (component weights, margins, clipping constants, and final thresholds actually used in experiments)? What is the full pre-training recipe for the shared reference model (e.g., optimizer, learning-rate schedule, batch size, training duration/epochs, data splits and augmentations) used for DenseTNT, or whether a public checkpoint was used? for HGPO fine-tuning, what are the concrete values for group size (N), pair-selection margin ($\\delta_m$), the number of candidates (K) per context, and any additional regularization or gradient-scaling coefficients?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "SAGE provides test-time steerability across the adversariality–realism trade-off, and handles feasibility by treating map rules as hard constraints. Empirically, in open-loop benchmarking, SAGE traces a better realism for a given adversarial reward and achieves lower map-violation penalties while remaining competitive in attack metrics; figures and tables exemplify smooth control as the adversarial weight increases.", "weaknesses": "1. **Novelty concerns:** The core ideas (i.e., preference alignment and weight-space interpolation between experts) may feel incremental, which explicitly build on prior literature. Please emphasize the novelty of SAGE. The key idea includes filtering out map-infeasible samples (F=0) from the preference set, and also constructing group-wise, multi-pair preferences within the feasible set and optimizing a reference-regularized HGPO objective. That said, the contribution remains incremental within the existing preference-optimization paradigm rather than a fundamentally new algorithmic framework.\n\n2. **Generation Efficiency:**  Against the Replay policy, SAGE and its variations report lower attack success rates and lower rewards than their baselines, raising questions about the effectiveness of the generated adversarial behaviors. \n\n3. **Closed-loop result discrepancy and fairness:** In the paper’s closed-loop results, CAT underperforms Replay (no adv) on completion/collision, whereas the original CAT paper typically reports gains from adversarial training. Please explain why CAT underperforms here and provide diagnostics to ensure a well-tuned baseline. Besides, CAT’s closed-loop results here appear worse than those reported in the original paper in terms of collision and completion; this gap should be reconciled as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TuBqqcapf5", "reviewer": "Reviewer_be6Z", "summary": "This research addresses limitations in existing adversarial scenario generation methods for autonomous driving, which often lack flexibility to tailor scenarios for specific needs. The authors introduce SAGE (Steerable Adversarial scenario GEnerator), a new framework that allows fine-grained control over the balance between adversariality and realism during scenario creation without retraining. By using preference optimization and interpolating expert policies, SAGE generates more effective training data and improves driving policy performance.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Walking along Pareto frontier between realism and adversarial objectives\n* Ability to do it w/o retraining\n* Correct (at least looking to be) theoretical statement about the suboptimality gap limited by weighted squared norm of weights difference from above\n* Correct (at least looking to be) theoretical statement about the advantage of weight mixing over explicit output ensemble", "weaknesses": "* Eq. (8) and lines 304-305 (+ additionally in the Appendix C.3): that's still not clear why the first term - benefit from path flatness - is the dominant one. Would be nice to see any proof, empirical or (better) theoretical one\n* It's not clear how preference labels (see \"Preference within Feasibility\", lines 196-198) are generated\n* No ablation on $\\delta_{m}$ (line 198); moreover, is it the same as $\\sigma = 0.2$ (line 902)? \n* Appendix B.2: unclear, what are values of $\\omega_{turn}$ and $\\omega_{stop-turn}$ in Eq. (13)\n* Quite unclear, why the approach is not that useful for IDM in comparison to Replay that makes it questionable to use in practice (Appendix D.1)\n* Minor remark: need to define $R_{real}$ earlier (as the negative expectation of $P_{real}$ on page 1373)\n\nOverall, the theoretical concepts brought to autonomous driving really makes the reviewed paper suitable for ICLR conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "93ujgd58Np", "reviewer": "Reviewer_jZqG", "summary": "This paper proposes a method to fine-tune a pre-trained motion generation model to generate adversarial scenarios. The post-training would result in two separate expert models excelling at demonstrating realistic behavior and adversarial behavior, respectively. Model merging by linear interpolation then allows us to control the trade-off between realism and the capability of inducing collision. The model shows good performance in the open-loop benchmark, showing a more natural behavior than other baselines with a nearly SOTA collision rate. The following RL experiments show that agents trained with the adversarial scenario generated by the proposed model achieve SOTA in both nominal scenarios and long-tail scenarios, outperforming agents trained in scenarios generated by other baselines. Further study shows the effectiveness of model merging and the post-training method HGPO.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Long-tail/near-accidental/safety-critical scenario generation is important for real-world robotics applications. In this field, adversarial policy modeling is important, which trades off the realism of the generated trajectories. The method proposed in this paper strikes a good balance between these two contradictory goals and enables controlling the extent to which the adversarial behavior should be.\n2. The experiment is comprehensive, and the results indeed demonstrate the usefulness of the proposed method. It shows the method can achieve an SOTA attack success rate while maintaining a good trajectory realism. \n3. The model merging and proposed post-training method, HGPO, is validated to be useful and plays the key role of the success of the whole pipeline.\n4. Paper writing is easy to understand, and experimental details are well-documented", "weaknesses": "In the RL experiment, the RL agents trained in scenarios generated by different methods should be benchmarked in their respective generated test splits as well. For now, SAGE/CAT/Rule-based agents are benchmarked in scenarios generated by only SAGE and log-replay. It makes me wonder whether the superior performance of the SAGE agent comes from overfitting due to the similarity between the training and testing split, as they are all generated by the SAGE method.\nIt would be better to additionally benchmark all 3 agents in scenarios generated by the CAT and Rule-based method. The best results we can expect are that the SAGE agent shows comparable even better, results than CAT agents on CAT-generated held-out scenarios. But even if it is a bit worse, the good results of SAGE agents on the log-replay benchmark can still demonstrate another benefit of SAGE: maintaining the nominal driving ability of RL trained policies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)