Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Model Using Implicit Feedback from Pre-training Demonstrations
Authors: Thomas Tian
Kratarth Goel
Venue: ICLR 2025 Spotlight (2025)
Keywords: Efficient Post-training Preference Alignment
Alignment from demonstrations
Multi-agent Motion Generation
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose an efficient post-training alignment approach that significantly improves the pre-trained motion generation model’s quality without requiring additional post-training human preference annotation or expansive compute.

Abstract: Recent advancements in Large Language Models (LLMs) have revolutionized motion generation models in embodied applications such as autonomous driving and robotic manipulation. While LLM-type auto-regressive motion generation models benefit from training scalability, there remains a discrepancy between their token prediction objectives and human preferences. As a result, models pre-trained solely with token-prediction objectives often generate behaviors that deviate from what humans would prefer, making post-training preference alignment crucial for producing human-preferred motions. Unfortunately, post-training alignment requires extensive preference rankings of motions generated by the pre-trained model, which are costly and time-consuming to annotate, especially in multi-agent motion generation settings. Recently, there has been growing interest in leveraging expert demonstrations previously used during pre-training to scalably generate preference data for post-training alignment. However, these methods often adopt an adversarial assumption, treating all pre-trained model-generated samples as unpreferred examples and relying solely on pre-training expert demonstrations to construct preferred examples. This adversarial approach overlooks the valuable signal provided by preference rankings among the model's own generations, ultimately reducing alignment effectiveness and potentially leading to misaligned behaviors. In this work, instead of treating all generated samples as equally bad, we propose a principled approach that leverages implicit preferences encoded in pre-training expert demonstrations to construct preference rankings among the pre-trained model's generations, offering more nuanced preference alignment guidance with zero human cost. We apply our approach to large-scale traffic simulation (more than 100 agents) and demonstrate its effectiveness in improving the realism of pre-trained model's generated behaviors, making a lightweight 1M motion generation model comparable to state-of-the-art large imitation-based models by relying solely on implicit feedback from pre-training demonstrations, without requiring additional post-training human preference annotations or incurring high computational costs. Furthermore, we provide an in-depth analysis of preference data scaling laws and their effects on over-optimization, offering valuable insights for future studies.

Published as a conference paper at ICLR 2025
DIRECT POST-TRAINING PREFERENCE ALIGNMENT FOR MULTI-AGENT MOTION GENERATION MODELS USING IMPLICIT FEEDBACK FROM PRE-TRAINING
DEMONSTRATIONS
Ran Tian Kratarth Goel
UC Berkeley Waymo
ABSTRACT
Recent advancements in Large Language Models (LLMs) have revolutionized motion generation models in embodied applications such as autonomous driv- ing and robotic manipulation. While LLM-type auto-regressive motion gener- ation models benefit from training scalability, there remains a discrepancy be- tween their token prediction objectives and human preferences. As result, models pre-trained solely with token-prediction objectives often generate behav- iors that deviate from what humans would prefer, making post-training prefer- ence alignment crucial for producing human-preferred motions. Unfortunately, post-training alignment requires extensive preference rankings of motions gen- erated by the pre-train pre-trained model, which are costly and time-consuming to anno- tate especially in multi-agent motion generation settings. Recently, there has been growing interest in leveraging expert demonstrations previously used dur- ing pre-training to scalably generate preference data for post-training alignment. However, these methods often adopt adversarial assumption, treating all pre- trained model-generated samples as unpreferred examples and relying solely on pre-training expert demonstrations construct preferred examples. This adver- sarial approach overlooks the valuable signal provided by preference rankings among the model's own generations, ultimately reducing aligmment effectiveness and potentially leading to misaligned behaviors. In this work, instead of treat- ing all generated samples as equally bad, we propose principled approach that leverages implicit preferences encoded in pre-training expert demonstrations construct preference rankings among the pre-trained model's generations, offer- ing more nuanced preference alignment guidance with zero human cost. We apply our approach to large-scale traffic simulation (more than 100 agents) and demonstrate its effectiveness in improving the realism of pre-trained model's gen- erated behaviors, making a lightweight IM motion generation model comparable state-of-the-a large imitation-based models by relying solely on implicit feed- back from pre-training demonstrations, without requiring additional post-train post-training human preference annotations or incurring high computational costs. Further- more, we provide an in-depth analysis of preference data scaling laws and their effects on over-optimization, offering valuable insights for future studies.
INTRODUCTION
The recent advances in Large Language Models (LLMs) (Achiam et al., 2023) have significantly im- pacted the design of motion generation models for embodied tasks such as autonomous driving (Seff et al., 2023; Philion et al., 2024), robot manipulation (Brohan et al., 2023), and humanoid locomo- tion (Radosavovic et al., 2024). Formulating motion generation as a next-token prediction task not only provides unified framework for modeling sequential decision-makinn tasks but also provides opportunities for leveraging pre-trat mre-trained LLMs for more cost-effective training and improved gen- cralizability (Tian et al., 2024). However, despite the remarkable progress, robots relying on these large poken-prediction models do not automatically become better at doing what humans prefer due

---

Published as a conference paper at ICLR 2025
Pre-training
Direct Preference Alignment
Preference Ranking via
Occupancy Measure Matching Feedbbckk
max (
X
Exgert dame
P(ani)
(c.c) ... P(mp:Rer.e)
Multi-modal
Scene Uncoder
Motion Token Prediction Model
Br
Sample ple rollouts
Sampled rollects
Figure I: Direct preference alignment from occupancy measure matching feedback for realis- tic traffic simulation. DPA-OMF is a simple yet effective alignment-from-cee -ro-mmotttttttttttttto approach that aligns a pre-trained traffic simulation model with human preferences. It defines an implicit preference distance function that measures the alignment between generated sample and an expert demonstration in the same scene context through occupancy measure matching. This distance is then used to rank the reference model's generated samples for each training scene contex, enabling large- scale automatic preference data generation to align the motion generation model. The gray dotted lines above the motion token prediction model indicate the reference model's motion token distri- butions at each prediction step, and the orange lines represent the probabilities after the alignment process. å denotes agents' action tokens sampled from the predicted distribution P(ay c, ât-1) during inference time, denotes the scene context representation (more details Section 3.1).
to the misalignment between the training objective and the underlying reward function that incen- tivizes expert demonstrations. This discrepancy underscores the challenge of ensuring that motion models trained with next-token prediction are effectively aligned with expert-preferred behaviors.
Preference-based alignment has emerged crucial component in the LLM post-training stage, aiming to reconcile the disparity between the next-token prediction objective and human prefer- ences. Among various frameworks, direct alignment algorithms (e.g.. direct preference optimiza- tion (Rafailov et al., 2024b)) are particularly appealing due to their training simplicity and compu- tational efficiency. Specifically, these algorithms collect human preferences over pre-trained model generations and directly update the model to maximize the likelihood of preferred behaviors over unpreferred ones. However, complex embodied settings, such as joint motion generation involv- ing hundreds of agents, obtaining such preference data at scale can be very challenging. Human annotators must analyze intricate and nuanced motions, which is time-consuming process, making the scalability of direct alignment methods difficult in these scenarios.
While soliciting rankings from experts provides explicit preference information, we argue that expert demonstrations used in the pre-training stage inherently encode implicit human preferences, which can be reused to align a pre-trained motion generation model in a cost-effective way, beyond their role in supervised pre-training. Recently, Alignment from Demonstrations (AFD) (Li et al., 2024; Sun & van der Schaar, 2024; Chen et al., 2024b) has emerged as a valuable technique for auto- matically generating preference data using pre-training expert demonstrations, allowing preference alignment to scale at a low cost. However, previous methods typically assume a worst-case scenario: treating all pre-trained model-generated samples as unpreferred and relying solely on pre-training expert demonstrations to construct preferred examples (Chen et al., 2024b; Sun & van der Schaar, 2024). This adversarial approach overlooks the valuable signal provided by preference rankings among the model's 's own generations, ultimately reducing alignment effectiveness and potentially leading to misaligned behaviors.
Instead of treating all generated samples as equally bud, propose leveraging the implicit prefer- ences encoded in pre-training demonstrations automatically construct preference rankings among the pre-tran pre-trained model's 's generations, providing nuanced guidance with zero human cost.
Our approach draws inspiration from inverse reinforcement learning (Abbeel & Ng, 2004; Ho & Ermon, 2016), where alignment between generated behavior and the expert demonstration is mea- sured through occupancy measure matching. We propose Direct Preference Alignment from Oc- cupancy Measure Matching Feedback (DPA-OMF), principled approach using optimal transport to define an implicit preference distance function. This function measures the alignment between a generated sample and the expert demonstration through occupancy measure matching in seman-
2

---

Published as a conference paper at ICLR 2025
tically meaningful feature space, and is then used to rank the generated samples according their alignment with expert demonstrations, producing more nuanced preference data at scale to align the motion generation model (Figure 1).
We apply our approach to large-scale traffic simulation (more than 100 agents) and demonstrate its effectiveness in improving the realism of pre-trained model's generated behaviors, making lightweight IM motion generation model comparable to state-of-the-art large imitation- -baeed mod- els by relying solely on implicit feedback from pre-training demonstrations, without requiring addi- tional post-training human preference annotations or incurring high computational costs.
Contribution. In this work, we consider the problem of efficient post-training alignment of a token-pred prediction model for multi-agent motion generation. We propose Direct Prefer- ence Aligmment from Occupancy Measure Matching Feedback (DPA-OMF), a simple yet principled approach that leverages pre-training expert demonstrations to generate implicit preference feedback and significantly improves the pre-trained model's generation quality without additional post-training human preference annotation, reward learning, or complex reinforcement learning. To the best of our knowledge, this is the first work to demonstrate the benefits of preference alignment for large-scale multi-agent motion generations using implicit feedback from pre-training demonstrations. Additionally, we provide a detailed analysis of preference data scaling laws and their impact preference over-optimization.
2 RELATED WORKS
Scaling alignment using AI feedback. Previous works have proposed leveraging synthetic Al feed- back to scale preference data LLM applications, either from a single model (Bai et al., 2022; Lee et al.; Mu et al.), ensemble of teacher models (Tunstall et al., 2023), through targeted context prompting, where the model is instructed to generate both good and bad outputs for comparison (Yang et al., 2023; Liu et al., 2024). Unfortunately, these frameworks are not directly applicable to embodied contexts, such as autonomous driving, due to the absence of high-quality, open-source, and ioput-unified foundation models. What makes input-unified foundation models especially chal- lenging is that different embodied models often use incompatible input modalities or features, mak- ing it difficult to transfer feedback across models effectively.
Alignment from expert demonstrations. Alignment from Demonstrations (AFD) has emerged as a valuable technique for automatically generating preference data using expert demonstrations, enabling preference alignment to scale effectively. For example, Chen et al. (2024b) fine-tunes a pre-trained reference model in self-play manner, where the optimized model maximizes the log- likelihood ratio between expert demonstrations and self-generated samples. However, this method treats all generate samples as unpreferred, overlooking the valuable information embedded in the preference rankings among those samples. Additionally, it suffers from bias introduced by
the heterogeneity of the preference data: since the preference data are drawn from two different sources (human optimized model), the discrimination objective may emphasize the differences between models rather than focusing on the key aspects that truly evaluate the quality of the gen- erated behaviors. To address the heterogeneity issue, Sun & van der Schaar (2024) proposes using expert demonstrations to first fine-tune the reference model through supervised learning, creating an expert model. Then, samples generated by the fine-tuned expert model are treated as positive examples, while samples from the initial model are treated as negative examples to construct pref- erence dataset. While this approach helps mitigate the heterogeneity problem, it requires training an additional expert model, and there is no guarantee that all generations from the expert model will consistently be superior to those from the initial model. Unlike previous works, our approach constructs preference rankings among the generations produced by the reference model, effectively addressing the heterogeneity issue while providing more nuanced guidance at a lower cost.
Motion alignment via divergence minimization. Adversarial imitation learning (IL.) aims to align the behavior of the learning agent with expert demonstrations by minimizing the Jensen-Shannon divergence between the agent's action occupancy measure and that of the expert (Ho & Ermon, 2016; Song et al., 2018). These methods jointly train a policy model and a discriminator: the policy model is trained to imitate expert demonstrations, while the discriminator is trained to separate the generated samples from the expert demonstrations and informs the policy model. Additionally, ad- versarial training is known for its instability and high computational cost (Kodali et al., 2017; Yang
3

---

Published as a conference paper at ICLR 2025
et al., 2022), making it particularly problematic in post-training alignment, where stability and com- putational efficiency are crucial. Similar to adversarial IL. we employ occupancy measure matching to assess the alignment between generated samples and expert demonstrations. However, our ap- proach differs by using preference rankings over the generated samples to inform the alignment process, rather than relying on a signal learned from separating all generated samples from expert demonstrations. Our approach provides more informative guidance while also being significantly more computationally efficient and more suitable for post-training alignment.
Preference alignment for realistic traffic simulation. Preference-basee alignment has recently been used enhance the realism of traffic generation (Cao et al., 2024; Wang et al., 2024b). How- ever, these previous works rely on Reinforcement Learning with Human Feedback (RLHF) as the alignment approach and learn rewards from low-fidelity simulator data. In contrast, our work seeks to improve traffic generation models without the need for reinforcement learning or explicit reward learning. Instead, we utilize an implicit preference distance derived from real human demonstrations to guide the alignment, and demonstrate improvements in a much larger scale.
3 EFFICIENT POST- TRAINIGG PREFERENCE ALIGNMENT FOR MULTI-AGENT MOTION GENERATION
3.1 PRE-TRAINING OF MULTI-AGENT MOTION GENERATION MODEL
We assume access to a set of expert demonstrations, De, where each example consists of joint action sequence { = {ao..... .,. a and a scene context representation c. The joint action sequence represents the behaviors of N agents over generation horizon I, conditioned on the given scene context c. Each joint action at time step is a collection action tokens for all N agents, denoted a, (a. ..... a."). Following Seff et al. (2023), our learning objective is train generative model parameterized by 8, which maximizes the likelihood of the ground truth joint action at time step conditioned on all previous joint actions and the initial scene context:
max (E,e) [T (ac))))))
where To Ta(arla == P(a a1-1..... an; c). We denote the model pre-trained with the above token prediction objective as the reference model Tref- Note that the optimal solution to (1) cor- responds to finding a policy that minimizes the forward Kullback- the expert policy: ming KL(Te) TT). As a result, the learned policy typically exhibits mass-covering behavior (Wang et al., 2024a), making it prone to deviations from the expert's 's true policy. This misalignment necessitates post-training preference alignment.
(KL) divergence from
3.2 IMPLICIT PREFERENCE FEEDBACK FROM PRE-TRAINING DEMONSTRATIONS
In this section, we introduce our approach to leveraging pre- pre-traning expert demonstrations for scal- able preference feedback generation. The key idea is to define an implicit preference distance func- tion that measures the alignment between generated sample and the expert demonstration given the same scene context. This distance is then used to rank the reference model's generated samples for each training scene context.
Specifically, we aim to find a preference distance function d that approximately measures the preference-based notion of similarity over triplet (Ee, çí <2), where & represents pre-training expert demonstration and E.d are two rollouts sampled from the reference model given the scene context of &e. We treat < as an anchor; if the preference distance between ç and & smaller than that between E and &.. then & is considered more aligned with the anchor than E and is thus more preferred according to the expert preference encoded fa: d(Ee) > d(f.. = & x
Fundamental work in IRL (Abbeel & Ng, 2004) advocates for using occupancy measure matching to assess the alignment between the policy induced by a recovered reward function and human preference. A policy whose occupancy measure closely matches that of a human is considered better aligned with the human's 's true preferences Building on this insight, we leverage the Optimal Transport (OT) method (Villami al., 2009) principled approach to define an implicit preference distance function that measures the occupancy measure matching between a policy rollout and the expert demonstration. We then use this preference distance to rank rollouts from the reference
4

---

sampled from the reference Published as a conference paper at ICLR 2025
model to construct the preference dataset. OT has been successfully used to measure alignment between behaviors in prior single-agent reinforcement learning works (Xiao et al., 2019) with the key difference in our work being used in multi-agent settings to fine-tune generative model without the need for reinforcement learning (see Q4 in Appendix for more details).
Occupancy measure. Let {oe) represent sequence of scene observations obtained by rolling out a joint action sequence Ç in the initial scene e over time steps. The empirical occupancy measure associated with (ç. c) is a discrete probability measure defined as P(C.c) = Ei=o oo(o))) where (o,)) is Dirac distribution centered on (0:) [ ..... o( )]. Here, Φ is feature encoder that maps each agent's s information into feature vector. Intuitively, a policy rollout oceu- pancy measure represents the distribution of features visited by the generation policy throughout the generation horizon.
Implicit preference distance. Optimal transport quantifies the distance between occupancy measures by solving the optimal coupling μ E RRXT that transports rollout occupancy measure, P(e..)) to the expert occupancy measure, P(E..)), with minimal cost. Instead of computing the scene- level optimal coupling, we compute the agent-level optimal coupling and then aggregate them for computing the scene-level alignment, assuming the generative model only generates the behaviors of pre-defined agents but not insert nor remove agents from the scene. Specifically, for each agent i, we compute optimal coupling between agent i's empirical occupancy measure induced by the sampled rollout ç and agent i's empirical occupancy measure from the demonstration č by minimizing the Wasserstein distance between the two:
T ΣΣ 1=1=1
T
arg min MEMiNICs
(lo
(2)
P(e.e)) is the of coupling matrices, é I x R is a cost function defined on the support of the measure (e.g.. cosine distance), and n is the dimension of the feature space. This gives rise to the following distance that measures the preference-base notion of similarity between rollout and the expert demonstration:
where M(Prc ((CL,))
E TxT
1d = Piet.)),
N
T
d(Es, Eei =
Preference data curation. With the reference model pre-trained using the next-token prediction objective (1), we construct a rollout set, Drigi consisting of generated motions sampled from the model. Each example in Dr includes pre-training scene context (drawn from xx.), the corre- sponding expert demonstration. and set of K rollouts, { <*) model given e. We use the implicit preference distance function defined in (3) to rank the K rollouts of each training example in Die,e and construct the preference dataset D Typer- Each example in Diper contains N Nped pairwise comparisons, (g* { "......Npr along with the associated scene context We note that this preference dataset is constructed without any human annotations, relying solely on pre-training expert demonstrations.
3.3 DIRECT PREFERENCE ALIGNMENT VIA CONTRASTIVE PREFERENCE LEARNING
Using the automatically constructed preference dataset D Di Tiee we apply a multi-agent extension of the contrastive preference learning algorithm (Hejna et al., 2023) directly fine-tune the reference motion generation model:
Ta(nī (at marla, (a2(.e)
exp E, a'alog
max Eig Doer
log exp E; a log (o( (nt + exp E, a log (a/ [
(4)
no(a; (n Tarla (
4 EXPERIMENT DESIGN
Realistic traffic scene generation. We validate our approach in the large-scale realistic traffic scene generation challenge (WOSAC), where the model is tasked with generating eight seconds of realistic
5

---

Published as a conference paper at ICLR 2025
Aligned
Partially aligned
Misaligned
Sample rollouts
Reference Multi- Agent Motion Generation Model
Ground truth
d(t.. (,) = 0.38
d(f.. L.82
d(s.. 4.69
Figure 2: Alignment visualization. The heat map visualizes the optimal coupling between a gener- ated traffic simulation and the ground truth scene evolution. More peaks along the diagonal indicate better alignment between the behaviors (i.e., a smaller preference distance). The traffic simulation with a small preference distance (left) shows behavior that is well-aligned with the ground truth, while the simulation with a larger distance (right) exhibits inconsistencies, such as the pedestrian's 's generated motion colliding with an oncoming vehicle.
interactions among multiple heterogeneous agents (up to 128) 10 Hz, based on one second of past observations (Montali et al., 2024). The WOSAC challenge uses realism metric to evaluate the quality of the traffic generation model. The realism of a generative model is defined as the empirical negative log likelihood of the ground truth scene evolution under the distribution induced by 32 scene rollouts sampled from that model (binned into discrete histograms).
Token prediction model and preference dataset construction. We use the MotionLM model (Seff et al., 2023) as our generation model (IM trainable parameters) (more details Appendix C) and first train it using the tokent prediction objective the Waymo Open Motion Dataset for 1.2M steps to obtain a reference model. For each training example, we sample 64 rollouts from the reference model and rank them using the preference distance function. The 16 closest rollouts are treated as preferred samples, while the 16 farthest are considered unpreferred, constructing 16 comparisons per example. Following previous work on learning rewards for autonomous driving (Sun et al., 2018; Chen et al., 2024a), we use features that capture the modeled agent's 's safety, com- fort, and progress to encode the agent's 's state information at each time step when building the rollout occupancy measure, including: [collision status, distance to road boundary, minimum clearance to other road users, control effort, speed]. When solving the optimal transport plan (2), we use the L2 cost between features with the following weights: [10, 5, 2, 1, I]. These features are also used to encode the agent's state in the realism metric. It is important to note that the preference distance not the same as the realism metric. The preference distance measures the alignment between roll- out and the expert demonstration, whereas the realism metric assesses the likelihood of the expert demonstration given all the rollouts.
RESULTS
5.1 ON THE VALIDITY OF THE IMPLICIT PREFERENCE DISTANCE FUNCTION
In this section, we qualitatively and quantitatively investigate the implicit preference distance reflects the behavior alignment (i.e., smaller distance means more aligned behavior).
Qualitative example. In Figure 2, we present a qualitative example illustrating how the coupling matrix reflects the behavior alignment between generated traffic simulations and expert demonstra- tions. Traffic simulations with smaller preference distances exhibit behavior more closely aligned with the expert demonstrations. Additional qualitative examples can be found in the AppendixD.
To quantitatively evaluate our preference distance function in a controlled experiment, we conduct post-selection analysis, where select sampled traffic simulations from the reference model and analyze the relationship between the realism of these samples and their group-averaged distance to the expert demonstration. We use the Average Displacement Error (ADE) (1.2 distance
Baseline.

---

Published as a conference paper at ICLR 2025
between sampled traffic simulation and the expert demonstration) baseline, which is a com- monly used distance for evaluating trajectory generation performance in autonomous driving.
Experiment setup. We control the distance the expert demonstration, and measure the realism of the selected sampled traffic simulations. Specifically, we first sample 128 traffic simulations from the pre-trained reference model, rank them using the candidate distance (ADE preference distance), and then select traffic simulations as the final model output using a sliding window of size 32. For example, model variant I outputs the top 32 ranked traffic simulations, model vari- ant outputs simulations ranked from to 33, and so on. We then measure the corresponding WOSAC realism of each model variant to study its relationship each candidate distance metric.
Results. In Figure 3, we illustrate the relationship be- tween the WOSAC realism of selected traffic simulations and their croup-averager distance from the expert demon- stration. We observe that ADE is informative in reflecting behavior alignment only up to certain point: initially, as ADE decreases, the realism improves. However, once the traffic simulations reach reasonable level of realism, further reductions in ADE do not result in significant im- provements in the realism. contrast, the preference dis- tance function correlates more strongly with the realism and demonstrates better effective range when using the same reference model. This highlights its effectiveness in measuring the alignment between a generated traffic sim- ulation and the expert demonstration.
Group Ave. Preference Dist.
Group Ave. ADE
4.5
3.5
3
3.5
2.5
1.5
0.54
0.66
Realism
Relationship between the WOSAC realism of selected traffic sim- ulations and their group-averaged dis- tance from the expert demonstration.
Figure 3:
5.2 ON THE VALUE OF PREFERENCE ALIGNMENT
In this section, compare approach with various post-training alignment methods demon- strate its effectiveness. Our experiments focus on evaluating the impact of two factors: 1) different feedback signals, and 2) the alignment approach.
Baseline. We compare DPA-OMF against (1) DPA- ADEF, which constructs preference rankings using ADE as the distance function to rank traffic simulations sam- pled from the reference model, then fine-tunes the refer- ence model using (4); (2) SFT-bestOA, which selects the top 32 ranked traffic simulations, measured by the prefer- ence distance, from the reference model and uses them as new labels for supervised fine-tuning, aiming to di- rectly distill the preference into the reference model; (3) SFT-bestADE, similar to SFT-bestOA but use the ADE to pick the top ranked samples:(4) We also list the per- formance of SOTA models that are typically much larger with sophisticated designs. We use the same reference model for post-training alignment and fine-tune for 200k steps in all experiments.
SPT bestOt prof. SFT-beotDA-unpee
OMP
DPA-OMI-EE PAN--OE-UUmpp
-225
-230
log (P(E.)
-235
240
-245
-250
-255
50
100
150
200
steps (x100)
Figure 4: The log-likelihood of pre- ferred/unppeferreed rollouts from the ref- erence model when using DPA-OMF versus SFT-bestOA for alignment.
Results. We present the quantitative evaluation of each method's realism and trajectory L2 error Table DPA- OMF significantly outperforms all baselines in terms of alignment between the generated traffic simulations and the expert demonstrations, while the baselines struggle to improve- and in some cases, even degrade (DPA-ADEF and (FT-bestADE)-the realism. Interestingly, SFT-bestOA does not seem to improve the realistic metric too much although it is using the same preferred traffic simulations learning signals just like DPA-OMF, Figure 4 sheds more light on this. In Figure 4, of both preferred and unpreferred traffic simulations geenrated
we show the Segative-loglikelibee from the reference model when using DPA-OMF and SFT-bestOA to align the model. We that the likelihood of preferred traffic simulations is increasing and the likelihood of unpreferred traffic simulations is decreasing when using DPA-OMF to align the model, while the likelihood
7

---

Published as a conference paper at ICLR 2025
Expert
DPA-OMP((Uns)
Ref. motion
APAE
SFT-besOA
SFT-beRADE
Expert
DPA-OMF(Oum
Ref mutionLM
DPA-ADEF
SFT
SFT-bestADE
Figure 5: Traffic simulation generation visualization. Our approach produces traffic simulations that are more closely aligned with expert demonstrations, while baseline models generate simula- tions that are only partially aligned misaligned (highlighted in red texts).
of unpreferred traffic simulations is increasing when using SFT-bestOA align the model. This may be due to the fact that both the preferred and the unpreferred samples are from the same dis- tribution. This demonstrates the importance of explicitly considering the negative signals when aligning the model. In Figure 5, we present a qualitative visualization of the generated traffic sim- ulations from each approach. We sample 64 rollouts from each model and display the most likely generation. Our approach produces traffic simulations that are more closely aligned with expert demonstrations, whereas the baseline models generate simulations that are only partially aligned or misaligned. While approach improves the reference model through cost-effective preference alignment, it still falls short a bit compared to some SOTA methods. provide more discussions in Q3 of Appendix A.
Method
Kinematic
Interactive?
Map compliance?
Composite realism?
minADE
MotionLM (IM reference model)
0.417
0.778
0.815
0.721
1.398
DPA-ADEF
0.393
0.780
0.812
0.714
1.379
SFT-hestADE
0.406
0.773
0.816
0.715
1.392
SFT-bestOA
0.410
0.781
0.826
0.723
1.428
DPA-OME (ours)
0.415
0.786
0.867
0.739
1.413
BehaviorGPT/Zhon et al., 2024) (3M)
0.433
0.799
0.859
0.747
1.415
Trajeglish(Philion al.. 2024) (35M)
0.415
0.786
0.867
0.721
1.544
SMART(Wu et al., 2024) (102M)
0.479
0.806
0.864
0.761
1.372
Table 1: Realism of different methods. Our approach improves the realism of the reference model
(shaded in gray) without requiring additional human-cost, reward learning reinforcement learning.
On the importance of features. Despite the OT-based preference score's effectiveness in measuring the alignment between generated traffic simulation and the expert demonstration, the relationship between the OT-based preference score and human preference is not strictly monotonic. This re- lationship heavily depends on the features used to compute the rollout occupancy measure. We consider a set of features commonly used in IRL research to compute the preference distance (de- scribed in Section 4). In Table 2, we present an ablation study analyzing the effect each feature on the effectiveness of the approach. Although the best performance is achieved when all features are active, it is interesting to note that using only the collision status when computing the preference distance for model alignment still leads to improvements. We hypothesize that this is because the reference model already performs reasonably well generating behaviors but lacks awareness of collisions. However, when using only the progress comfort feature compute the preference
8

---

Published as a conference paper at ICLR 2025
distance, both the realism (due an increased collision rate) and ADE regress. This highlights the importance of using a comprehensive set of features to accurately characterize driving behaviors. Further discussions on the limitations and potential solutions can be found in Q1 of Appendix A.
Features
Kinematic
Interactive
Map compliance
Composite realism
minADE
Collision only
0.389
0.788
0.833
0.724
1.527
Progress only
0.421
0.760
0.812
0.710
1.483
Comfort only
0.411
0.744
0.820
0.705
1.581
Full
0.415
0.786
0.867
0.739
1.413
Table 2: The effect of each feature on the effectiveness of our approach.
5.3 COMPARISON WITH ADVERSARIAL PREFERENCE ALIGNMENT FROM DEMONSTRATIONS
We further compare our method with the standard AFD approach, which treats all samples from the reference model as negative samples. For each training sample, we construct rankings by sampling 16 generated traffic simulations from the reference model (i.e., both our method and AFD utilize the same amount of preference data). We measure the WOSAC realism of the fine-tuned model, the model's ability assign higher likelihood to preferred traffic simulations ranked by our preference distance (measured as classification accuracy), ), and the minADE. As shown in Table 3, approach significantly outperforms the adversarial AFD in all metrics, demonstrating the effec- tiveness of our method.
Features
Classification accuracy
Composite realism 1
minADE |
Ours Adversarial AFD
0.84
0.739
1.413
0.52
0.720
1.539
Table 3: The comparison between DPA-OMF with adversarial AFD. Our approach significantly outperforms the adversarial AFD in all metrics.
To further analyze why adversarial preference alignment is less effective, we plot the neg- ative log-likelihood of expert demonstrations, preferred and unpreferred traffic simulations (ranked by our preference distance) in Fig- ure 6. The results reveal that the likelihood of expert demonstrations is consistently much higher than that of both preferred and unpre- ferred samples throughout the alignment pro- cess. This stems from the pre-training phase, where expert demonstrations are used to train the reference model. Moreover, during the pref- erence alignment phase, the model primarily in- creases the likelihood of expert demonstrations while leaving the likelihood of preferred and unpreferred samples relatively unchanged. This indicates that the model is unable to capture nuanced differences between preferred and unpreferred samples, leading to suboptimal alignment performance (see more analysis in Appendix F).
AFD-Expert AFD-Pref. AFD-Unprel
-200
log [P)
50
75
100
125
130
375
200
Eval sleps (x100)
Figure 6: The log-likelihood of expert demos and prefereed/unpreferred rollouts from the reference model when AFD for alignment.
5.4 PREFERENCE SCALING
One of the key advantages of DPA-OMF is its ability to leverage expert demonstrations to auto- matically construct preference rankings without requiring additional human annotations, making highly scalable. In this section, we evaluate the performance of DPA-OME as we scale the number of preference rankings. In Figure (left), we show the relationship between the aligned model's performance and the average number of preference rankings per training example (e.g., value of 4 indicates that the dataset used for alignment is four times larger than the original training set). Interestingly, we observe that small amount of preference feedback not only fails to improve the

---

Published as a conference paper at ICLR 2025
model but actually degrades it. However, the number of preference rankings increases, the align- ment objective begins to demonstrate its effectiveness. We hypothesize that this degradation is due to preference over-optimization, which we explore further in the next section.
0.74
0.74
0.5
8
16
Realism
0.73
Realism
0.73
0.72
0.72
0.71
0.71
5
10
15
-1
0
2
Ave. rankings per sample
log log( (KL)
Performance of the alignment under different preference train- ing data sizes. The gray line represents the performance of the reference model; [right preference over-optimization] The trade-off between the policy drift and the fine-tuning performance gain under various preference data sizes
Figure 7: [left- preferecce scaling]:
5.5 PREFERENCE OVER EXPL OITATINA
Preference over-optimization has been studied the context of LLMSs for both online methods (e.g., RLHF) and direct preference alignment methods (Tang et al., 2024; Rafailov et al., 2024a). In this section, we investigate this phenomenon in the context multi-agent motion generation. While previous research has focused the impact of preference over-optimization across different model sizes, our study examines its effects with varying data sizes.
Goodhart's law states that "when measure becomes target, it ceases to be good measure" (Goodhart & Goodhart, 1984). In our case, this effect manifests when there is insufficient feedback, causing the model to over-optimize we examine this effect by measuring the KI. divergence between the reference model and the fine- tuned model various preference data sizes during the alignment process. KL divergence quantifies how much the optimized policy deviates from the reference model's policy during preference learn- ing, and can be interpreted the optimization cost incurred by the alignment. In Figure (right), with little preference data (e.g., 0.5 rankings per training example). the optimized policy drifts away from the reference but degrades performance. As we scale the preference data, the same policy drift budget results in better performance, though further increases in the KI. budget eventually reduce performance. Nevertheless, this investigation shows that scaling preference data can mitigate the effects of preference over-optimization and highlights the importance of doing in cost-effective manner.
incomplete preference signal. Following (Tang al., 2024),
CONCLUSION
In this work, we consider the problem of efficient post-training alignment for multi-agent motion generation. We propose Direct Preference Alignment from Occupancy Measure Matching Feed- back, a simple yet principled approach that leverages pre-training expert demonstrations to generate implicit preference feedback and improves the pre-trained model's generation quality without addi- tional post-training human preference annotation, reward learning, or complex reinforcement learn- ing. We presented the first investigation of direct preference alignment for multi-agent motion gen- eration models using implicit preference feedback from pre-training demonstrations. We applied our approach to large-scale traffic simulation (more than 100 agents) and demonstrated its effectiveness in improving the realism of pre-trained model's 's generated behaviors, making a lightweight 1M mo- tion generation model comparable saate-of-the-ae large imitation-batt based models by relying solely on implicit feedback from pre-training demonstrations, without requiring additional post-training human preference annotations or incurring high computational costs. Additionally, we provided an in-depth analysis of preference data scaling laws and their effects over-optimization, offering valuable insights for future investigations.
10

---

Published as a conference paper at ICLR 2025
Reproducibility Statement. To enhance the reproducibility of our research, have provided detailed explanation our motion generation model and its source in Appendix C. Additionally, we have thoroughly described the calculation of the preference distance in Section 3.2 and the process of constructing our preference dataset in Section 4.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings the twenty-first international conference Machine learning, pp. 2004.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08077. 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding. Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-ann models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. 2023.
Yulong Cao, Boris Ivanovic, Chaowei Xiao, and Marco Pavone, Reinforcement learning with human feedback for realistic traffic simulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4428-114344 IEEE, 2024.
Zhaorun Chen, Siyue Wang, Zhuokai Zhao, Chaoli Mao, Yiyang Zhou, Jiayu He, and Albert Sibo Hu. Escirl: Evolving self-comtrastive irl for trajectory prediction in autonomous driving. In 8th Annual Conference on Robot Learning, 2024a.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXlv:2401.01335, 2024b.
Charles AE Goodhart and CAE Goodhart, Problems of monetary management: the UK experience. Springer, 1984.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.1366 2023.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint 23Xi: 2023.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv: /705.07215. 2017.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling rein- forcement learning from human feedback with ai feedback. In Forty-first International Confer- ence on Machine Learning.
Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Getting more juice out of the sft data: Reward learning from human demonstration improves sft for Ilm alignment. arXiv preprint 2xXV:22405.17888 2024.
Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Simon Wang, Jiulong Shan, Meng Cao, and Lijie Wen. Direct large language model alignment through self-rewarding contrastive prompt distillation. arXiv preprint arXiv:2402.1 1907, 2024.

---

Published as a conference paper at ICLR 2025
Yicheng Luo, zhengyao jiang, Samuel Cohen, Edward Grefenstette, and Mare Peter Deisen- roth. Optimal transport for offline imitation learning. In The Eleventh International Confer- Learning Representations, 2023. URL https:/ /ttp//ppppwn////eeeeeeeee net. weweenreriww.nnt/fouuaaaaa MhuF2Fsrfvis
Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, al. The waymo open sim agents chal- lenge. Advances in Neural Information Processing Systems, 36, 2024.
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety.
Nigamaa Nayakanti, Rami AI-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 2980-2987. IEEE, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling next-token predic- tion. In The Twelfth International Conference on Learning Representations, 2024.
Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Rajasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, and Jitendra Malik. Humanoid locomotion as next token prediction. arXiv preprint arXiv:2402. 19469, 2024.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikehi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.029000 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Re- faat, Rami AI-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 8579-8590, 2023.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018.
Hao Sun and Mihaela van der Schaar. Inverse-rignmmen:: Inverse reinforcement learning from demonstrations for IIm alignment. arXiv preprint 4rXiv:2405.15624 2024.
Liting Sun, Wei Zhan, and Masayoshi Tomizuka. Probabilistic prediction of interactive driving behavior via hierarchical inverse reinforcement learning. In 2018 2/ International Conference on Intelligent Transportation Systems (ITSC), pp. 2111-2117. IEEE, 2018.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the perfor- mance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.066 084488 2024.
Ran Tian, Chenfeng Xu. Masayoshi Tomizuka, Jitendra Malik, and Andrea Bajesy. What mat- ters to you? towards visual representation alignment for robot learning. arXiv preprint arXiv:2310.07932. 2023.
Thomas Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling. Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. In 8th Annual Conference Robot Learning, 2024.
12

---

Published as a conference paper at ICLR 2025
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arX1v:2310.16944, 2023.
Cédric Villani al. Optimal transport: old and new, volume 338, Springer, 2009.
Chaoqi Wang, Yibo Jiang, Chenghao Yang. Han Liu, and Yuxin Chen. Beyond reverse kl: Gener- alizing direct preference optimization with diverse divergence constraints. In The International Conference on Learning Representations, 2024a.
Yuting Wang. Lu Liu, Maonan Wang, and Xi Xiong. Reinforcement Icarninng from human feedback for lane changing of autonomous vehicles in mixed traffic. arXiv preprint arXiv:2408.04.. 04447, 2024b.
Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: Scalable multi-agent real-time simu- lation via next-token prediction. arXiv preprint arXiv:2405.15677 2024.
Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong
Linh. Wasserstein adversarial imitation learning. arXiv preprint arXiv: 1906.08 13, 2019.
Ceyuan Yang, Yujun Shen, Yinghao Xu, Deli Zhao, Bo Dai, and Bolei Zhou. Improving gans with a dynamic discriminator. Advances in Neural Information Processing Systems, 55:15093-15104 2022.
Kevin Yang. Dan Klein, Asli Celikyilmaz, Nanyun Peng. and Yuandong Tian. Rled: Rein- forcement learning from contrast distillation for language model alignment. arXiv preprint arXiv:2307.129500 2023.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invari- ant representations for reinforcement learning without reconstruction. International Conference on Learning Representations, 2020.
Zikang Zhou, Haibo Hu, Xinhong Chen, Jianping Wang, Nan Guan, Kui Wu, Yung-Hui Li, Yu-Kai Huang, and Chun Jason Xue. Behaviorgpt: Smart agent simulation for autonomous driving with next-patch prediction. arXiv preprint 243.::4405..777.,, 2024.
13


Reviews:
[{"review_id": "JuKI6xpwmv", "reviewer": "Reviewer_mv7M", "summary": "The paper introduces a method for aligning a token-based motion forecasting model better with demonstrations. The method is based on fine-tuning a pretrained model using a contrastive approach.", "questions": "- **Q.1:** The writing could use a proofreading pass. There are some minor spelling issues throughout.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **S.1:** The method shows great results on bringing a 1M parameter model up to the performance of larger models.\n- **S.2:** The writing is mostly clear.\n- **S.3:** The SFT comparison and Fig.4 are interesting.", "weaknesses": "- **W.1:** I don't fully understand from the paper how the embedding works, the agent feature encoder. Could you please either give me some implementation details or some better high-level overview?\n- **W.2:** Some figures are confusing. Fig.1: What are the orange lines on the left above the motion token pred. model? A-hat is not explained. Fig.3: I don't know how to read this diagram. What's the takeaway? Fig.6: I'm completely lost as to what I'm supposed to do with these.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x1oXKJol9R", "reviewer": "Reviewer_3Bbi", "summary": "This paper introduces a novel alignment from demonstration (AFD) strategy for multi-agent motion generation in the autonomous driving setting. Compared to direct annotation of preferences by humans, AFD scales better for the multi-agent setting. However prior AFD methods assume all base model's (or fine-tuned version of it) motion samples to be non-optimal while all demonstrations to be optimal. These alignment strategies are inefficient compared to the proposed method, which also compares the relative quality of generated samples among themselves. The paper shows improved alignment after their proposed AFD measured in terms of collision / progress / comfort features in the autonomous driving motion prediction tasks.", "questions": "1. Can you compare your method to prior AFD methods and show both quantitatively and qualitatively (in figures) why comparing model generations among themselves help? Can you show some examples of the bias introduced by the heterogeneity of the preference data?\n2. What are limitations of AFD? How many human direct annotations do you need vs how many demonstrations of how many cars do you need? How does alignment improves in terms of the labels provided in both cases? (scaling concern in multiagent setting) \n3. If demonstrations are multi-modal, will your method of comparing sampled based of OT-distance metric introduce conflicting gradients and leading to mode collapse?\n4. How does your OT-distance metric factor in collision / progress / comfort features?\n5. Do you have qualitative figures that help readers understand why L2 distance drop at the end in Fig 3? Why does L2 distance metric will lead to missed turn in Fig5?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The paper is very well motivated and presented. The idea is novel and results are well explained by their visualizations. The proposed optimal transport based distance metric is compared to L2-distance baseline. Insights such as why their method works better than supervised fine-tuning as training continues, preference scaling, preference vs exploitations are also investigated.", "weaknesses": "1. There is a lack of discussion of the assumptions or limitations of the proposed method. For example, one assumption is that the OT-based distance between demonstration and generated samples captures the preferences in a monotonic fashion. Is this always the case in self-driving setting? Another assumption is that asking humans to provide dense trajectory demonstration for multiagent interactions is easier to just rank them (even though there are might be many more pair-wise rankings). Do you have any statistics or references that show the prior scale better than the latter annotation scheme? \n\n2. The biggest concern is that there is limited if not no comparison with methods this paper is set out to improve: prior AFD methods that assume all base model's (or fine-tuned version of it) motion samples to be non-optimal. While the paper does compare their OT-distance metric works better than L2 distance as well as AFD works better than SFT, their main motivation of improving prior AFD methods is not validated.\n\n3. While they show OT distance works better than L2 distance and AFD better than SFT, the improvement in table 1 results is quite incremental, limiting the contribution of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Awgm4IyZ5", "reviewer": "Reviewer_YM4s", "summary": "The paper presents a novel approach inspired by inverse reinforcement learning, proposing **Direct Preference Alignment from Occupancy Measure Matching Feedback**. This method aims to align generated behaviors with expert demonstrations by matching occupancy measures in a semantically meaningful feature space. This method does not rely on additional human annotations or complex reinforcement learning but instead leverages the implicit preferences encoded in expert demonstrations. DPA-OMF ranks model-generated samples based on their alignment with expert behaviors using occupancy measure matching in a semantically meaningful feature space. The model is capable of handling up to **128 agents** using a **1M token-prediction model**.\n\n#### Strengths:\n1. **Scaling Experiments:** The paper includes comprehensive ablation studies, highlighting the model’s performance when scaling up the number of agents.\n2. **Detailed Experimental Setup:** The authors provide thorough descriptions of the experimental setups, including parameters and conditions, contributing to the reproducibility and clarity of their results.\n\n#### Weaknesses:\n1. **Some miss proofs in the paper:** \"These algorithms collect preference rankings from humans over model generations and directly update the model to maximize the likelihood of preferred behaviors over unpreferred ones.\" and \" Human annotators must analyze intricate and nuanced motions, which is a time-consuming process, making the scalability of direct alignment methods difficult in these scenarios.\" need more proofs. I think there should be some citations or experiments to show.\n2. **Overuse of Colors and Fonts:** The excessive use of different colors and fonts in the main text affects the readability and cohesiveness of the presentation. A more consistent design would improve the clarity of the paper.\n3. **Visual Clarity of Images:** Some images in the paper are difficult to interpret due to potential resolution, contrast, or layout issues, which could hinder the reader’s ability to understand the visual data being presented.\n\n\nI will refine this review based on the author's rebuttal and feedback from other reviewers. As the discussion progresses, further improvements or adjustments to the evaluation will be considered.", "questions": "Current experiments have demonstrated the feasibility of the approach on a 1M-scale model. Will it still be effective on larger-scale models?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. **Scaling Experiments:** The paper includes comprehensive ablation studies, which highlight the model’s performance when scaling up the number of agents.\n2. **Detailed Experimental Setup:** The authors provide thorough descriptions of the experimental setups, including parameters and conditions, contributing to the reproducibility and clarity of their results.", "weaknesses": "1. **Some miss proofs in the paper:** \"These algorithms collect preference rankings from humans over model generations and directly update the model to maximize the likelihood of preferred behaviors over unpreferred ones.\" and \" Human annotators must analyze intricate and nuanced motions, which is a time-consuming process, making the scalability of direct alignment methods difficult in these scenarios.\" need more proofs. I think there should be some citations or experiments to show.\n2. **Overuse of Colors and Fonts:** The excessive use of different colors and fonts in the main text affects the readability and cohesiveness of the presentation. A more consistent design would improve the clarity of the paper.\n3. **Visual Clarity of Images:** Some images in the paper are difficult to interpret due to potential issues with resolution, contrast, or layout, which could hinder the reader’s ability to understand the visual data being presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)