Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Model Using Implicit Feedback from Pre-training Demonstrations

Authors: Thomas Tian, Kratarth Goel
Primary area: applications to robotics, autonomy, planning
Keywords: efficient post-training preference alignment, alignment from demonstrations, multi-agent motion generation

Abstract:
Recent advancements in Large Language Models (LLMs) have revolutionized motion generation models in embodied applications such as autonomous driving and robotic manipulation. While LLM-type auto-regressive motion generation models benefit from training scalability, there remains a discrepancy between their token prediction objectives and human preferences. As a result, models pre-trained solely with token-prediction objectives often generate behaviors that deviate from what humans would prefer, making post-training preference alignment crucial for producing human-preferred motions. Unfortunately, post-training alignment requires extensive preference rankings of motions generated by the pre-trained model, which are costly and time-consuming to annotate, especially in multi-agent motion generation settings. Recently, there has been growing interest in leveraging expert demonstrations previously used during pre-training to scalably generate preference data for post-training alignment. However, these methods often adopt an adversarial assumption, treating all pre-trained model-generated samples as unpreferred examples and relying solely on pre-training expert demonstrations to construct preferred examples. This adversarial approach overlooks the valuable signal provided by preference rankings among the model's own generations, ultimately reducing alignment effectiveness and potentially leading to misaligned behaviors. In this work, instead of treating all generated samples as equally bad, we propose a principled approach that leverages implicit preferences encoded in pre-training expert demonstrations to construct preference rankings among the pre-trained model's generations, offering more nuanced preference alignment guidance with zero human cost. We apply our approach to large-scale traffic simulation (more than 100 agents) and demonstrate its effectiveness in improving the realism of pre-trained model's generated behaviors, making a lightweight 1M motion generation model comparable to state-of-the-art large imitation-based models by relying solely on implicit feedback from pre-training demonstrations, without requiring additional post-training human preference annotations or incurring high computational costs. Furthermore, we provide an in-depth analysis of preference data scaling laws and their effects on over-optimization, offering valuable insights for future studies.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1729249796595

thread_type:
official_review

<thread_object>
creator: reviewer_ym4s

<content>
summary:
The paper presents a novel approach inspired by inverse reinforcement learning, proposing **Direct Preference Alignment from Occupancy Measure Matching Feedback**. This method aims to align generated behaviors with expert demonstrations by matching occupancy measures in a semantically meaningful feature space. This method does not rely on additional human annotations or complex reinforcement learning but instead leverages the implicit preferences encoded in expert demonstrations. DPA-OMF ranks model-generated samples based on their alignment with expert behaviors using occupancy measure matching in a semantically meaningful feature space. The model is capable of handling up to **128 agents** using a **1M token-prediction model**.

#### Strengths:
1. **Scaling Experiments:** The paper includes comprehensive ablation studies, highlighting the model’s performance when scaling up the number of agents.
2. **Detailed Experimental Setup:** The authors provide thorough descriptions of the experimental setups, including parameters and conditions, contributing to the reproducibility and clarity of their results.

#### Weaknesses:
1. **Some miss proofs in the paper:** "These algorithms collect preference rankings from humans over model generations and directly update the model to maximize the likelihood of preferred behaviors over unpreferred ones." and " Human annotators must analyze intricate and nuanced motions, which is a time-consuming process, making the scalability of direct alignment methods difficult in these scenarios." need more proofs. I think there should be some citations or experiments to show.
2. **Overuse of Colors and Fonts:** The excessive use of different colors and fonts in the main text affects the readability and cohesiveness of the presentation. A more consistent design would improve the clarity of the paper.
3. **Visual Clarity of Images:** Some images in the paper are difficult to interpret due to potential resolution, contrast, or layout issues, which could hinder the reader’s ability to understand the visual data being presented.


I will refine this review based on the author's rebuttal and feedback from other reviewers. As the discussion progresses, further improvements or adjustments to the evaluation will be considered.

strengths:
1. **Scaling Experiments:** The paper includes comprehensive ablation studies, which highlight the model’s performance when scaling up the number of agents.
2. **Detailed Experimental Setup:** The authors provide thorough descriptions of the experimental setups, including parameters and conditions, contributing to the reproducibility and clarity of their results.

weaknesses:
1. **Some miss proofs in the paper:** "These algorithms collect preference rankings from humans over model generations and directly update the model to maximize the likelihood of preferred behaviors over unpreferred ones." and " Human annotators must analyze intricate and nuanced motions, which is a time-consuming process, making the scalability of direct alignment methods difficult in these scenarios." need more proofs. I think there should be some citations or experiments to show.
2. **Overuse of Colors and Fonts:** The excessive use of different colors and fonts in the main text affects the readability and cohesiveness of the presentation. A more consistent design would improve the clarity of the paper.
3. **Visual Clarity of Images:** Some images in the paper are difficult to interpret due to potential issues with resolution, contrast, or layout, which could hinder the reader’s ability to understand the visual data being presented.

questions:
Current experiments have demonstrated the feasibility of the approach on a 1M-scale model. Will it still be effective on larger-scale models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634428588

thread_type:
official_review

<thread_object>
creator: reviewer_3bbi

<content>
summary:
This paper introduces a novel alignment from demonstration (AFD) strategy for multi-agent motion generation in the autonomous driving setting. Compared to direct annotation of preferences by humans, AFD scales better for the multi-agent setting. However prior AFD methods assume all base model's (or fine-tuned version of it) motion samples to be non-optimal while all demonstrations to be optimal. These alignment strategies are inefficient compared to the proposed method, which also compares the relative quality of generated samples among themselves. The paper shows improved alignment after their proposed AFD measured in terms of collision / progress / comfort features in the autonomous driving motion prediction tasks.

strengths:
The paper is very well motivated and presented. The idea is novel and results are well explained by their visualizations. The proposed optimal transport based distance metric is compared to L2-distance baseline. Insights such as why their method works better than supervised fine-tuning as training continues, preference scaling, preference vs exploitations are also investigated.

weaknesses:
1. There is a lack of discussion of the assumptions or limitations of the proposed method. For example, one assumption is that the OT-based distance between demonstration and generated samples captures the preferences in a monotonic fashion. Is this always the case in self-driving setting? Another assumption is that asking humans to provide dense trajectory demonstration for multiagent interactions is easier to just rank them (even though there are might be many more pair-wise rankings). Do you have any statistics or references that show the prior scale better than the latter annotation scheme? 

2. The biggest concern is that there is limited if not no comparison with methods this paper is set out to improve: prior AFD methods that assume all base model's (or fine-tuned version of it) motion samples to be non-optimal. While the paper does compare their OT-distance metric works better than L2 distance as well as AFD works better than SFT, their main motivation of improving prior AFD methods is not validated.

3. While they show OT distance works better than L2 distance and AFD better than SFT, the improvement in table 1 results is quite incremental, limiting the contribution of the work.

questions:
1. Can you compare your method to prior AFD methods and show both quantitatively and qualitatively (in figures) why comparing model generations among themselves help? Can you show some examples of the bias introduced by the heterogeneity of the preference data?
2. What are limitations of AFD? How many human direct annotations do you need vs how many demonstrations of how many cars do you need? How does alignment improves in terms of the labels provided in both cases? (scaling concern in multiagent setting) 
3. If demonstrations are multi-modal, will your method of comparing sampled based of OT-distance metric introduce conflicting gradients and leading to mode collapse?
4. How does your OT-distance metric factor in collision / progress / comfort features?
5. Do you have qualitative figures that help readers understand why L2 distance drop at the end in Fig 3? Why does L2 distance metric will lead to missed turn in Fig5?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731024597188

thread_type:
official_review

<thread_object>
creator: reviewer_mv7m

<content>
summary:
The paper introduces a method for aligning a token-based motion forecasting model better with demonstrations. The method is based on fine-tuning a pretrained model using a contrastive approach.

strengths:
- **S.1:** The method shows great results on bringing a 1M parameter model up to the performance of larger models.
- **S.2:** The writing is mostly clear.
- **S.3:** The SFT comparison and Fig.4 are interesting.

weaknesses:
- **W.1:** I don't fully understand from the paper how the embedding works, the agent feature encoder. Could you please either give me some implementation details or some better high-level overview?
- **W.2:** Some figures are confusing. Fig.1: What are the orange lines on the left above the motion token pred. model? A-hat is not explained. Fig.3: I don't know how to read this diagram. What's the takeaway? Fig.6: I'm completely lost as to what I'm supposed to do with these.

questions:
- **Q.1:** The writing could use a proofreading pass. There are some minor spelling issues throughout.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732324814855

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General statement [1/2]

comment:
We sincerely thank all the reviewers for their helpful comments and suggestions. We are happy that the reviewers found our paper well-motivated, our idea of preference alignment from occupancy measure matching is novel, and aour approach demonstrates strong results supported by comprehensive ablations. We appreciate this opportunity to address the questions and make improvements to the manuscript. The rebuttal contents are also incorporated in the manuscript (highlighted in blue). Specifically, we made the following major changes:

**G1. Additional experiment to compare the proposed approach with the adversarial preference alignment baseline in which samples from the reference model are treated as negative samples.**

We appreciate the reviewer’s suggestion to compare our approach with prior adversarial preference alignment method, and we have provided both quantitative and qualitative analyses (detailed results and analyses are provided in Appendix F of the updated main manuscript).

Following this suggestion, we compared our method with the AFD approach that treats all samples from the reference model as negative samples. **Our findings indicate that our method outperforms the adversarial preference alignment baseline in terms of the realism of the fine-tuned model, the ability to assign higher likelihood to preferred traffic simulations from the reference model (measured as classification accuracy), and minADE, as shown in Table below**. 

| Features         | Classification Accuracy ↑ | Composite Realism ↑ | minADE ↓  |
|------------------|---------------------------|----------------------|-----------|
| **Ours**         | **0.84**                  | **0.739**            | **1.413** |
| **Adversarial AFD**  | 0.52                      | 0.720                | 1.539     |

**Table 3:** The comparison between DPA-OMF with adversarial AFD. Our approach significantly outperforms the adversarial AFD in all metrics.

To further analyze why adversarial preference alignment from demonstrations is less effective, we plotted the negative log-likelihood of expert demonstrations, preferred traffic simulations, and unpreferred traffic simulations in Figure 9 (in Appendix F). The plot shows that the likelihood of expert demonstrations is consistently much higher ($\approx$-205) than that of both preferred and unpreferred samples ($\approx$-245) throughout the alignment process (this stems from the pre-training phase, where expert demonstrations are used to train the reference model), and the likelihoods of preferred and unpreferred samples are very similar. This indicates that the model is unable to capture nuanced differences between preferred and unpreferred samples, leading to suboptimal alignment performance.


**G2. Additional experiments to demonstrate the bias introduced by the heterogeneity of the preference data.** 

In G1, we showed that using expert demonstrations as preferred samples and model generations as unpreferred samples results in increasing the likelihood of expert demonstrations without significantly affecting the likelihood of either preferred or unpreferred generated samples.  This suggests that the model struggles to associate the features that make expert demonstrations preferred with the generated preferred samples.

To further explore this, we conducted a separate experiment demonstrating how a discriminative objective using expert demonstrations as positive samples and model generations as negative samples can lead to spurious correlations. Detailed results are included in Appendix G of the updated main manuscript.

In this experiment, we trained a discriminator using a contrastive objective to distinguish between expert demonstrations and model generations. The discriminator achieved a classification accuracy of $0.83$ on the evaluation dataset, indicating it can reasonably classify motions as either expert demonstrations or reference model generations. When the trained discriminator was used to rank pairs of model-generated motions, we observed a pattern: motions with zig-zag trajectories were often classified as unpreferred, while relatively smooth motions were classified as preferred, even when there is un-human like behaviors (e.g., stuck on roads) (see the example in Figure 10 of the updated main manuscript).

This behavior arises because of the heterogeneity of the two data sources: most human demonstrations exhibit smooth motions, while model generations are not constrained by vehicle dynamics. Consequently, the contrastive objective may incentivize the model to pick up this spurious correlation, prioritizing smoothness over other critical attributes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors introduce a novel framework for multi-agent motion generation. Their approach leverages the implicit preference ordering given by expert demonstrations, meaning that they can extract richer supervision from less data. The approach is applied to an autonomous car domain, wherein the authors show this approach scales to 128 agents using a 1 million token prediction model.

The idea is novel, the paper is well explained, and the results appear solid. The reviewers agree the ideas are interesting and that the paper is mostly clear and well-written.