Title: Human-assisted Robotic Policy Refinement via Action Preference Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: interactive imitation learning

robotic manipulaiton

preference alignment optimization
Abstract: Establishing a reliable and iteratively refined robotic system is essential for deploying real-world applications. 
    While Vision-Language-Action (VLA) models are widely recognized as the foundation model for such robotic deployment, their reliance on offline expert demonstrations critically limits their capacity for post-deployment refinement. 
    To mitigate this limitation, we introduce Action Preference Optimization (APO), a method designed to refine VLA models by human-assisted preference alignment gathered through interaction with environments.
    This method begins with a human-robot collaboration framework for reliable failure correction and interaction trajectory collection through human intervention.  
    However, directly leveraging these interaction trajectories for preference optimization is non-trivial due to the challenges of irreversible robotic actions and token distribution mismatch. To solve this, APO proposes an adaptive reweighting algorithm with binary desirability signals derived from interaction, empowering VLA models effectively suppress failure-prone actions while enhancing corrective action adaptation.
    Ultimately, APO equips VLA models with the crucial capability to learn from failure, paving the way for their iterative refinement and reliable deployment in dynamic environments.
    The experiments conducted in simulation and real-world scenarios prove superior generalization and robustness of our human-assisted framework across a variety of manipulation tasks. We believe this work could bring insights for efficient and stable optimization of VLA models through human-robot collaboration. The code and dataset are released at https://github.com/GeWu-Lab/Action-Preference-Optimization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to leverage the intervention of humans during robot manipulation as a preference to guide the learning of the policy for robust behavior. \n\nWith a base policy trained on demonstrations, a human will take over when the robot encounters problems, forming an intervention dataset. The base policy will be iteratively fine-tuned on the collected data with preference-based optimization. KL divergence is applied to make sure the policy do not shift too much from the base policy. A reweighting method helps the learning focus on important data points.\n\nEvaluation in both simulation and the real world shows improvement compared to BC methods and other preference-based methods.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The idea of making VLA continuously improve during deployment while minimizing the human efforts is well-motivated.\n2. The setting of the experiments and corresponding analysis is great.\n3. The writing is clear and easy to follow.\n\n# Weaknesses\n1. My first concern is about the position of this paper. While the authors point out that previous methods learning from human feedbacks [1] necessitate constant human supervision, this paper follows the same way of intervention. Can I understand that the focus of this paper is on perfence-based optimization and reweighting?\n2. The design of the reweighting needs further justification. The proposed reweighting is based on the loss term, assuming that more significance should be given to the data that causes bigger shift in the base model. Does this contradict to the KL divergence? How to determine the balance between being adaptive and keeping priors? Besides, it not very clear to me why such reweighting could solve the problem of tokenization. Is there any intuition or mathematical proof behind this?\n3. The setting for comparison with baselines is collecting 50 trajectories with intervention and optimize. I am wondering the performance difference if there are more iterations of collection-optimized-collection but keep the total number of trajectories to be the same (i.e. 50).\n\n[1] Liu, Huihan, et al. \"Robot learning on the job: Human-in-the-loop autonomy and learning during deployment.\"\n\nQUESTIONS:\nPlease see the weaknesses. I am willing to raise the score if my concerns are solved.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-05"}

comments: SUMMARY:
The authors introduce HAPO, a preference-tuning algorithm that enables interactive imitation learning for VLAs. Whilst the goal and some of the pipeline is similar to prior algorithms, such as Sirius, the authors show that adapting VLAs require more specialised update rules. The authors show the benefits of their algorithm against baselines, generalisation and continual learning, application to more than one VLA, and real-world robot experiments.

STRENGTHS AND WEAKNESSES:
Given some experience with robot learning, I feel the main strength of this work is the experimental section, which demonstrates the performance improvement given by using their algorithm, as opposed to various more model-agnostic algorithms and other preference tuning algorithms.

Initially, I would have questioned the advancement of HAPO over Sirius, but the authors do use Sirius as a baseline and show that their method is better for optimising VLAs. Nowadays most robot learning policies tend to use diffusion over autoregressive prediction, but I would consider this beyond the scope of this work (the authors also acknowledge this in the limitations section).

The authors could expand the explanation of HAPO to make it easier to grasp, but given space limitations the current size of the various sections are reasonable.

QUESTIONS:
How were hyperparameter chosen for the different methods in the experiments? It's not clear that every method would perform best with the same set of hyperparameters.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a VLA optimization method, Human-assisted Action Preference Optimization (HAPO), which includes a human-robot collaboration data collection framework and an adaptive reweighting algorithm. The authors claim that the proposed adaptive reweighting algorithm addresses intrinsic issues in VLA optimization—specifically, the irreversible nature of interactions during data collection and the mismatch between token probabilities and continuous action errors. The paper presents extensive experiments demonstrating that HAPA outperforms DPO, KPO, TPO, etc., in optimizing VLAs, improving robustness to dynamic environmental changes, and generalizing across different VLA architectures.\n\nSTRENGTHS AND WEAKNESSES:\nThe work conducts a wide range of experiments and ablations to evaluate HAPO—not only for its core function in model optimization, but also for its robustness to dynamic environmental changes and its generalization across different VLAs. I believe all of these are important metrics for assessing a VLA optimization method.\n\nQUESTIONS:\n1. **Trade-off for more precise and fine-grained optimization**: Since HAPO uses Prospect Theory to construct binary labels instead of pairwise data for policy optimization, one concern is that providing only coarse supervision—i.e., judging whether a trajectory is good or bad—may be too vague. In this case, the model might learn undesirable shortcuts or stereotyped behaviors that minimize the loss, rather than learning diverse and robust motion patterns to achieve the goal.\n2. **Questionable quantitative results**: Based on Table 1, the overall performance gain of HAPO over KTO does not appear to be very substantial. Without error bars, it is difficult to assess whether the reported improvements are statistically meaningful. Furthermore, it seems that most of the gains come primarily from the Coffee_D0 task, which raises concerns about statistical significance and broader applicability.\n\nLIMITATIONS:\nPlease refer to the Weaknesses section to address the primary concerns. I also have the following additional questions:\n\n- As mentioned above, Prospect Theory introduces concerns regarding the use of coarse supervision signals. I’m curious that when humans collaborate with the robot to create new trajectories based on the failed ones from the base model, what kind of behavior do they follow? Do they try to mimic the failed trajectory but correct it slightly, or do they focus entirely on achieving the goal, without caring about replicating the original path?\n\n- The paper also claims to address the token mismatch issue through adaptive reweighting (Equations 4–6), but lacks an ablation study isolating the benefit of this specific component. It would be helpful to understand how much performance gain can be attributed directly to this reweighting mechanism.\n\n- Finally, regarding statistical significance: across both the simulation and real-world experiments, the paper presents results on a limited set of tasks and without error bars. This is particularly concerning given that the performance boost of HAPO over the baselines is not very large in absolute terms. More rigorous statistical reporting would help support the claimed improvements.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Human-assisted Action Preference Optimization (HAPO) method to improve the deployment reliability and adaptation of Vision-Language-Action (VLA) models in robotic manipulation tasks. HAPO uses a weighted action preference optimization method and an reweighting method to bridge the gap between discrete token prediction and continuous action regression. Experiments in both simulation and real-world demonstrate its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- HAPO helps to bridge the gap between discrete token prediction and continuous action regression.\n\n- The authors conducted throughout experiments both in simulation and real-world, their conclusions are well supported by convincing experimental results.\n\n- The paper is generally well-organized.\n\nWeaknesses:\n- Experimental details of the baseline are missing. Suggest adding details about the baseline, such as specific loss function formulas.\n\n- There may be non-rigorous baseline re-implementation. If Dagger is re-implemented properly, then it is unlikely to be worse than the base model. There may be non-rigorous experimental details such as: did you use data with $c_t \\neq 0$ for dagger? Since $c_t=0$ is known to be undesirable data, using such data would be a targeted poisoning of the behavioral cloning methods.\n\n- Many experiments to show that HAPO is better than the baseline in various scenarios (e.g., simulation, real-world, novel tasks, lifelong learning, other VLA, etc.) is nice, of course, but discussions of the intrinsic mechanism of HAPO is missing. For example:\n\n    - Let me illustrate with pseudo-code (Algorithm 1), what are the proportions of $c_t=0,1,2$ in each loop (line 9) respectively? Does the proportion of human intervention gradually decreases as training proceeds? Please draw a figure with iteration $i$ as the horizontal axis and proportion as the vertical axis, with three curves, including $c_0/(c_0+c_1+c_2)$, $c_1/(c_0+c_1+c_2)$, and $c_2/(c_0+c_1+c_2)$. \n\n    - What are the curves of $\\lambda_D$ and $\\lambda_U$? Do they both vary much during training? If not much, is it possible that I can solve the reweighting issues by setting a good and fixed hyperparameter value for $\\lambda_D$ and $\\lambda_U$?\n\nIf the author could provide more analysis and experiments on the mechanisms inherent in HAPO, I would be willing to raise the score.\n\nQUESTIONS:\n- In the real-world experiments, what is the total number of human-labeled trajectories needed for significant improvements?\n\n- Since the binary labeling (desirable vs undesirable) is based on K-step pre-intervention windows, could the authors comment on how sensitive is performence to the choice of K?\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}