Human-assisted Robotic Policy Refinement via Action Preference Optimization
Authors: Wenke Xia
Yichu Yang
Hongtao Wu
Xiao Ma
Tao Kong
Di Hu
Venue: NeurIPS 2025 poster ()
Keywords: interactive imitation learning
robotic manipulaiton
preference alignment optimization
Primary area: deep_learning

Abstract: Establishing a reliable and iteratively refined robotic system is essential for deploying real-world applications. 
    While Vision-Language-Action (VLA) models are widely recognized as the foundation model for such robotic deployment, their reliance on offline expert demonstrations critically limits their capacity for post-deployment refinement. 
    To mitigate this limitation, we introduce Action Preference Optimization (APO), a method designed to refine VLA models by human-assisted preference alignment gathered through interaction with environments.
    This method begins with a human-robot collaboration framework for reliable failure correction and interaction trajectory collection through human intervention.  
    However, directly leveraging these interaction trajectories for preference optimization is non-trivial due to the challenges of irreversible robotic actions and token distribution mismatch. To solve this, APO proposes an adaptive reweighting algorithm with binary desirability signals derived from interaction, empowering VLA models effectively suppress failure-prone actions while enhancing corrective action adaptation.
    Ultimately, APO equips VLA models with the crucial capability to learn from failure, paving the way for their iterative refinement and reliable deployment in dynamic environments.
    The experiments conducted in simulation and real-world scenarios prove superior generalization and robustness of our human-assisted framework across a variety of manipulation tasks. We believe this work could bring insights for efficient and stable optimization of VLA models through human-robot collaboration. The code and dataset are released at https://github.com/GeWu-Lab/Action-Preference-Optimization.

Human-assisted Robotic Policy Refinement via Action Preference Optimization
Wenke Xia Yichu Yang", Hongtao Wu?, Xiao Ma², Tao Kong², Di Hu
"aollin School of Artificial Intelligence, Renmin University of China, Beijing
ByteeDance Seed
3 Engincering Research Center of Next-Generation Intelligent Search and Recommendation, MOE Beijing Key Laboratory of Research on Large Models and Intelligent Governance
Abstract
Establishing a reliable and iteratively refined robotic system is essential for de- ploying real-world applications. While AssionLanguuge-AAee Action (VLA) models are widely recognized the foundation model for such robotic deployment, their reliance on offline expert demonstrations critically limits their capacity for post- deployment refinement. To mitigate this limitation, we introduce Action Preference Optimization (APO), method designed to refine VLA models by numan-assisted preference alignment gathered through interaction with environments. This method begins with human-robor collaboration framework for reliable failure correction and interaction trajectory collection through human intervention. However, directly leveraging these interaction trajectories for preference optimization is non-trivial due to challenges of irreversible robotic actions and token distribution mis- match. To solve this, APO proposes adaptive reweighting algorithm with binary desirability signals derived from interaction, empowering VLA models effectively suppress failure- proee actions while enhancing corrective action adaptation. Ulti- mately, APO equips VLA models with the crucial capability to learn from failure, paving the way for their iterative refinement and reliable deployment in dynamic environments. The experiments conducted in simulation and real-world scenarios prove superior generalization and robustness of our human-assiste framework across a variety of manipulation tasks. We believe this work could bring insights for efficient and stable optimization of VLA models through human-robot collabo- ration. The code and dataset are released at https://github.comm Wu-Labb/Actionn Preference-Oppimizttoe
Introduction
Fostering continuous improvement is crucial for the development of robust robotic manipulation systems real-world scenarios [1O Benefiting from the capacity for generalizable reasoning and scalable learning, widely recognized the foundation model for such robotic deployment systems. However, prevailing training paradigm for these models hinges large-scale, offline datasets of expert demonstrations. This severely limits their post-deployment refinement, as they lack the intrinsic ability to continually learn from failures adapt to novel scenarios encountered in the real world.
(VLA) models [3] 4 I8 43 21 47) have been
To enhance the continuous learning ability of robotic systems, interactive imitation learning frame- works 7 [ have been developed to refine error-prone trajectories via iterative human-in- numan-in-the-llop
'Wor is done during internship ByteDance Seed 'Corresoonding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(e) Human robot Colubbration Depleyment
Action Preference Optimization
Fillure Ocours
Takkover
Salanced
Adaptive Reweightlin
Maaimiz
PPlicy Execution
Interacioo Traj
Deploy
Figure 1: Our method consists of two key components: (a) the human-robot collaboration deployment framework for reliable deployment and interaction trajectory collection with human intervention. (b) the action preference optimization process with adaptive reweighting for VLA models learning from sub-optimal interaction trajectories. The size of each circle represents its weight during training.
correction feedback. Among them, behavior cloning 15
has been widely utilized to fine-tune a base policy model with manually corrected intervention data in a supervised learning fashion. In contrast, recent methods [20] [26] seek to propose effective off- off-policy reinforcement learning algorithms from sub-optimal human intervention trajectories. However, behavior cloning fails to fully exploit failure trajectories, which are valuable signals for learning robust policies. At the same time, reinforcement learning methods encounter significant scalability limitations in training large-scale VLA models, due to the inherent instability and challenge of developing generalizable value functions. To date, the effective adaptation of VLA models for downstream manipulation tasks remains understudied, particularly within sub-optimal human intervention paradigms.
23 35
To bridge this gap, we propose Action Preference Optimization (APO), a new paradigm moves beyond the limitations of both behavior cloning and reinforcement learning. By learning from action- level preferences captured during interactions, our approach fully exploits the valuable information in failure trajectories while maintaining the oplimization stability required for large-scale VLA models.
Our method is founded deployment while simultaneously generating data for policy refinement, as illustrated in Figure a When the robot encounters challenging situations, real-time human interventions not only guarantee successful task completion but also provide corrective trajectories. These trajectories are collected as preference pairs refine the policy. Furthermore, address the imbalanced distribution of action types in the collected data, we employ a balanced sampling method. This ensures proportional representation of all interaction data for the subsequent VLA preference optimization
human-robot collaboration framework designed ensure reliable
However, directly applying this preference data to fine-tune autoregressive VLA models presents two significant challenges: (1) Irreversible interaction: While LLMs often require paired preference data, the irreversible nature of physical interaction makes it difficult to gather perfect positive-negatio action samples under identical conditions. (2) Token probability mismatch: Autoregressive VLA models discretize continuous actions into tokens, causing a mismatch between token probabilities and the action loss, which complicates preference alignment. To address these problems, we first employ Kahneman & Tversky's 's prospect theory [T] 40) to formulate a preference alignment objective that learns from binary desirability signals derived from interaction. This objective relaxes the demands of preference pairs, making it suitable for learning from irreversible robotic interaction trajectories. Furthermore, we propose an adaptive reweighting method that leverages decoded continuous actions to guide preference optimization in the discrete action token space. This approach addresses the challenge action token probability mismatch via the dynamic modulation of sample- wise training weighting, thereby concentrating gradient optimization on failure- interaction actions. Through weight refinement, we apply preference alignment optimization to VLA models, enhancing performance when learning from sub sub-optimal manipulation correction trajectories.
To systematically evaluate the effectiveness of our proposed system, we conduct comprehensive set of experiments in RoboMimic (29 simulation environments. The empirical results demonstrate that APO facilitates rapid adaptation in in-distribution scenarios while maintaining robust performance across variety of unseen perturbations. Furthermore, lifelong learning experiments demonstrate the framework's 's capacity for iterative improvement through human intervention. To evaluate the practical viability of the proposed framework, we conducted real-world experiments on fine-graines insertion tasks under range of disruption conditions, demonstrating robustness and applicability in world robotic manipulation scenarios.

---

2 Related Works
2.1 Lision-Languuge-Action Models
Achieving generalizable robotic manipulation remains a significant challenge within the field of robotics. Motivated by recent advances in foundation models [9
39
42
32
45
48), some
works 6 12 T attempt to construct large-scale real-world robotic datasets to facilitate devel- opment of generalizable Vision- Visiou-Lan Action (VLA) models. Building upon these datasets, recent research [ 18 33 formulates robotic action prediction next token prediction problem
within the framework of VLMs. In contrast, alternative studies [24 38 investigate the applicability of diffusion-based methods to model multi-modal action distributions, thereby facilitating robustness in manipulation tasks. While these works focus on behavior cloning from expert demonstrations, Grape [49] proposes trajectory-level preference alignment method to boost generalizability by incorporating both successful and failed trials. However, the requirements of paired trajectories under the same conditions make it infcasible in real-world scenarios. In this work, we propose the action preference optimization method to continuously refine VLA models by integrating human-ing numan-in-ther in-the-loop intervention preference data.
2.2 Preference Alignment of Large Language Models
50 implement Reinforcement Learning from Human Feedback (RLHF) through two-stage method, which first trains reward estimation model and optimizes LLMs to maximize the given estimated reward with a reinforcement learning method [T3] 36 25). However, this paradigm slow and unstable in practice. DPO (34) proposes single-step alternative that reparameterizes the RLHF objective into a closed-form loss function to directly maximize the log-likelihood margin between preferred and dispreferred outputs. Extending this framework, KTO IT introduces the human-aware losses for learning from a binary signal of whether an output desirable, which bypasses the need for intricate preference annotation altogether. In this work, models. Through
Contemporary methods
adapt the preference alignment optimization method for Vision Language- adaptive reweighting approach, mitigate the inreversible interactions and token probability mismatch challenges when transferring preference learning methods from LLMs to VLA models.
2.3 Human-robot Interactive Learning
Interactive imitation learning
7 has been proposed to refine robot actions through human feedback.
While prior research [17 actions, more recent studies [4
23 35 necessitates constant human supervision to intervene in the robot's
22 46] have introduced dynamic models for automatic failure detection and real-time monitoring. In contrast, RLIF [26] leverages human intervention signals rewards for off-policy RI., while HIL-SERI. [27] presents numan-in-the-lone vision-based RL system tailored for dexterous manipulation tasks. However, these RL approaches encounter difficulties in large-scale VLA model training, primarily due to unstable gradient optimization. In this work, we propose the action preference optimization method to ensure the stable optimization of policies from action-level preferences captured during interaction.
3 Method
In this section, we introduce Action Preference Optimization (APO), a method designed to facilitate continuous iterative improvement of Latlon-Lnggage-Actti (VLA) models. As detailed in Algo- rithm [ APO aligns the model with human preferences gathered through human-robot collaboration deployment within environment.
3.1 Human-robot Collaboration Deployment
To ensure reliable deployment and interaction trajectory collection, our method is founded on the human-robot collaboration framework for real-time intervention and interaction data acquisition.
We first collect an expert demonstration dataset D. = {r:)i=N where each trajectory -ー. consists of observation-action pairs with expert annotations: T. {[of, af )) where ef = indicates that af is executed by human expert. We employ behavior cloning to fine-tune the pretrained VLA model
3

---

on these expert demonstrations, obtaining an initial base policy mg. This policy is then deployed for interaction trajectory collection.
During policy execution, the human operator monitors policy execution and intervenes when the policy encounters challenging scenarios. Through this process, could collect of interaction
trajectories DA = { i== h where cj = 2 represents the action is corrected by human intervention while e = denotes the action is executed by policy. Further, we re-label the interaction trajectories to categorize the actions taken in the K steps preceding human interventions as undesirable, annotated with cj = 0. For each trajectory, we discretize the continuous action a into discrete action token a. Finally, we combine the expert demonstrations D. and the interaction dataset Dr for further robotic action preference optimization.
3.2 Action Preference Optimization
To maximize the utility of sub-optimal interaction trajectories and ensure stable fine-tuning of the VLA model, we adopt the preference alignment optimization method to guide the model to learn from corrections and avoid failures.
Although previous Reinforcement Learning with Human Feedback (RLHF) methods have proven effective in LLM fine-tuning, there are additional challenges for the VLA models preference optimization in robotic manipulation:
The irreversible robotic manipulation process makes it challenging to acquire meaningful paired positive-negative actions under the same observational conditions.
The mapping of continuous robotic actions to discrete tokens by autoregressive VLAs causes a mismatch between token probability and continuous action errors, complicating preference optimization in action token prediction.
To address these issues, we adopt Kahneman & Tversky's prospect theory 40) for preference alignment optimization with binary desirability signals and propose an adaptive reweighting method to bridge the gap between discrete token prediction and continuous action regression. We first
estimate the reward function ro of our model To standard approach 31 37
(a(o))
re (o, a) = log (āl())
(1)
where is the discrete action token and the reference model Tref is the base model Tá. the beginning of each utility function v as below to estimate the relative gain on the robotic data:
loop shown in algorithm Following [1I] [4]] we formulate the
AAo (ro(o, à) 20) if âdesirable Appo (=n - ro(o,a)) if âundesirables
v(o, a)
where Åo and Aur are utilized for importance sampling, the a is the sigmoid function. To ensure that the model my does not deviate excessively from the reference model Tref- a penalty term 2g introduced. This term is defined the KL-Divergence between Ti and Trefi = K LLTTell Tre.) ). Incorporating 2g into the optimization process guides the model to learn from preference pair data while simultaneously preserving knowledge acquired from prior models. We employ the following loss function L to optimize the model my using preference optimization with desirability signals:
L[ Fref) = D (= [-v(x,)) v)]-
By minimizing the loss function, we aim for the model Te to get higher rewards for desirable pairs while avoiding predicting undesirable actions, in comparison to the reference model Tref-
However, directly applying the preference alignment optimization from LLMs to autoregressive VLA models is problematic, primarily due to the differences in their respective token definitions. While word tokens correspond to distinct subwords, action tokens necessitate non-differentiable mapping to continuous ground truth actions. This creates a discrepancy between the token classification probabilities and the regression loss associated with the continuous robotic actions.
To bridge the gap between token classification and continuous action regression autoregressive VLA models, introduce an adaptive reweighting method. This approach guides the model to

---

prioritize samples exhibiting large regression errors by first estimating the L1 loss of the continuous action for each sample, followed by batch-level normalization as detailed below:
W/ 21=1 4
(4)
The normalized weighting scheme operates by: 1) for desirable data, increasing the weight of samples with high action prediction errors, and 2) for undesirable data, increasing the weight of samples whose actions are proximate to the failure actions. By adaptively adjusting the values of Ap and λ in Equation I Jusing the normalized weights, we gain fine-grained control over the relative influence of each sample during training:
Ao = 1 - C AD*W
(5)
λu = St
(6)
By incorporating preference alignment optimization via sample-wise weight refinement, we enhance the performance and optimization stability of the VLA model, when learning from sub-optimal manipulation correction trajectories.
In conclusion, we propose the action preference optimization method as demonstrated algorithm
This approach leverages the human-robot collaboration deployment for reliable task execution and interaction trajectories collection, while the action preference optimization process provides stable autoregressive VLA optimization with adaptive reweighting. Through iterative human-robot collabo- ration deployment and action preference optimization, we could achieve continual improvement from interaction with environments for autoregressive VLA models.
Algorithm Action Preference Optimization
1: Notations:
2: De: expert demonstrations, Dz: interaction dataset, т: interaction policy
3: Warm-start phase
4: Collect D. + {ff..... J
5: Initialize BC policy x
6: 8" arg maxę E (a,a)DD [log (aal)]
7: Do De
8: Deploymeentoppiimizatio loop
9: for i < 0 to X do
Tref + T
D 1 + DEPLOYMENT(Tİ, Di))
++ + OPTIMIZATION (Ty, Trefi Ditt1) Wa
10: end for
11:
function DEPLOYMENT(Ta, Dr)
26:
function OPTIMIZATIONN for n gradient steps do
12:
for n interaction rollouts do while task does succeed do human intervenes then
27:
13:
28:
Balanced Sample (0i, 0z. c:) Dr
14:
29:
1 + (T: 0s
15:
ag human, Ct 2 G-K +0
30:
w; f SE".
16:
31:
if c # then
else
Ap = e Bpaws
18:
= 11
end if
32:
else
20:
T/² + U
Au. e Ao
end while DA < DD, U end for return Dr
33:
end if
22:
8" arg maxe El-v(o, a, ne, Kref Åo, X=]]
35:
end for
24:
36:
return TT
end function
37:
end function
5

---

Experiments
To comprchensively evaluate our action preference optimization method for effective downstream adaptation, we propose experiments to validate the following questions:
How effective is APO promoting adaptation to in-distribution scenarios? Section 4.2
Does APO maintain effective learning performance in novel scenarios despite various disruptions? Section 4.3
Does APO demonstrate the ability achieve iterative improvement during deployment?
Section 4.4
How well does APO generalize to different VLA models? Scetion 4.5
Could APO be applied in fine-grained real-world scenarios? Section 4.6
To what extent does the action-level preference optimization method enable APO to learn action correction? Section 4.7
4.1 Experiment Settings
Implementation Details. this work, we fine-tune the OpenVL.A 41 model for target manipulation
tasks the base model. We employ LoRA (I6) for parameter-efficient tuning, configuring rank r r= 32 with batch size of 16 across 8 NVIDIA A 110 GPUs. Further, we deploy the base model to interact with environments, where human operators perform real-time corrective interventions via SpaceMouse device to rectify failures during execution. We set K 10 to identify and annotate undesirable behaviors automatically. The auman-assiston interaction trajectory is shown in Figure [] which is segmented into robotic automatic execution, the failure action, and the human intervention types by the timing of human correction. Based on the interaction trajectories collected during task execution, we fine-tune the hase model Tref with our action preference optimization method, using learning rate of 5e-5 and a batch size across 4 NVIDIA A100 GPUs. To ensure the stability of preference alignment training, we employ balanced sampling to ensure that each batch contains 50% expert actions, 25% human intervention actions, and 25% failure actions.
Automatic Execution
Falure Action
Humnn Interve rtion
Automatic Execulton
Figure The demonstration of our human- interaction trajectory.
Simulation Environments Details. For a comprehensive evaluation, we validate these methods fine- grained manipulation tasks within the RoboMimic [29] simulation environment, such as 'make coffee' and 'toy assembly'. In the RoboMimic environment, we fine-tune the pretrained Open VLA model for these 4 long-horizon manipulation tasks with 300 expert demonstrations. To optimize policy with human preferences, we collect 50 trajectories per task under different seeds for RoboMimic tasks. For evaluation, conduct 50 trials under three unseen seeds for each task, and report the average success rate.
4.2 Comparison Experiments
We compare APO with other approaches to evaluate the effectiveness for VLA model fine-tuning. To ensure fairness, we fine-tune Open [18) for manipulation tasks as a base model and improve the base model with other comparison methods.
Dagger 35 : We mix the expert demonstrations with interaction trajectories, fine-tuning
the base model using behavior cloning objective.
we apply sample reweighting to prioritize human intervention data and fine-tune the base model using weighted behavior cloning loss.
Sirius 23 :
DPO 34 We generate paired negative samples for interaction trajectories by perturbing the actions predicted from the base model with Gaussian noise, and fine-tune the base model using these paired data with the DPO method.

---

Fixed Stick
Bg.
Gray Bg.
Red Block
Wooden Block
(>) Disruption Square_Da
(b) Background Disruption
Disruption
Figure 3: In the position disruption setting, change the position of the stick from fixed point to a random position from the rectangle m in the Square DO task as illustrated in (a). In the background disruption setting, replace the background with the gray one in the StackThree DO task as shown (b). In the texture disruption setting, we replace the red blocks with the wooden ones.
TPO [4)]
We select positive and negative samples in interaction trajectories based on the timing of intervention, then fine-tune model with trajectory-wäns preference optimization.
We select positive and negative samples interaction trajectories based on the timing of intervention. Then, sample positive and negative trajectories and optimize the base model with KTO, with the constraint = KL((T) [ree)..
KTO 11
As shown in Table first compare the behavior cloning objective methods. The results reveal that after fine-tuning with interaction data, these methods fail to outperform the base model, which demonstrates that existing behavior cloning approaches struggle achieve efficient adaptation in the context of large-scale VLA models. A key challenge stems from the distribution shift between expert trajectories and interaction trajectories. Without mechanisms retain the base model's knowledge under the under standard behavior cloning objectives, this shift makes it particularly difficult for large-scale VLA models effectively fit the complex, multimodal distribution arising from the combined expert and interaction datasets.
We further provide results of the preference optimization based methods. By integrating regulariza- tion constraint with the reference model kref. these methods could maintain useful knowledge from the reference model while achieving improvement from interaction trajectories.
Among all compared preference learning based methods, DPO yields the weakest performance. This result stems from its exclusive reliance on synthetic paired failure actions for optimization, which lacks exposure real-world errors essential for teaching robots mistake avoidance through interaction. On the other side, TPO fails to deliver stable performance gains on multiple tasks while APO attains stable performance gains relative to the base model. The TPO method employs negative samples to regularize model preference alignment optimization, but introduces instability through random sampling. In contrast, APO utilizes KL divergence to estimate the mean margin between the updated model and the reference model, which not only enables more stable learning but also better preserves prior knowledge. Compared with KTO, APÔ leverages the adaptive reweighting method to achieve more precise control over the importance weights of both positive and negative samples, delivering more notable performance improvements.
Table 1: Comparison experiment results across 4 manipulation tasks in RoboMimic Simulation. The results demonstrate that our adaptive reweighting preference optimization method achieves stable improvement compared with other behavior cloning and preference optimization methods.
Methods
Coffee_DO DO
StackThree_DD
Asseell
Square_Da
Mean
Base policy
44%
46%
44%
28%
40.5%
Dagger
35
42%
50%
36%
28%
39.0%
Sirius
23
34%
52%
34%
38%
39.5%
DPO
34
52%
46%
28%
22%
37.0%
TPO
49
54%
54%
40%
18%
41.5%
KTO
48%
52%
46%
32%
43.5%
APO
60% %
54%
46%
32%
48.0%

---

Table 2: The results disruption scenarios.
Table 3: The results original tasks.
Methods
Pos Dis.
Bg Dis.
Tex Dis.
Mean
Methods
Square
StackThrre
ThreePiece
Mean
Base policy
12%
42%
10%
21.3%
Base policy
28%
46%
44%
39.3%
Dagger
18%
46%
4%
22.7%
Dagger
16%
46%
30%
30.7%
Strius
12%
42%
2%
18.7%
Sirius
18%
48%
18%
28%
DPO
14%
26%
2%
14.0%:
DPO
20%
50%
30%
33.35.
TPO
18%
32%
8%
19.3%
TPO
30%
36%
40%
35.3%
KTO
20%
46%
6%
24.0%
KTO
30%
46%
42%
39.3%
APO
26%
46%
12%
28.0%
APO
34%
62%
40%
45.3%
Baseline
Baseline
20-43
Our Method
Our Method
40-43
Rudoo on
Rollout Beratioss (4) Coffee, DO Success Rate
Rollose Becxtion
Stechthree (4) Human Intervention Frequency
Cothec.t
(b) StackThree DO Success Rate
Figure 4: Lifelong learning results of APO method.
4.3
Generalization to Novel Tasks
In this section, we assess APO's generalization capability under three novel scenarios, as illustrated in Figure B (I) Position Disruption: For the Square DO task, we replace the fixed initial stick position with randomized placements within a bounded operational area. (2) Background Disruption: In the StackThree DO task. we substitute the default white background with a gray one. (3) Texture Disruption: In the ThreePiece DO task, the original red blocks are transmuted to wood-grain visual properties. These experiments systematically evaluate robustness against spatial, background, and visual texture variations. To fine-tune the base model novel disruption scenarios, we collect 20 interaction trajectories under disruption scenarios and combine them with 20 expert demonstrations from the original task for subsequent fine-tuning.
Our objective to develop an action preference optimization method that facilitates continuous improvement, enabling performance enhancements in novel disruption scenarios while retaining original task capabilities during model fine-tuning. Thus, we evaluate the performance of the fine-tuned model across both disruption scenarios and original scenarios.
As shown in Tabel 2 the base policy exhibits scenarios. However, the performance decline is relatively minor cases background disruption, whereas disruptions in object texture and position significantly impact performance. Both behavior cloning methods and preference optimization methods struggle to achieve significant performance improvements in novel disruption scenarios. contrast, APO can effectively adapt new disruption scenarios through adaptive reweighting.
degree of performance degradation in disruption
Table presents the performance of the optimized model after being fine-tuned from disruption data on the original task. The results reveal that behavior cloning methods exhibit severe catastrophic forgetting, resulting in substantial performance degradation. By contrast, the preference optimization method achieves mitigated performance decline with the constraints of the reference model.
Besides, APO utilizes adaptive reweighting to effectively integrate knowledge from both expert demonstrations and interaction trajectories. This mechanism not only facilitates learning from diverse data sources but also leads improved performance the original task.
4.4 The Performance of Lifelong Learning
To investigate whether APO can iteratively improve via environment interaction, we deploy APO to interact with environments while updating the model every 20 interaction rollouts. provide compar- ison results using behavior cloning policy trained with the same number of expert demonstrations our baseline. For each updated model, we conduct 50 trials and report the success rate.

---

Table 4: The results TO-FAST model.
Table 5: The results real-world experiments.
Methods
Coffee__ii DD
StackThree DO
Insert Square
Methods
In Dis.
Pos Dis.
Bg Dis
Tex Dis.
Base policy
68%
64%
85%
Base policy
65%
25%
10%
25%
Dagger 64%
66%
Dagger
65%
10%
10%
25%
TPO
48%
52%
90%
TPO
75%
40%
20%
45%
APO
76%
74%
95%
APO
85%
55%
30%
55%
Insert the square into the stick
(a) Expert Demonstration
Dis.
Pos Dis.
Black Stick
Brown Bg.
Black Stick
Green Stidk
(b) Position Disruption
(c) Background Disruption
(d) Texture Disruption
Figure 5: Demonstrations of real-world experiments with disruption settings.
As shown in Figure f b), APO achieves superior performance compared the baseline, demonstrat- ing its ability to effectively leverage sub-optimal human intervention trajectories for iterative model improvement. When the base policy exhibits diminishing improvement with increasing expert demon- strations, APO enables continual performance gains from the interaction trajectories. Besides, this improvement trend accompanied by a corresponding reduction the required human intervention ratio, as shown in Figure
4.5
Generalization to various VLA models
To validate that APO can adapted to different VIA models, we applied APO fine-tune the T0-FAST (33] model. T0-FAST applies discrete cosine transform encoding to encode the action chunking into discrete tokens for VI.A training. To adopt this model for downstream tasks, We regenerate the action tokenizer with 5 action chunking step for each task.
As shown T the base model could achieve higher success rate compared with the Open VLA model, benefiting from its ability to predict action chunking for robotic manipulation. Further, we compare APO with both the behavior cloning method and preference optimization method, the results demonstrate that APO could achieve consistent improvement for the a0-FAST fine-tuning The results prove that APO could be applied to the fine-tunin mee-ttning of various VLA models, achieving consistent performance gains.
4.6 Real-world Experiments
In this work, we conduct the challenging fine-gra tne-grained robotic manipulation task "Insert the square into the stick" as shown in Figure [a). which requires the robot to grasp the square and precisely insert into the stick. To collect expert demonstrations, we utilize the spacemouse device to gather 100 high-quality trajectories at action frequency of 20 Hz. We fine-tune the Open VL. model with the collected demonstration as the base model. Further, we deploy the base model to interact with environments and propose the real-time human-in human-in-the-loop interventions to collect 20 interaction trajectories for subsequent action preference optimization. All methods are evaluated under the same experimental setup, and we report the average success rate from 20 trials. For comprehensive evaluation in real-world scenarios, APO was also tested on the "hang cup on the rack" and "put lemon on the plate" tasks. The corresponding results are detailed in the supplementary material.
To comprehensively evaluate APO, experiments are conducted not only under in-distribution but also across three distinct disruption settings as shown in Figure b (1) Position Disruption: We change the position distribution of the stick. (2) Background Disruption: We replace the tablecloth from white to brown. (3) Texture Disruption: We replace the black stick to the green one.

---

As demonstrated in T APO demonstrated robust adaptability to these downstream disruption scenarios. The results empinically validate the method's practical utility for real-world deployment in unstructured environments.
We also adopt APO fine-tune the a0-FAST model in the real world scenario. The results in Table T prove APO could achieve consistent performance gains over other methods.
4.7 Correction from Failure scenarios
In this work, propose the APO method that enables models not only to avoid failure modes but also to self-correct within failure scenarios. As shown in Figure 6 we provide examples of failure correction across multiple tasks, demonstrating the corrective strategies learned by APO.
For instance, in Figure b),,,, when the model initially fails grasp an object, APO identifies the failure and initiates a re-grapp attempt. Similarly, in Figure (Jc). when precise insertion operation obstructed, APO learns to iteratively adjust its gripper position until the insertion successfully completed. These examples illustrate that APO has successfully learned to recover from common failure scenarios, which directly contributes to its improved overall performance.
(a) Insort the square into the söick
(b) Put the lommn on the plato
[c Make caffea
Simulation
Figure 6: The rollout trajectory of APO. As indicated by the bold red and green boxes, APO can autonomously correct form failure scenarios.
Conclusion
In this work, we introduce the Action Preference Optimization (APO) method to fully exploit valuable information in failure trajectories while maintaining the stability required for large-scale VLA models. This method builds human-robot collaboration framework for reliable deployment, and utilizes an adaptive reweighting preference optimization algorithm with action-level binary desirability signals for stable VLA model optimization. Through APO, could promote continuous improvement during the deployment of VLA models. We hope APO could bring insights for efficient and effective VLA model adaptation downstream manipulation tasks.
Discussion and Future Work. While our work study the preference alignment optimization VLA models, the experiments are based solely autoregressive VLA models. Future work should explore a broader range of VLA frameworks, including regression-bant approaches and diffusion policy models, to ensure the generalizability of our method across different architectures.
10

---

6 Acknowledgement
The project was supported by the fund for building world-class universities (disciplines) of Renmin University of China, and CCF-Zhipu.Al Large Model Innovation Fund. The project was also supported by National Natural Science Foundation of China (NO.62106272).
References
[1] Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469-483, 2009.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprini XXi::2204005862 2022.
[3] Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, al. Gr0Ot n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503. 2025.
[4] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi0: A vision- uision-anguuage-cttinn flow model for general robot control. arXiv preprint arXiv:2410.24164 2024.
[5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15811 2023.
[6] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang. Shu Jiang, al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503. 06669, 2025.
[7] Carlos Celemin, Rodrigo Pérez-Dattari, Eugenio Chisari, Giovanni Franzese, Leandro de Souza Rosa, Ravi Prakash, Zlatan Ajanović, Marta Ferraz, Abhinav Valada, Jens Kober, et al. Interactive imitation learning in robotics: survey. Foundations and Trends@ in Robotics, 10(1-2):1-197, 2022.
[8] Chi-Lam Cheang. Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative iileo-languageacction model with web-scale knowledge for robot manipulation. arXiv preprint 1rXiv:2410.06158, 2024.
[9] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[10] Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, and Dorsa Sadigh. No, the right: Online language corrections for robotic manipulation via shared autonomy. Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, HRI '23, page 93-101, New York, NY, USA, 2023. Association for Computing Machinery.
[11] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela, Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01300 2024.
[12] Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: comprehensive robotic dataset for learning diverse skills in one-shot. arXiv preprint arXiv:2307.0059) 2023.
[13] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization.
In International Conference Machine Learning, pages 10835-10866 PMLR, 2023.
11

---

[14] Cem Gokmen, Daniel Ho, and Mohi Khansari. Asking for help: Failure prediction in behavioral cloning through value approximation. In 2023 JEEE International Conference on Roboties and Automation (ICRA), pages 5821-5828. IEFE, 2023.
[15] Ryan Hoque, Lawrence Yunliang Chen, Satvik Sharma, Karthik Dharmarajan, Brijen Thanan- jeyan, Pieter Abbeel, and Ken Goldberg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Conference on Robot Learning, pages 368-331880 138-080. PMLR, 2023,
[16] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[17] Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell and Mykel J Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083, IEEE, 2019.
[18] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprini urXiv:2406.09246. 2024.
[19] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li. Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03322 2024.
[20] Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. arXiv preprint arXiv: 2022.
[21] Kong Ling-Huan, He Wei, Chen Wen-Shi, Zhang Hui, and Wang Yao-Nan. Dynamic movement primitives based robot skills learning. Machine Intelligence Research, 20:396-4000 407, 2023.
[22] Huihan Liu, Shivin Dass, Roberto Martín-Martin, and Yuke Zhu. Model-based runtime moni- toring with interactive imitation learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4154 4161. IEEE, 2024.
[23] Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning on the job: Human-in cemmmnn--n-thh-eoe on-the-loop autonomy and learning during deployment. In Robotics: Science and Systems (RSS), 2023.
[24] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang. Ke Xu, Hang Su, and Jun Zhu. Rdt- lb: a diffusion foundation model for bimanual manipulation. arXiv preprint 4rxiv:2410.07864, 2024.
[25] Jingxian Lu, Wenke Xia, Dong Wang. Zhigang Wang, Bin Zhao, Di Hu, and Xuelong Li. Koi: Accelerating online imitation learning via hybrid key-state guidance. arXiv preprint arXiv:2408.022 029112 2024.
[26] Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma, and Sergey Levine Rlif: Interactive imitation learning as reinforcement learning, arXiv preprint arXiv:2311.12944 2023.
[27] Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via numan-in-the--le reinforcement learning. arXiv preprint 41X.222410.2/845. 2024.
[28) Ajay Mandlekar, Danfei Xu, Roberto Martín-Martin, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Human-in-the-lown imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733 2020.
[29] Ajay Mandlekar, Danfci Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martin What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.033 032998 2021.
[30] Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webgpt: Browser-assisted question- with human feedback. arXiv preprint arXiv:2112 09332, 2021.
12

---

[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[32] Xincheng Pang, Wenke Xia, Zhigang Wang, Bin Zhao, Di Hu, Dong Wang, and Xuelong Li. Depth helps: Improving pre-trained rgb-based policy with depth information injection. In 2024 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 7251-7256, Oct 2024.
[33] Karl Pertsch, Kyle Stachowicz, Brian Ichter. Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.0974. 2025.
[34] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances Neural Information Processing Systems, 36:53728 -3744,, 2023.
[35] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634f 2017.
[37] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
[38] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy, arXiv preprint arXiv:2405. 12213, 2024.
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13397, 2023.
[40] Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainry, 5:297-323, 1992.
[41] Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 2023.
[42] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24833724-24837, 2022.
[43] Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. Dexvla: Vision-language model with plug-in diffusion expert for general robot control. arXiv preprint arXiv:2502.05885,, 2025.
[44] Wenke Xia, Ruoxuan Feng. Dong Wang, and Di Hu. Phoenix: A motion-based self-reftection framework for fine-grained robotic action correction. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 6981-6990, June 2025.
[45] Wenke Xia, Dong Wang, Xincheng Pang, Zhigang Wang, Bin Zhao, Di Hu, and Xuelong Li. Kinematic-aware prompting for generalizable articulated object manipulation with 1ms.. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2073-2080, 2024.
13

---

[46] Esen Yel and Nicola Bezzo. Fast run on-time monitoring, replanning, and recovery for safe autonomous system operations. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1661-1667. IEFE, 2019.
[47] Zheng Ying, Yao Lei, Su Yuejiao, Zhang Yi, Wang Yi, Zhao Sicheng, Zhang Yiyi, and Chau Lap-Pui. A survey of embodied learning for object-centric robotic manipulation. Machine Intelligence Research, 22:588-624 2025.
[48] Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, et al. Learning manipulation by predicting interaction. arXiv preprint urXiv:2406.00439, 2024.
[49] Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao. Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:2411.19300 2024.
[50] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 190. 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have claimed the contributions in Abstract Section and Introduction
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in the Conclusion Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification: In this work, we apply Kahneman & Tversky's prospect theory to prefer- ence alignment of Vision-Languuge-Action models, without introducing new theoretical contributions
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have reported the implementation details support reproduction in the Experiment Settings Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have released the data and code for reproduction the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//ddeeeSbmimcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the implementation details in Experiment Settings Sce- t.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We conduct the main experiments across 3 random seeds and report the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the compute resources in Experiment Settings Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///aaarrs.s...//////./tttt
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In this work, we propose human-robot collaboration framework ensure op- erational safety during robotic deployment, facilitating the broader adoption of autonomous robotic systems in real-world applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model and dataset pose no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license of VLA VLA model.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: trajectories.
We would release code, model, and corresponding human intervention
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We collect human intervention trajectories by ourselves.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
20

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We utilize the LLMs the foundation models for our further Vision-Language- Action Models fine-tuning.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to LLM policy for what should or should not be described.
21

---

SUPPLEMENTARY MATERIAL
1 Supplementary Video
In this work, we propose the action preference optimization method correct interaction failure and achieve stable optimization for VLA models. In the supplementary video, we illustrate our human-assisted interaction trajectories collection process as demonstrated Figure | We also provide comparison videos against other methods, highlighting the effectiveness of our approach in both real-worke and simulation scenarios.
Automatic Execution
Palure Action
Human Intervention
Autometic Execution
Figure 1: The demonstration of our human-assisted interaction trajectory.
Human-assisted Collaboration Deployment
In this work, we propose a human-assisted collaboration deployment framework to support reliable deployment and interaction trajectory collection. The blue block in Figure [ illlsstrates the initial deployment of the base policy for autonomous environment interaction. However, the base policy trained solely on expert demonstrations. When its predicted action causes failures, this model struggles to recover from these failure states, as shown in the red block. To address this, we provide human intervention to manually adjust the robotic arm's movements for failure correction, as shown in the green blocks.
Through this human-assin assisted approach, we ensure reliable deployment of the model manipulation tasks. Furthermore, we annotate these interaction trajectories for subsequent preference learning. Specifically, we designate the last 10 actions before human intervention as undesirable data (repre- senting failure actions), while the remaining trajectories serve as desirable data.
3 Implementation Details
In our work, we build the utility function as below to estimate the relative gain on the robotic data:
Apo (ro(o, a) =0) if à adesirable
v(o, a)
(1)
Ayo (zo r))), if à dundesirables
where =o = K [LLTTol[Tre) guide the model to learn from preference pair data while simultaneously preserving knowledge acquired from prior models. We compute the KL-divergence 2u by leveraging the KTO [II] method, which leverages mismatched sample pairs for KL estimation. Further, we ignore the reject reward the gripper action token to prevent erroneous rejection the same gripper state.
4 More Real-world Experiments
4.1 Generalization various VLA models
In this section, we adopt our method to fine-tune the TO-FAST model. As shown in Table I the FASAAT model achieves higher success rate, benefiting from its action chunking prediction. Besides, our method could achieve consistent performance gains in real-world experiments. Because our method needs to decode to continuous action for adaptive reweighting, however, the FAST model may fail to decode predicted action tokens into meaningful continuous actions, thus when the predicted action token sequences cannot be decoded x, we would set the weight promote the model focus predicting correct action token sequences.
22

---

(a) Hang cup the
(b) Put the lemon the plate
Figure 2:
The demonstrations of real-world experiments.
Table I: The results on +0-FAST model.
Table 2: The results on real-world experiments.
Methods
Square
Methods
Hang
Put
Base policy
85%
Base policy
70%
85%
Dagger
85%
Dagger
65%
85%
TPO
90%
TPO
75%
80%
Ours
95% %
Ours
90%
100% %
4.2 More real-world tasks
For each task, we collect 100 expert demonstrations to train the base policy. Further, we deploy the base policy to interact with environments and collect 20 human-intervened trajectories. We mix the 20 human- intervened trajectories with 20 expert demonstrations for model preference optimization. As shown in T our method could achieve better performance compared with other behavior cloning and preference optimization methods.
In this section, we provide two more real-world experiments as shown in Figure 2
23


Reviews:
[{"review_id": "aVU1J3ySX8", "reviewer": "Reviewer_uJj6", "summary": "This paper proposes Human-assisted Action Preference Optimization (HAPO) method to improve the deployment reliability and adaptation of Vision-Language-Action (VLA) models in robotic manipulation tasks. HAPO uses a weighted action preference optimization method and an reweighting method to bridge the gap between discrete token prediction and continuous action regression. Experiments in both simulation and real-world demonstrate its effectiveness.", "questions": "- In the real-world experiments, what is the total number of human-labeled trajectories needed for significant improvements?\n\n- Since the binary labeling (desirable vs undesirable) is based on K-step pre-intervention windows, could the authors comment on how sensitive is performence to the choice of K?", "limitations": "See weaknesses and questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- HAPO helps to bridge the gap between discrete token prediction and continuous action regression.\n\n- The authors conducted throughout experiments both in simulation and real-world, their conclusions are well supported by convincing experimental results.\n\n- The paper is generally well-organized.\n\nWeaknesses:\n- Experimental details of the baseline are missing. Suggest adding details about the baseline, such as specific loss function formulas.\n\n- There may be non-rigorous baseline re-implementation. If Dagger is re-implemented properly, then it is unlikely to be worse than the base model. There may be non-rigorous experimental details such as: did you use data with $c_t \\neq 0$ for dagger? Since $c_t=0$ is known to be undesirable data, using such data would be a targeted poisoning of the behavioral cloning methods.\n\n- Many experiments to show that HAPO is better than the baseline in various scenarios (e.g., simulation, real-world, novel tasks, lifelong learning, other VLA, etc.) is nice, of course, but discussions of the intrinsic mechanism of HAPO is missing. For example:\n\n    - Let me illustrate with pseudo-code (Algorithm 1), what are the proportions of $c_t=0,1,2$ in each loop (line 9) respectively? Does the proportion of human intervention gradually decreases as training proceeds? Please draw a figure with iteration $i$ as the horizontal axis and proportion as the vertical axis, with three curves, including $c_0/(c_0+c_1+c_2)$, $c_1/(c_0+c_1+c_2)$, and $c_2/(c_0+c_1+c_2)$. \n\n    - What are the curves of $\\lambda_D$ and $\\lambda_U$? Do they both vary much during training? If not much, is it possible that I can solve the reweighting issues by setting a good and fixed hyperparameter value for $\\lambda_D$ and $\\lambda_U$?\n\nIf the author could provide more analysis and experiments on the mechanisms inherent in HAPO, I would be willing to raise the score.", "was_revised": true, "final_justification": "The authors address my initial concerns:\n\n1. The missing loss functions of the baseline and re-implementation details of DAgger.\n\n2. More analysis on the intrinsic mechanism of HAPO, including the proportions of different action types, ablation study of the reweighting method, and the reward curves of HAPO. These give me a more comprehensive understanding of the underlying mechanism of the proposed method, which was the most important reason for my score increase.", "rebuttal": "Thanks for dedicating time and effort to review our work and providing valuable feedback.\n\n$\\color{green}Q1:$ The experimental details of the baseline.\n\n$\\color{blue}A1:$ \nThanks for this suggestion, we will provide a more comprehensive description in the revised version.\n\nFor all baselines, we deploy base model $\\pi_\\theta$ to collect 50 human-intervened trajectories and mix them with 50 expert demonstrations for fine-tuning. All methods use a learning rate of 5e-5. For BC-based methods, we treat observation-action pairs $(o_t, a_t)$ with $c_t \\neq 0$ as supervised learning samples. For preference-based methods, pairs with $c_t \\neq 0$ are desirable, and those with $c_t = 0$ are undesirable.\n\nBelow are loss functions and implementation details. For brevity, we only show desirable sample loss for KTO and HAPO:\n\n| Method | Loss Function | Implementation Details |\n|--------|---------------|------------------------|\n| Dagger | $L_{dagger}(\\pi_\\theta) = -\\sum_{(o_i,a_i)\\in D} \\log \\pi_\\theta(a_i\\|o_i)$  |  |\n| Sirius |  $ L_{sirius}(\\pi_\\theta) = -\\sum_{(o_i,a_i)\\in D} w_i \\log \\pi_\\theta(a_i\\|o_i)$ | $w_i$ is estimated by ratio of number of human-intervened actions.|\n| DPO | $L_{DPO}(\\pi_\\theta; \\pi_{ref}) = -E_{(o_i,a_d,a_u) \\sim D}[\\log \\sigma( \\log(\\frac{\\pi_\\theta(a_d\\|o_i)}{\\pi_{ref}(a_d\\|o_i)} - \\log(\\frac{\\pi_\\theta(a_u\\|o_i)}{\\pi_{ref}(a_u\\|o_i)}))]$ | undesirable action $a_u$ is generated by base model $\\pi_{ref}$ on observation $o_i$ with random action noise. |\n| TPO | $L_{TPO}(\\pi_\\theta; \\pi_{ref}) = -E_{(\\zeta_d, \\zeta_u) \\sim D}[\\log \\sigma( \\log(\\frac{\\pi_\\theta(\\zeta_d)}{\\pi_{ref}(\\zeta_d)} - \\log(\\frac{\\pi_\\theta(\\zeta_u)}{\\pi_{ref}(\\zeta_u)}))]$ |  We recognize pre-intervention sub-trajectory as undesirable sub-trajectory and other sub-trajectory as desirable one. |\n| KTO | $L_{KTO}(\\pi_\\theta; \\pi_{ref}) = -E_{(o_i,a_i) \\sim D}[ \\sigma( \\frac{\\pi_\\theta(a_i\\|o_i)}{\\pi_{ref}(a_i\\|o_i)} - KL(\\pi_\\theta, \\pi_{ref}))]$ |  |\n| HAPO(ours) | $L_{HAPO}(\\pi_\\theta; \\pi_{ref}) = -E_{(o_i,a_i) \\sim D}[ (1 - e^{-\\beta_D*w_i}) * \\sigma( \\frac{\\pi_\\theta(a_i\\|o_i)}{\\pi_{ref}(a_i\\|o_i)} - KL(\\pi_\\theta, \\pi_{ref}))]$| We achieve sample-wise reweighting control on $w_i$ to prioritize error-prone samples. |\n\nThe experiments on various downstream tasks validate the effectiveness of our HAPO method, and we also provide more comprehensive analysis in Q3.\n\n\n---\n$\\color{green}Q2:$ The re-implementation of DAgger.\n\n$\\color{blue}A2:$ Thanks for this question to ensure fair comparison for all baselines.\n\nWe followed Sirius[1] setup to re-implement both DAgger and Sirius, using only observation-action pairs $(o_i, a_i)$ with $c_i \\neq 0$ as supervised learning samples in our paper. This ensures a rigorous re-implementation of DAgger.\n\nIt is common that the updated model underperforms the base model in interactive imitation learning, as shown in Table 2 of the OLAF paper[2] and the Sirius-Runtime-Monitor paper[3], where HG-DAgger and ThriftyDAgger are variants of DAgger. As discussed in Section 4.1 of our paper, one possible reason for this phenomenon is that BC-based methods often struggle to retain the knowledge of the base model when learning from the distribution-shifted human-intervened data.\n\nThe Table 2 results in OLAF paper:\n\n| Method | Success Rate |\n|---|---|\n| Base Model |  84.4 |\n| HG-DAgger |  75.0 |\n\nThe Table 2 results in sirius-runtime-monitor paper:\n\n| Method | Round 1 | Round 2 | Round 3 |\n|---|---|---|---|\n| ThriftyDAgger |  79.5 | 75.6 | 77.5 |\n\n---\n$\\color{green}Q3:$ More analysis on intrinsic mechanism of HAPO.\n\n$\\color{green}Q3.1:$ Proportions of different action types.\n\n$\\color{blue}A3.1:$ \nWe appreciate this insightful question. We have reported the human-intervention proportions ratio and succcess rate during multi-round deployment-optimization loops in Figure 3 of our paper, demonstrating that our HAPO method continuously improves through interaction with environment. Below is a more detailed table:\n\nProportions in Coffee_D0:\n\n| Action Type  | Round 1 | Round 2 | Round 3 |\n|----|--------------|------- |------- |\n|  Failure Action $\\frac{c_0}{c_0 + c_1 + c_2}$| 5.17%         | 5.57% | 3.73% |\n|  Autonomous Action $\\frac{c_1}{c_0 + c_1 + c_2}$ | 77.44%     | 79.28% | 87.51% |\n|  Human-intervened Action $\\frac{c_2}{c_0 + c_1 + c_2}$ | 17.39%       | 15.15% | 8.76% |\n\nProportions in StackThree_D0:\n\n| Action Type  | Round 1 | Round 2 | Round 3 |\n|----|--------------|------- |------- |\n|  Failure Action $\\frac{c_0}{c_0 + c_1 + c_2}$| 4.59%         | 4.07% | 3.52% |\n|  Autonomous Action $\\frac{c_1}{c_0 + c_1 + c_2}$ | 81.97%     | 83.26% | 85.09% |\n|  Human-intervened Action $\\frac{c_2}{c_0 + c_1 + c_2}$ | 13.43%       | 12.67% | 11.39% |\n\nThe results prove that our HAPO method could effectively learn from interaction data and achieve self-improvement continually.\n\n$\\color{green}Q3.2:$ Curves of $\\lambda_U$ and $\\lambda_D$. Is it possible to solve the reweighting issues with a good and fixed hyperparameter?\n\n$\\color{blue}A3.2:$ \nOur $\\lambda_U$ and $\\lambda_D$ are sample-wise weights with batch-level normalization, designed to prioritize samples with large regression errors. Due to their sample-wise nature, it's difficult to capture trends with a curve. Instead, we provide an example below showing how $\\lambda$ values vary across samples based on their regression errors.\n\n|Type | L1 loss| $\\lambda$| \n|---|---|---|\n| Undesirable Action | [0.05, 0.25] | [0.83, 0.44] |\n| Desirable Action | [0.05, 0.01, 0.39, 0.0, 0.16, 0.15] | [0.06, 0.02, 0.40, 1.2e-4, 0.18, 0.18] |\n\nThe $\\lambda_U$ and $\\lambda_D$ values show that higher weights are given to desirable samples with large regression errors and undesirable samples similar to failure actions.\n\nTo assess whether fixed weights can replace adaptive reweighting, we conducted experiments using fixed values: $\\lambda_U = 0.5$; $\\lambda_D = 1$ for human-intervened actions and 0.5 for autonomous actions to emphasize human-intervened action:\n\nCoffee_D0:\n|Method | Success Rate|\n|---|---|\n|HAPO w/. fixed weight | $51.3 \\pm 3$ |\n|HAPO | $60.0 \\pm 2$ |\n\n\nStackThree_D0\n|Method | Success Rate|\n|---|---|\n|HAPO w/. fixed weight | $45.3 \\pm 6.1$ |\n|HAPO | $53.3 \\pm 5$ |\n\nSquare_D0:\n|Method | Success Rate|\n|---|---|\n|HAPO w/. fixed weight | $30.0 \\pm 3.5$ |\n|HAPO | $31.3 \\pm 3.1$  |\n\nThreePiece_D0\n|Method | Success Rate|\n|---|---|\n|HAPO w/. fixed weight | $38.7 \\pm 2.3$ |\n|HAPO | $43.3 \\pm 5$ |\n\nResults prove that our adaptive, sample-wise reweighting is more effective than fixed parameters, leading to significantly better performance.\n\n$\\color{green}Q3.3:$ More analysis on the HAPO methods.\n\n$\\color{blue}A3.3:$\nTo evaluate how HAPO effectively promotes learning from human-intervened data via adaptive reweighting, we track the reward of KTO and HAPO during training. The reward is computed as $r_{\\theta}(o, a) = \\log \\frac{\\pi_\\theta(a|o)}{\\pi_{ref}(a|o)}$, where a higher reward for desirable actions indicates stronger model confidence, and a lower reward for undesirable actions suggests better avoidance.\n\nDesirable Reward\n\n| Method| Iter. 1000 | Iter. 2000 | Iter. 3000 |\n|---|---|---|---|\n| KTO | 11.0 | 11.8 | 15.7 |\n| HAPO | 10.2 | 12.9 | 17.4 |\n\nUndesirable Reward\n\n| Method| Iter. 1000 | Iter. 2000 | Iter. 3000 |\n|---|---|---|---|\n| KTO | -10.5 | -18.9 | -26.6 |\n| HAPO | -18.7 | -24.9 | -35.5 |\n\nThe reward curves demonstrate that HAPO effectively reweights training samples, encouraging the model to learn more valuable interaction data.\n\n---\n$\\color{green}Q4:$ The total number of human-labeled trajectories needed for significant improvements in the real-world experiments.\n\n$\\color{blue}A4:$ As detailed in Section 4.6 of our paper, we collect 100 expert demonstrations to train the base model, followed by 20 human-intervened trajectories for fine-tuning, leading to significant improvements in downstream tasks.\n\nWe observe that robot often fails due to similar failure modes, such as misalignment in the \"Insert\" task or failed grasps in the \"Pick\" task, as shown in the supplementary videos. By learning corrective actions from human interventions, the model adapts to these tasks and avoids such failures, even with limited data.\n\nResults in Table 5 of our paper and Table 2 in the supplementary material confirm that our approach enhances both performance and generalization to new disruptions. The supplementary videos further demonstrate the learned correction behaviors.\n\n---\n$\\color{green}Q5:$ How does the pre-intervention window size $K$ affect model performance?\n\n$\\color{blue}A5:$ \nThank you for this insightful question. The choice of $K$ determines \"failure action\" in preference optimization method, plays a crucial role in identifying which actions should be considered failures for model to avoid:\n\n| Task Name   | $K=5$  | $K=10$ | $K=15$ | $K=20$ |\n|-------------|------|------|------|------|\n| Coffee_D0   | $54.6 \\pm 6.1$ | $60.0 \\pm 2$ | $59.3 \\pm 7.5$  | $63.3 \\pm 4.6$ |\n| StackThree_D0   | $51.3 \\pm 2.3$ | $53.3 \\pm 5$| $56.6 \\pm 4.6$ | $54.6 \\pm 5.0$  |\n| Square_D0   | $28.6 \\pm 3.1$  | $31.3 \\pm 3.1$ | $32.0 \\pm 5.3 $ | $29.3 \\pm 2.3$ |\n| ThreePieceAssembly_D0   | $39.3 \\pm 5.0$ | $43.3 \\pm 5.0$| $42.6 \\pm 3.1$ | $34.6 \\pm 4.1$ |\n\nAs shown, a small $K$ may fail to capture the full failure actions, potentially causing the model to misclassify them as desirable. Larger $K$ better capture failure sequences but may introduce noise depending on the task's failure modes. Based on empirical results, we select $K=10$ as a general setting across tasks. \n\nWe believe automatically identifying appropriate failure action sequences for preference learning remains an interesting direction for future work.\n\n---\n[1] Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment. RSS 2023, Best Paper Finalist.\n\n[2] Interactive Robot Learning from Verbal Correction. CoRL 2023.\n\n[3] Model-based runtime monitoring with interactive imitation learning. ICRA 2024."}, {"review_id": "vk6ZVFwqsY", "reviewer": "Reviewer_ScuL", "summary": "This paper proposes a VLA optimization method, Human-assisted Action Preference Optimization (HAPO), which includes a human-robot collaboration data collection framework and an adaptive reweighting algorithm. The authors claim that the proposed adaptive reweighting algorithm addresses intrinsic issues in VLA optimization—specifically, the irreversible nature of interactions during data collection and the mismatch between token probabilities and continuous action errors. The paper presents extensive experiments demonstrating that HAPA outperforms DPO, KPO, TPO, etc., in optimizing VLAs, improving robustness to dynamic environmental changes, and generalizing across different VLA architectures.", "questions": "1. **Trade-off for more precise and fine-grained optimization**: Since HAPO uses Prospect Theory to construct binary labels instead of pairwise data for policy optimization, one concern is that providing only coarse supervision—i.e., judging whether a trajectory is good or bad—may be too vague. In this case, the model might learn undesirable shortcuts or stereotyped behaviors that minimize the loss, rather than learning diverse and robust motion patterns to achieve the goal.\n2. **Questionable quantitative results**: Based on Table 1, the overall performance gain of HAPO over KTO does not appear to be very substantial. Without error bars, it is difficult to assess whether the reported improvements are statistically meaningful. Furthermore, it seems that most of the gains come primarily from the Coffee_D0 task, which raises concerns about statistical significance and broader applicability.", "limitations": "Please refer to the Weaknesses section to address the primary concerns. I also have the following additional questions:\n\n- As mentioned above, Prospect Theory introduces concerns regarding the use of coarse supervision signals. I’m curious that when humans collaborate with the robot to create new trajectories based on the failed ones from the base model, what kind of behavior do they follow? Do they try to mimic the failed trajectory but correct it slightly, or do they focus entirely on achieving the goal, without caring about replicating the original path?\n\n- The paper also claims to address the token mismatch issue through adaptive reweighting (Equations 4–6), but lacks an ablation study isolating the benefit of this specific component. It would be helpful to understand how much performance gain can be attributed directly to this reweighting mechanism.\n\n- Finally, regarding statistical significance: across both the simulation and real-world experiments, the paper presents results on a limited set of tasks and without error bars. This is particularly concerning given that the performance boost of HAPO over the baselines is not very large in absolute terms. More rigorous statistical reporting would help support the claimed improvements.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The work conducts a wide range of experiments and ablations to evaluate HAPO—not only for its core function in model optimization, but also for its robustness to dynamic environmental changes and its generalization across different VLAs. I believe all of these are important metrics for assessing a VLA optimization method.", "was_revised": true, "final_justification": "The work conducts a wide range of experiments and ablations to evaluate HAPO—not only for its core function in model optimization, but also for its robustness to dynamic environmental changes and its generalization across different VLAs. I believe all of these are important metrics for assessing a VLA optimization method. During the rebuttal, the authors have conducted extensive additional experiments to address my questions, including the statistical significance and additional explanation of reweighting methods. Therefore, I recommend acceptance of this paper.", "rebuttal": "Thank you for recognizing our work. We sincerely appreciate your time and effort in reviewing our work and offering insightful feedback.\n\n$\\color{green}Q1:$ Trade-off for more precise and fine-grained optimization.\n\n$\\color{blue}A1:$ \nThanks for this valuable question. One of the motivations of our work is to **provide fine-grained action correction for the VLA models**. To encourage the model to learn how to avoid the failure action and achieve reliable failure correction, we design the human-assisted action preference optimization (HAPO) method, which learns from the **fine-grained action-level binary desirability signals** instead of the trajectory-level coarsed-grained signals.\n\nSpecifically, as illustrated in Algorithm 1, during the deployment phase, we first deploy the base model $\\pi_\\theta$ in the environment to collect human-intervened interaction trajectories. These trajectories are then re-labeled with binary action-level desirability signals based on the timing of human interventions: the $K$ steps preceding each intervention are considered undesirable action sequences.\n\nTo ensure that the preference optimization method is well-suited for VLA models, we incorporate Kahneman \\& Tversky’s prospect theory to guide the optimization process with fine-grained binary desirability signals. We also propose a reweighting strategy to bridge the gap between discrete token prediction and continuous action regression, promoting the model to pay more attention to the fine-grained error-prone actions.\n\nThrough the above algorithm, our human-assisted action preference optimization method can **learn from the action-level binary desirability signals to provide fine-grained action correction for the VLA models**. We also provide the supplementary videos to show the learned correction patterns of our model.\n\n---\n$\\color{green}Q2:$ The behavior of the our human-assisted action preference optimization method.\n\n$\\color{blue}A2:$ \nIt is an interesting question to check the behavior of our human-assisted action preference optimization method. We observe that the robot often fails on some similar failure modes. For instance, in the \"Coffee_D0\" simulation task, the most common failure involves the model being unable to insert the coffee pod into the machine. Similarly, in the \"Insert\" real-world task, the model often fails to align the stick with the square hole.\n\nDuring the deployment phase, we manually intervene to correct such failures. These failure actions are labeled as undesirable data, while the corresponding human-intervened actions are treated as desirable data for preference optimization.\n\nBy learning corrective actions from human interventions, the model effectively adapts to downstream tasks and avoids these failure patterns, even with limited data.\nAs demonstrated in the supplementary videos, the model optimized via our HAPO method is capable of recovering from these failure scenarios. For example, in the Coffee_D0 task, the model learns to slightly adjust the gripper position to successfully insert the coffee pod. In the real-world square insertion task, the model learns to retry the alignment when the stick is initially misaligned with the hole. For a more comprehensive understanding of our method’s behavior, please refer to the supplementary videos.\n\n---\n$\\color{green}Q3:$ More quantitative results with statistical significance.\n\n$\\color{blue}A3:$\nThanks for this valuable question. In this work, to comprehensively evaluate the performance of our method, we compared our HAPO methods with other baselines in two settings: the in-distribution setting, the out-of-distribution setting.\n\nIn the in-distribution setting, to thoroughly validate the effectiveness of our method, we report the average success rate and standard deviation for simulation tasks in Table 1, based on 50 rollouts with 3 different seeds. Additionally, we introduce a new simulation task, \"Cleanup Mug,\" and a real-world task, \"Stack Blocks,\" to further assess our method's performance. The \"Cleanup Mug\" task is a long-horizon task in which the robot must open a drawer, place a mug inside, and then close the drawer. The \"Stack Blocks\" task is a fine-grained task that requires the robot to stack a 2cm block on top of another. For real-world tasks, we perform 20 rollouts. The results are shown in the following table:\n\nCoffee_D0:\n|Method | Success Rate|\n|---|---|\n|Base Model | $44.7 \\pm 1.2$ |\n|KTO | $48.6 \\pm 4.2$ |\n|HAPO | $60.0 \\pm 2$ |\n\n\nStackThree_D0:\n|Method | Success Rate|\n|---|---|\n|Base Model | $42 \\pm 7.2$ |\n|KTO | $50.7 \\pm 4.2$ |\n|HAPO | $53.3 \\pm 5$ |\n\nSquare_D0:\n|Method | Success Rate|\n|---|---|\n|Base Model | $28.0 \\pm 4.0$ |\n|KTO | $28.6 \\pm 3.1$ |\n|HAPO | $31.3 \\pm 3.1$  |\n\nThreePiece_D0:\n|Method | Success Rate|\n|---|---|\n|Base Model | $ 39.3 \\pm 5.0$ |\n|KTO | $41.3 \\pm 5.7$ |\n|HAPO | $43.3 \\pm 5.0$ |\n\nReal World Stack Block(New):\n|Method | Success Rate|\n|---|---|\n|Base Model | $ 48.3 \\pm 2.9$ |\n|KTO | $51.7 \\pm 2.9$ |\n|HAPO | $63.3 \\pm 5.7$ |\n\nSimulation Cleanup Mug(New):\n|Method | Success Rate|\n|---|---|\n|Base Model | $ 30.6 \\pm 9.8$ |\n|KTO | $36.6 \\pm 3.0$ |\n|HAPO | $42 \\pm 8.7$ |\n\n\nIn the out-of-distribution setting, the results in Table 2 and Table 3 of our paper show that our method achieves consistent improvement over the baseline methods in most scenarios. We also evaluate the baseline methods for 50 rollouts with 3 different seeds and report the average success rate and standard deviation to validate the stability of our method:\n\nBackground Disruption:\n|Method | Success Rate|\n|---|---|\n|Base Model | $39.3 \\pm 4.2$ |\n|KTO | $40.6 \\pm 3.0$ |\n|HAPO | $44.7 \\pm 1.1$ |\n\nThe results show that our method could achieve consistent improvement over the baseline methods in both in-distribution and out-of-distribution settings with statistical significance.\n\n---\n$\\color{green}Q4:$ The analysis of the reweighting method.\n\n$\\color{blue}A4:$\nThank you for this valuable question about the proposed adaptive reweighting method.\n\n### 1. Explaination of the reweighting methods:\nFirst, we would like to clarify the motivation behind our reweighting method. The core motivation of our action-level preference optimization approach is to encourage the model to learn correct behaviors while avoiding failure actions. Traditional preference optimization methods in LLMs focus solely on token probabilities and overlook the semantic meaning of the decoded words. For example, if the ground-truth token is _50_, both _10_ and _49_ would be considered equally incorrect.\n\nHowever, in robot decision making with discrete action space, the continuous action space is encoded to discrete tokens, which means different tokens correspond to different physical actions. For example, the action token $i$ represents 'move along the x-axis by $0.i$ cm'. Therefore, it is inappropriate to treat token _10_ and _49_ as equally incorrect. For instance, the desirable action token _50_ may represent \"move along the x-axis by 5 cm\", while token _49_ corresponds to \"move along the x-axis by 4.9 cm\", and token _10_ to \"move along the x-axis by 1 cm\". Clearly, action token _49_ is much closer to the desired action than _10_.\n\nOur reweighting method is designed to leverage this continuous action information for discrete action token preference optimization. This approach bridges the gap between discrete token prediction and continuous action regression, making preference optimization more effective for autoregressive VLA models.\n\n### 2. Analysis of the reweighting method:\n\nOur $\\lambda_U$ and $\\lambda_D$ are sample-wise weights with batch-level normalization, designed to prioritize samples with large regression errors. Here, we provide an example below showing how $\\lambda$ values vary across samples based on their regression errors.\n\n|Type | L1 loss| $\\lambda$| \n|---|---|---|\n| Undesirable Action | [0.05, 0.25] | [0.83, 0.44] |\n| Desirable Action | [0.05, 0.01, 0.39, 0.0, 0.16, 0.15] | [0.06, 0.02, 0.40, 1.2e-4, 0.18, 0.18] |\n\nThe values of $\\lambda_U$ and $\\lambda_D$ indicate that higher weights are assigned to desirable samples with large regression errors and to undesirable samples similar to failure actions.\n\nTo assess whether our adaptive reweighting method is effective, we conducted experiments using fixed values: $\\lambda_U = 0.5$; $\\lambda_D = 1$ for human-intervened actions and 0.5 for autonomous actions to emphasize the human-intervened action. Results are shown below:\n\nCoffee_D0:\n|Method | Success Rate|\n|---|---|\n|HAPO w/o. reweighting | $51.3 \\pm 3.0$ |\n|HAPO | $60.0 \\pm 2$ |\n\nStackThree_D0:\n|Method | Success Rate|\n|---|---|\n|HAPO w/o. reweighting  | $45.3 \\pm 6.1$ |\n|HAPO | $53.3 \\pm 5$ |\n\nSquare_D0:\n|Method | Success Rate|\n|---|---|\n|HAPO w/o. reweighting | $30.0 \\pm 3.5$ |\n|HAPO | $31.3 \\pm 3.1$  |\n\nThreePiece_D0\n|Method | Success Rate|\n|---|---|\n|HAPO w/o. reweighting | $38.7 \\pm 2.3$ |\n|HAPO | $43.3 \\pm 5.0$ |\n\nThese results prove that adaptive, sample-wise reweighting could lead to significantly better performance.\n\n### 3. Analysis of the reward curves:\nFurther, to evaluate whether HAPO effectively promotes learning from human-intervened data via adaptive reweighting, we track the reward of KTO and HAPO during training. The reward is computed as $r_{\\theta}(o, a) = \\log \\frac{\\pi_\\theta(a|o)}{\\pi_{ref}(a|o)}$, where a higher reward for desirable actions indicates stronger model confidence, and a lower reward for undesirable actions suggests better avoidance.\n\nDesirable Reward\n\n| Method| Iter. 1000 | Iter. 2000 | Iter. 3000 |\n|---|---|---|---|\n| KTO | 11.0 | 11.8 | 15.7 |\n| HAPO | 10.2 | 12.9 | 17.4 |\n\nUndesirable Reward\n\n| Method| Iter. 1000 | Iter. 2000 | Iter. 3000 |\n|---|---|---|---|\n| KTO | -10.5 | -18.9 | -26.6 |\n| HAPO | -18.7 | -24.9 | -35.5 |\n\nThe reward curves demonstrate that HAPO effectively reweights training samples, encouraging the model to learn more valuable interaction data."}, {"review_id": "UrnaD0SmWw", "reviewer": "Reviewer_3Xzy", "summary": "The authors introduce HAPO, a preference-tuning algorithm that enables interactive imitation learning for VLAs. Whilst the goal and some of the pipeline is similar to prior algorithms, such as Sirius, the authors show that adapting VLAs require more specialised update rules. The authors show the benefits of their algorithm against baselines, generalisation and continual learning, application to more than one VLA, and real-world robot experiments.", "questions": "How were hyperparameter chosen for the different methods in the experiments? It's not clear that every method would perform best with the same set of hyperparameters.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Given some experience with robot learning, I feel the main strength of this work is the experimental section, which demonstrates the performance improvement given by using their algorithm, as opposed to various more model-agnostic algorithms and other preference tuning algorithms.\n\nInitially, I would have questioned the advancement of HAPO over Sirius, but the authors do use Sirius as a baseline and show that their method is better for optimising VLAs. Nowadays most robot learning policies tend to use diffusion over autoregressive prediction, but I would consider this beyond the scope of this work (the authors also acknowledge this in the limitations section).\n\nThe authors could expand the explanation of HAPO to make it easier to grasp, but given space limitations the current size of the various sections are reasonable.", "was_revised": true, "final_justification": "I have read the other reviews and the authors' responses, and believe that my rating of 5 is warranted due to the contributions and depth of the work.", "rebuttal": "Thank you for recognizing our work. We sincerely appreciate the time and effort you dedicated to reviewing our paper and providing valuable feedback.\n\n$\\color{green}Q1:$ The advancement of HAPO over Sirius.\n\n$\\color{blue}A1:$ \nThanks for this question. We believe Sirius is a great and solid work in the area of human-robot collaboration and continual policy learning.\n\nThe award of this paper (RSS 2023 Best Paper Finalist) demonstrates the embodied AI community's strong emphasis on the **reliable robotic deployment and lifelong learning of robot policies** in real-world scenarios.\n\nHowever, Sirius relies on behavior cloning to continually optimize its policy, which leads to several limitations listed below. Thus, we propose a human-assisted action preference optimization method to solve these issues:\n\n1. Behavior cloning cannot effectively leverage failed actions from human-robot interaction trajectories, even though these are highly informative for teaching the model what to avoid. Our preference optimization method **incorporates these failure actions to guide learning away from undesired behaviors**.\n\n2. As shown in the experiment results, the behavior cloning method is struggling to both retain the prior knowledge and adapt to new distribution shifts of interaction trajectories. While our method utilize the reference model to constrain the policy optimization for prior knowledge retain, and utilize a reweighting strategy to **encourage model to avoid failure actions and correct from failure cases**.\n\nThe experiment results demonstrate that our mehtod could achieve better performance on various novel downstream tasks compared with Sirius and other baselines, different autoregressive VLA models and real-world tasks.\n\n---\n$\\color{green}Q2:$ The difference between diffusion policy and autoregressive policy.\n\n$\\color{blue}A2:$ \nThanks for this question. It is a hot and important topic about the optimal architecture for VLA models. \nDue to the lack of robotic data and the wonderful generalizability of VLMs, no matter the diffusion-based or autoregressive-based VLA models are mainly built on the VLM backbone. For example, the OpenVLA[1], RT-2[2], $\\pi_0$-FAST[3] encode the action to discrete tokens and model the action prediction task as an autoregressive task. While $\\pi_0$[4], Go-1[5] utilize flow-based models as an additional action expert for robotic action prediction. \n\nHowever, there is still no definitive conclusion that whether diffusion-based methods or autoregressive methods are better for generating robotic actions. Although the diffusion-based methods have better ability to build the multi-modal action distribution, recent papers show that the autoregressive methods can also achieve good performance in robotic action prediction. \nFor example, the OpenVLA-OFT[6] show that the OpenVLA model can also achieve better performance than $\\pi_0$ with the action chunking trick. While the knowledge insulation[7] paper shows that the gradient of the flow-based model would harm the knowledge retention of the VLM. \n\nIn our work, the proposed HAPO method is mainly designed for the autoregressive model, and we are also interested in how to apply our method to the diffusion-based model, which would be a future work. Thanks for this valuable question.\n\n\n\n---\n$\\color{green}Q3:$ The motivation and explanation of our human-assisted action preference optimization.\n\n$\\color{blue}A3:$ \nThanks for this question. We would provide a more comprehensive description of the motivation and explanation in the revised version of this work.\n\nOur vision is to enable robots to truly serve the real world, which requires that **robots can be deployed reliably and continuously improve their capabilities** in real-world scenarios.\n\nPrior works (Sirius) on human-in-the-loop learning mainly focus on the behavior cloning method, which cannot fully utilize the failure actions of the human-robot interaction trajectory and struggle to retain the prior knowledge for robotic lifelong learning on large-scale VLA models.\n\nThus, we propose the human-assisted action preference optimization method to correct action-level robotic behavior to solve these issues as mentioned in the Answer 1.\n\nConcretely, our HAPO method include two components:\n\n+ The **Human-robot Collaboration Deployment phase** would deploy the base policy to interact with environment to collect human-intervented interaction trajectory. Depending on the time of human intervention, we would label the action sequence before human intervention as undesirable action and the others as desirable action.\n\n+ The **Action Preference Optimization phase** would optimize the base policy with collected human-intervened trajectories. To make preference learning method work on VLA models optimizaiton, we propose the following techniques:\n    + We introduce the  Kahneman \\& Tversky’s prospect theory for preference optimization with binary desirability signals, which release the demand constrain of paired positive-negative actions under the same observational conditions\n    + We propose a reweighting strategy to bridge the gap between token classification and continuous action regression in autoregressive VLA models, which would adaptively improve the weight of error-prone actions.\n\nThrough iterative human-robot collaboration deployment and action preference optimization, we could achieve continual improvement from interaction with environments for autoregressive VLA models.\n\n---\n$\\color{green}Q4:$ The hyperparameter for different models.\n\n$\\color{blue}A4:$ Thanks for this suggestion, we would provide a more comprehensive description of the implementation details in the further vision of this work.\n\nFor all baselines, we first deploy the base model $\\pi_\\theta$ on the environment to collect 50 human-intervened interaction trajectories. Subsequently, we mix these 50 human-intervened trajectories with 50 expert demonstrations to fine-tune the base model using different baseline methods. All methods are trained with a learning rate of 5e-5. For BC-based methods, we use the observation-action pairs (o_t, a_t) with $c_t \\neq 0$ as supervised learning samples. While for preference learning based methods, we recognize the observation-action pairs (o_t, a_t) with $c_t = 0$ as undesirable samples and recognize the observation-action pairs (o_t, a_t) with $c_t \\neq 0$ as desirable samples.\n\nBesides, it is an interesting question to explore the influence of failure action selection on the performance of different methods, because the selection of $K$ plays a crucial role in identifying which actions should be considered failures for the model to avoid. We provide the performance of different methods with different $K$ values in the following table.\n\nResults on the Coffee_D0 task:\n\n| Task Name   | $K=5$  | $K=10$ | $K=15$ | $K=20$ |\n|-------------|------|------|------|------|\n| KTO   | $48.6 \\pm 4.2$ | $52 \\pm 3.4$ | $54 \\pm 6$  | $54 \\pm 11.1$ |\n| HAPO (ours)   | $54.6 \\pm 6.1$ | $60.0 \\pm 2$ | $59.3 \\pm 7.5$  | $63.3 \\pm 4.6$ |\n\n\nResults on the StackThree_D0 task:\n\n| Task Name   | $K=5$  | $K=10$ | $K=15$ | $K=20$ |\n|-------------|------|------|------|------|\n| KTO   | $51.3 \\pm 4.6$ | $50 \\pm 5.3$ | $50.6 \\pm 4.6$  | $50.6 \\pm 4.1$ |\n| HAPO (ours)    | $51.3 \\pm 2.3$ | $53.3 \\pm 5$| $56.6 \\pm 4.6$ | $54.6 \\pm 5.0$  |\n\nThe results prove that our method could achieve consistent performance improvement than other baselines on different tasks.\n\n\n---\n[1] OpenVLA: An Open-Source Vision-Language-Action Model.\n\n[2] RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.\n\n[3] FAST: Efficient Robot Action Tokenization.\n\n[4] $\\pi_0$: Our First Generalist Policy.\n\n[5] Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.\n\n[6] Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success.\n\n[7] Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better."}, {"review_id": "TsDpa4t83n", "reviewer": "Reviewer_Qjwp", "summary": "The paper proposes to leverage the intervention of humans during robot manipulation as a preference to guide the learning of the policy for robust behavior. \n\nWith a base policy trained on demonstrations, a human will take over when the robot encounters problems, forming an intervention dataset. The base policy will be iteratively fine-tuned on the collected data with preference-based optimization. KL divergence is applied to make sure the policy do not shift too much from the base policy. A reweighting method helps the learning focus on important data points.\n\nEvaluation in both simulation and the real world shows improvement compared to BC methods and other preference-based methods.", "questions": "Please see the weaknesses. I am willing to raise the score if my concerns are solved.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. The idea of making VLA continuously improve during deployment while minimizing the human efforts is well-motivated.\n2. The setting of the experiments and corresponding analysis is great.\n3. The writing is clear and easy to follow.\n\n# Weaknesses\n1. My first concern is about the position of this paper. While the authors point out that previous methods learning from human feedbacks [1] necessitate constant human supervision, this paper follows the same way of intervention. Can I understand that the focus of this paper is on perfence-based optimization and reweighting?\n2. The design of the reweighting needs further justification. The proposed reweighting is based on the loss term, assuming that more significance should be given to the data that causes bigger shift in the base model. Does this contradict to the KL divergence? How to determine the balance between being adaptive and keeping priors? Besides, it not very clear to me why such reweighting could solve the problem of tokenization. Is there any intuition or mathematical proof behind this?\n3. The setting for comparison with baselines is collecting 50 trajectories with intervention and optimize. I am wondering the performance difference if there are more iterations of collection-optimized-collection but keep the total number of trajectories to be the same (i.e. 50).\n\n[1] Liu, Huihan, et al. \"Robot learning on the job: Human-in-the-loop autonomy and learning during deployment.\"", "was_revised": true, "final_justification": "During rebuttal. The author clarified the difference with previous work, which addresses my concern about the novelty. The reweighting design also makes more sense to me now.", "rebuttal": "Thanks for dedicating time and effort to review our work and providing valuable feedback.\n\n$\\color{green}Q1:$ position and focus of this paper.\n\n$\\color{blue}A1:$ \nOur vision is to enable VLA models to serve the real world, which requires that VLA models can be **deployed reliably** and **continuously improve their capabilities** in real-world scenarios.\n\nHowever, prior interactive imitation learning works (Sirius) focus on the behavior cloning method, which cannot fully utilize the failure actions of the human-robot interaction trajectory and struggle to retain the prior knowledge for robotic lifelong learning on large-scale VLA models. While existing RL-based methods encounter a challenging optimization landscape in training large-scale VLA models, due to the inherent instability and challenge of developing generalizable value functions. \n\nThus, we propose to build a framework to ensure **reliable continual policy optimization for large-scale VLA models**. From technical perspective, we propose the human-assisted action preference optimization method to achieve action-level robotic behavior correction with the reweighting strategy on the preference optimization.\n\nConcretely, our human-assisted action preference optimization method include two components:\n\n+ The Human-robot Collaboration Deployment phase would deploy the base policy to interact with environment to collect human-intervented interaction trajectory. Depending on the time of human intervention, we would label the action sequence before human intervention as undesirable action and the others as desirable action.\n\n+ The Action Preference Optimization phase would optimize the base policy with collected human-intervened trajectories. To make preference learning method work on VLA models optimizaiton, we propose the following techniques:\n    + We introduce the  Kahneman \\& Tversky’s prospect theory for preference optimization with binary desirability signals, which release the demand constrain of paired positive-negative actions under the same observational conditions\n    + We propose a reweighting strategy to bridge the gap between token classification and continuous action regression in autoregressive VLA models, which would adaptively improve the weight of error-prone actions.\n\nThrough iterative human-robot collaboration deployment and action preference optimization, we could achieve continual improvement from interaction with environments for autoregressive VLA models.\n\n---\n$\\color{green}Q2:$ The performance with multi-turn deployment-optimization loops.\n\n$\\color{blue}A2:$ Thanks for this valuable question. We have conducted the multi-round deployment-optimization loops in the Section 4.4 of our paper. In this lifelong learning experiment, we conduct the deployment-optimization loops for 3 rounds with every 20 interaction trajectories. \n\nThe results in Coffee_D0 of Figure 3 in our paper are shown below:\n| Base Policy | Round 1 | Round 2 | Round 3 |\n|---|---|---|---|\n| 44 | 54 | 56 | 62 |\n\nThe proportions of different action types in each round in Coffee_D0:\n| Action Type  | Round 1 | Round 2 | Round 3 |\n|----|--------------|------- |------- |\n|  Failure Action $\\frac{c_0}{c_0 + c_1 + c_2}$| 5.17%         | 5.57% | 3.73% |\n|  Autonomous Action $\\frac{c_1}{c_0 + c_1 + c_2}$ | 77.44%     | 79.28% | 87.51% |\n|  Human-intervened Action $\\frac{c_2}{c_0 + c_1 + c_2}$ | 17.39%       | 15.15% | 8.76% |\n\nThe results of Figure 3 in our paper prove that our method could effectively learn from human-intervened data and achieve self-improvement continually.\n\nTo comprehensively evaluate the performance of our method, we also conduct a 2-round deployment-optimization loop experiment with 25 interaction trajectories every round. This experiments ensure that the total number of trajectories is 50, the same as the comparison experiment setting. We also evaluate the performance with 3 random seeds, and report the mean success rate and standard deviation in the table below.\n\n| Base Policy | Round 1 | Round 2 |\n|---|---|---|\n| $44.7 \\pm 1.2$ | $54.6 \\pm 6.1$ | $60 \\pm 4$ |\n\nAlthough results prove that our method could effectively learn from human-intervened data and achieve self-improvement continually, performance of multi-round deployment-optimization loops is only comparable to the single-round deployment-optimization loop in Table 1 of our paper ($60 \\pm 2$). \nA possible explanation is that the failure cases in the target task are highly similar across rounds, so additional deployment-optimization loops do not yield significant improvements over a single round. Thanks for this question to promote us rethink and improve the design of multi-round deployment-optimization loops in further work.\n\n---\n$\\color{green}Q3:$ Explanation of reweighting method. How to determine balance between being adaptive and keeping priors?\n\n$\\color{blue}A3:$ Thank you for this valuable question about the proposed adaptive reweighting method.\n\n### 1. Explaination of reweighting methods and problem of tokenization:\n\nFirst, we clarify the motivation behind our reweighting method. The core motivation of our action-level preference optimization approach is to encourage the model to learn correct behaviors while avoiding failure actions. Traditional preference optimization methods in LLMs focus only on token probabilities, ignoring the semantic meaning of the decoded words. For instance, if the ground-truth token is _50_, both _10_ and _49_ are treated as equally incorrect.\n\nHowever, in robot decision making with discrete action space, the continuous action space is encoded to discrete tokens, where each token corresponds to a specific physical action. For example, the action token $i$ represents 'move along the x-axis by $0.i$ cm'. Therefore, it is inappropriate to treat token _10_ and _49_ as equally incorrect. For instance, the desirable action token _50_ may represent \"move along the x-axis by 5 cm\", while token _49_ corresponds to \"move along the x-axis by 4.9 cm\", and token _10_ to \"move along the x-axis by 1 cm\". Clearly, action token _49_ is much closer to the desired action than _10_.\n\nOur reweighting method is designed to leverage this continuous action information for discrete action token preference optimization. This approach bridges the gap between discrete token prediction and continuous action regression, making preference optimization more effective for autoregressive VLA models.\n\n### 2. The balance between being adaptive and keeping priors:\n\nThanks for this insightful question, which allows for a deeper exploration of the mechanism behind our reweighting method.\n\nFirst, it is important to clarify that reweighting method does not contradict the KL divergence constraint. The adaptive coefficients $\\lambda_U$ and $\\lambda_D$ are sample-wise weights with batch-level normalization, and are estimated using the L1 loss between the ground-truth action $a_{gt}$ and the predicted action $a_{pred} = \\pi_{\\theta}(o_t)$. While the KL divergence term ensures that the updated policy $\\pi_{\\theta}$ stays close to the reference policy $\\pi_{ref}$. When $\\pi_{ref}$ is a good prior, the reweighting method would also encourage the model to preserve prior knowledge.\n\nIn fact, the sample-wise reweighting offers an effective trade-off between adaptability and retaining prior knowledge. When the reference model $\\pi_{ref}$ provides a good prior, the estimated $l_1$ loss is small, encouraging the model to stay close to the prior distribution. Conversely, if $\\pi_{ref}$ is not a suitable prior, the $l_1$ loss becomes larger, prompting the model to adapt more actively to the new distribution.\nIn this way, our method adaptively balances preserving prior knowledge and learning from new distributions as needed.\n\nHere, we provide an example below showing how $\\lambda$ values vary across samples based on their regression errors.\n\n|Type | L1 loss| $\\lambda$| \n|---|---|---|\n| Undesirable Action | [0.05, 0.25] | [0.83, 0.44] |\n| Desirable Action | [0.05, 0.01, 0.39, 0.0, 0.16, 0.15] | [0.06, 0.02, 0.40, 1.2e-4, 0.18, 0.18] |\n\nThe values of $\\lambda_U$ and $\\lambda_D$ indicate that higher weights are assigned to desirable samples with large regression errors and to undesirable samples similar to failure actions.\n\nTo assess whether our adaptive reweighting method is effective, we conducted experiments using fixed values: $\\lambda_U = 0.5$; $\\lambda_D = 1$ for human-intervened actions and 0.5 for autonomous actions to emphasize the human-intervened action. Results are shown below:\n\nCoffee_D0:\n|Method | Success Rate|\n|---|---|\n|HAPO w/o. reweighting | $51.3 \\pm 3.0$ |\n|HAPO | $60.0 \\pm 2$ |\n\n\nStackThree_D0\n|Method | Success Rate|\n|---|---|\n|HAPO w/o. reweighting | $45.3 \\pm 6.1$ |\n|HAPO | $53.3 \\pm 5$ |\n\nSquare_D0:\n|Method | Success Rate|\n|---|---|\n|HAPO w/o. reweighting | $30.0 \\pm 3.5$ |\n|HAPO | $31.3 \\pm 3.1$  |\n\nThreePiece_D0\n|Method | Success Rate|\n|---|---|\n|HAPO w/o. reweighting | $38.7 \\pm 2.3$ |\n|HAPO | $43.3 \\pm 5.0$ |\n\nThese results prove that adaptive, sample-wise reweighting could lead to significantly better performance.\n\n### 3. Analysis of the reward curves:\nFurther, to evaluate whether HAPO effectively promotes learning from human-intervened data via adaptive reweighting, we track the reward of KTO and HAPO during training. The reward is computed as $r_{\\theta}(o, a) = \\log \\frac{\\pi_\\theta(a|o)}{\\pi_{ref}(a|o)}$, where a higher reward for desirable actions indicates stronger model confidence, and a lower reward for undesirable actions suggests better avoidance.\n\nDesirable Reward\n\n| Method| Iter. 1000 | Iter. 2000 | Iter. 3000 |\n|---|---|---|---|\n| KTO | 11.0 | 11.8 | 15.7 |\n| HAPO | 10.2 | 12.9 | 17.4 |\n\nUndesirable Reward\n\n| Method| Iter. 1000 | Iter. 2000 | Iter. 3000 |\n|---|---|---|---|\n| KTO | -10.5 | -18.9 | -26.6 |\n| HAPO | -18.7 | -24.9 | -35.5 |\n\nThe reward curves demonstrate that HAPO effectively reweights training samples, encouraging the model to learn more valuable interaction data."}]

Author rebuttal:

Meta-review: The paper introduces Human-assisted Action Preference Optimization (HAPO) for continual policy optimization of Vision-Language-Action models. It initially had mixed reviews with reject but finally received leaning-positive reviews: 3 Borderline Accepts and 1 Accept. 

Reviewers appreciated the strong motivation, well-structured method, and extensive experiments across both simulation and real-world robot settings. Main concerns centered on baseline details, statistical significance, and the explanation of the adaptive reweighting mechanism. During rebuttal, the authors provided additional experiments, rigorous implementation details, reward curves, and ablation studies, which help address these issues. Overall, HAPO provides a meaningful step toward reliable real-world deployment of VLA models. Given the reviewers’ updated scores, I recommend Accept.
Decision: Accept (poster)