ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning
Authors: Hongyin Zhang
Zifeng Zhuang
Han Zhao
Pengxiang Ding
Hongchao Lu
Donglin Wang
Venue: ICML 2025 poster (2025)
Keywords: VLA Model
Reinforcement Learning
Robotic Manipulation
Primary area: applications->robotics
TL;DR: We introduce Reinforced Robot GPT (ReinboT), a novel end-to-end VLA model that integrates the RL principles of maximizing cumulative reward.

Abstract: Vision-Language-Action (VLA) models have shown great potential in general robotic decision-making tasks via imitation learning. However, the variable quality of training data often constrains the performance of these models. On the other hand, offline Reinforcement Learning (RL) excels at learning robust policy models from mixed-quality data. In this paper, we introduce Reinforced robot GPT (ReinboT), a novel end-to-end VLA model that integrates the RL principle of maximizing cumulative reward. ReinboT achieves a deeper understanding of the data quality distribution by predicting dense returns that capture the nuances of manipulation tasks. The dense return prediction capability enables the robot to generate more robust decision-making actions, oriented towards maximizing future benefits. Extensive experiments show that ReinboT achieves state-of-the-art performance on the CALVIN mixed-quality dataset and exhibits superior few-shot learning and out-of-distribution generalization capabilities in real-world tasks.

ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning
Hongyin Zhang 12
Zifeng Zhuang²
Han Zhao
Pengxiang Ding? Hongchao Lu Donglin Wang?
Abstract
An important reason that limits the performance of VLA models is that the quality of training data sources is usu- ally uneven, even if they come from successful demonstra- tions (Hejna et al.). Although recent imitation learning meth- ods can effectively replicate the distribution of demonstra- tions (Vuong al., 2023; Brohan et al., 2023; Zhang et al., 2025), they have difficulty distinguishing between uneven data quality and making full use of mixed-quality data (Bai et al., 2025). On the other side, offline Reinforcement Learn- ing (RL) algorithms aim to leverage previously collected data without the need for online data collection (Levine et al., 2020). Despite initial attempts to integrate VLA with RL (Mark al., 2024; Zhai et al., 2024; Zhao al., 2025a; Guo et al., 2025), the design of widely applicable dense rewards for visual- lisuua-lagguage manipulation tasks and the in- corporation of the RL concept of maximizing benefits into the VLA model remain underexplored.
Aision-Language-ee (VLA) models have shown great potential in general robotic decision- making tasks via imitation learning. However, the variable quality of training data often con- strains the performance of these models. On the other hand, offiin Reinforcement Learning (RL) excels at learning robust policy models from mixed-quality data. In this paper, we introduce Reinforced robot GPT (ReinboT), novel end- to-end VLA model that integrates the RL prin- ciple of maximizing cumulative reward. Rein- boT achieves deeper understanding of the data quality distribution by predicting dense returns that capture the nuances of manipulation tasks. The dense return prediction capability enables the robot to generate more robust decision-making actions, oriented towards maximizing future ben- efits. Extensive experiments show that Rein- boT achieves state-of-the-art performance on the CALVIN mixed-quality dataset and exhibits su- perior few-shot learning and out-of-distributiin generalization capabilities in real-world tasks.
To this end, we propose Reinforced robot GPT (ReinboT), novel end-to-end VLA model to implement the RL con- cept of maximizing dense returns. Specifically, we effi- ciently and automatically decompose the long-horizon ma- nipulation task trajectory into multiple trajectory segments containing only single sub-goal, and design a dense re- ward that captures the characteristics of the manipulation task. In fact, complex robot manipulation tasks need to consider many factors, such as tracking targets, reducing energy consumption, and maintaining flexible and stable be- havior. Therefore, the design principle of proposed reward densification method is based on these considerations and remains widely applicable to various manipulation tasks.
1. Introduction
Research on vision-language-accionn (VLA) models for gen- eral embodied intelligence in robotics has recently flour- ished (Brohan al., 2022; 2023). VLA models are usu- ally based on the imitation learning paradigm, where pre-trained vision-language model is post-trained on down- stream robotic data (Ding et al., 2024; Zhao et al., 2025b). While semantic generalization has improved in VL.A models through extensive robotic training data, a critical gap persists in their manipulation accuracy for downstream tasks (Bro- han et al., 2023; Black et al.; Li et al., a).
In terms of ReinboT algorithm design, we consider that accurate estimation of the value function in RL. algorithms has always been thorny problem, especially in the Trans- former architecture (Parisotto al., 2020; Davis al., 2021). Therefore, we utilize cumulative rewards (i.e., Re- turnToGo (Chen et al., 2021)) new modality data to characterize data quality characteristics based on the con- structed dense reward. Inspired by previous work (Zhuang et al., 2024), we model the maximum return sequence over the joint distribution of language commands, image states (and proprioception). actions, and ReturnToGo. This is a supervised paradigm that integrates the RL goal of predict-
'Zhejiang University, Hangzhou, China "Wsttlake University, Hangzhou, China. Correspondence to: Donglin Wang <wang- donglin wew.saaa..ed..c2......
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ReinboT:: Amplifying Robot Visual-Langune Manipulation with Reinforcement Learning
ing the maximum return within the distribution given the current conditions, and thereby considering the likelihood of maximizing actions. Specifically,, we utilize expectile regres- sion (Aigner et al., 1976; Sobotka & Kneib, 2012) to make the predicted return as close as possible to the maximum re- turn that can be achieved under the current goals and states. With this ability, ReinboT can predict the maximum return during inference to guide the execution of better actions. Overall, the core contributions of this paper include:
goal (Yang al.), or the distance the goal is calculated with the help of LLM models and other pre-trained visual models (Zhang et al., 2024). However, these reward de- signs face the credit assignment problem that has not been fully solved in RI. (Sutton, 1984), or are limited by the hallucination problem of LLM (Zhang et al., 2023).
In terms of combining with RL algorithms,, these works mainly fine-tune existing VLA models that have undergone imitation learning, including introducing Q-functions to cor- rect action distribution (Nakamoto et al.), screening out high-value action fine-tuning policies (Mark et al., 2024; Zhang et al., 2024), and fine-tuning according to human pref- erences (Chen et al., 2025). Moreover, a recent work (Cheb- otar et al., 2023) utilizes auto-regressive Q-functions to learn visual language manipulation, but the sequence length and inference time of their models increase significantly with the increase of action dimensions. Different from these studies, we aim to propose new end-to-end reinforced VLA model based on dense rewards that capture the characteristics of manipulation tasks.
We propose ReinboT, novel end-to- end VLA model that integrates RI returns maximization to enhance robotic manipulation capabilities.
We introduce a reward densification method that en- ables ReinboT to gain deep insights into data quality for more robust learning.
Extensive experiments demonstrate ReinboT's state-of- the-art performance, significantly outperforming base- lines in both simulated and real-world tasks.
2. Related Work
3. Preliminaries
2.1. Offline RL via Sequence Modeling
3.1. Imitation Learning of VLA Model
After the emergence of Transformer (Vaswani et al., 2017) as an efficient sequence modeling model, a large number of works (Chen et al., 2021; Yamagata et al., 2022; Janner et al., 2021; Zhuang et al., 2024: Shafiullah et al.. 2022: Hu al., 2024) have explored the application of sequence models as agent policies to RL decision tasks. The Decision Transformer (Chen et al., 2021) (DT) trains the context- conditional policy model on offline datasets through a su- pervised learning paradigm, conditioned on historical ob- servations and ReturnToGo, and outputs the actions that the policy model should perform. Reinformer (Zhuang et al., 2024) further introduces the concept of maximizing returns on the basis of DT. During training, Reinformer not only predicts actions conditioned on the ReturnTooGo in offline data, but also predicts the maximized ReturnToGo that the policy model may subsequently obtain under observation. Different from these task fask-specific studies, our work aims to implement the RL return maximization concept in a general- purpose VLA model to enhance the robot's long-horizon manipulation capabilities.
As a typical VLA model for imitation learning. GR-1 (Wu et al.) demonstrates that visual robot manipulation can significantly benefit from large-scale video generation pre- training. Thanks to its flexible design, GR-1 can be seam- lessly fine-tuned on robotic data after being pre-trained on large-scale video datasets. GR-1 is a GPT--ttyle model that takes language instructions 1, historical image observa- tions 0 and proprioception st- hut as input. It predicts robot actions and future images in an end-to-end manner (ōL+1, a.) = क (l, (o, s)t-
3.2. Max-Return Sequence Modeling
The sequence model DT (Chen et al., 2021) maximizes the likelihood of actions based on historical trajectories and ReturToGo, which essentially transforms offline RL into supervised sequence modeling:
L. = B2[ log mo (8, 9, a) Sc, 9:)) (1)
where -:= -- Di= is ground-truth ReturnToGo in offline data. Reinformer (Zhuang et al., 2024) integrates the goal of maximizing return into sequence models. Specifically, Reinformer predicts the maximum returns that the current state might obtain within the data distribution represented by the dataset, rather than the ground-truth ReturnToGa of the current trajectory. Reinformer achieves this implicitly through the minimizing of expectile regression loss:
2.2. VLA Model Integrating with RL
Recent work has initially combined VLA with RL to study how to further improve the manipulation accuracy and adapt- ability of VLA models while still retaining their best ad- vantages in scale and generalization. In these works, the source of the reward signal is either a sparse form of whether the goal is reached (Chebotar et al., 2023; Nakamoto et al.; Mark et al., 2024), or the number of steps to reach the
C, E, [lm 1(Ag < (A))),
(2)
with Ag gt To((s, g. a) st),
2

---

ReinboT:: Amplifying Robot Visual-Langun Manipulation with Reinforcement Learning
Reward Densification:
Sub-goal
Task
ReturnToGo
Robot Action
Future Image
Achievement
Progress
RTG Decoder
Action Decoder
Image Decoder
Behavior Smoothness
Task Completion
ReinboT
Frozen ViT &
Frozen CLIP Text Encoder
Perceiver Resampler
MLP
[RTG]
[ACTION]
[IMAGE]
Grab the liger toy.
Language Instruction Image State
Robot Proprioception
Figure 1. The proposed ReinboTT model. We leverage CLIP (Radford et al., 2021) to encode robot language instructions, utilize ViT (Dosovitskiy et al., 2020; He et al., 2022) (and perceiver resampler (Jaegle et al., 2021)) to compress and encode the original pixel space of the image state, and utilize MLP encode the robot proprioception. Moreover, based the GPT-sttyle transformer (Radford. 2018), we introduce three prediction token embeddings ([TG)) [ACTION] and [IMAGE]) to predict ReturnToGo, robot action, and future image respectively. The last layer of hidden features in ReturnToGo decoder further utilized predict robot actions. The dense reward in ReturnToGo contains four aspects: sub-goal achievement, task progress, behavior smoothness and task completion.
where I(-) is binary indicator function, and m E (0, is the hyperparameter of expectile regression. An exces- sively large parameter m may cause the model to over- optimistically estimate the maximum possible ReturnToGo in the training data distribution. This will cause the pre- diction of ReturnToGo outside the distribution during inference, which will negatively affect action generation. Reinformer was trained by minimizing the sum of two loss functions L = f.. + Lg. Compared with DT, one advantage of the Reinformer is that it does not need to specify the initial value of Return ToGo and the reward returned by the environment during inference. The Reinformer can autore- gressively predict the maximum ReturnToGo and action of the next step through two model inferences:
4.1. Reward Densification
For long- oong-horizon visual-language manipulation tasks, VLA models are usually required to maintain robust and stable behavior at minimal energy cost while following the goal. Therefore, we mainly design a widely applicable dense reward around this principle to capture the nature of the manipulation task. Intuitively, in the robot trajectory, the reward that minimizes the state distance is a simple and effective scheme that encourages the robot to move directly to the target state. However, this reward is limited to the case when the task contains only one goal. For long-horizon tasks that require manipulating objects with multiple sub- goals, this reward will guide the robot to move directly to the final target state, resulting in failure (Zhao et al., 2024).
g: π ((s,,,,a)--itet
Therefore, first adopt heuristic method (James & Davi- 2022; Shridhar et al., 2023) divide the long-horizon manipulation task into multiple sub-goal sequences and de- sign dense reward for each sequence. The heuristic process iterates over the states each demonstrated trajectory and determines whether the state should be considered a criti- cal state. The judgment is based on two main constraints: joint velocities close zero and changes gripper state. Intuitively, this occurs when the robot reaches pre-grasp pose or transitions to a new task phase, when grasping or releasing an object. Therefore, utilizing the critical state as a sub-goal is a natural and reasonable choice.
(3)
4. Methodology
In this paper, we aim to build a novel end-to-end VLA model that incorporates the principle of maximizing dense returns into robot visuomotor control, as shown Fig. 1. First, we consider four main factors when designing dense rewards to capture the nature of the robot's long-horizer manipulation task (Sec. 4.1). Then in Sec. 4.2, we elaborate how to build a novel end-to-end reinforced VLA model and test execution pipeline. Finally, we discuss and analyze how the proposed R organically integrates the principle of RL maximizing return (Sec. 4.3).
Sub-goal achievement. Both the image state or and propri- oception s, contain rich environmental perception informa- tion. Therefore, the sub-goal achievement reward ri covers
3

---

ReinboT:: Amplifying Robot Visual-Languen Manipulation with Reinforcement Learning
proprioceptive tracking, pixel intensity, image visual quality, and image feature points:
magnitude between demonstration trajectories. By utilizing the designed reward signal, ReinboTT can have broader and deeper understanding and recognition of the quality distribution of training data, thereby guiding the robot to perform more robust and stable robot decision actions.
11 (nn((t))))) + e (0).0)))
(4)
(fsuul(o))))) + efomn(or,))).
We utilize Mean Square Error (MSE) to calculate the direct difference between the image state on (and proprioception 8() and the sub-goal image state of (and sub-goal propri- oception s)), and utilize the Structural Similarity Index (SSIM) to measure the visual quality of the image. The Oriented FAST and Rotated BRIEF (ORB) (Rublee et al., 2011) algorithm utilized to calculate the reward focuses on the extraction and matching of image feature points. Specif- ically, we first detect the key points on the current image state and the sub-goal image state, perform feature match- ing and matching point screening, and finally calculate the similarity by the number of matching points.
4.2. End-to-end Reinforced VLA model
Through the proposed dense reward, we can obtain Re- turnToGoo (RTG) 9t = Σー= ry for long-horizon visual- language manipulation tasks. We further explain how to build a novel end-to-end reinforced VLA model to imple- ment the RL principle of maximizing return. The proposed ReinboT model utilizes GPT--ttyle transformer (Radford, 2018) 8) as the backbone network Tæ because it can flexibly and efficiently use different types of modal data as input and output. The CLIP (Radford et al., 2021) is utilized to encode language instructions, ViT (Dosovitskiy et al., 2020; He et al., 2022) (and perceiver resampler (Jaegle et al., 2021)) is utilized to compress and encode image states, and MLP is utilized to encode proprioception. We introduce action and image token embeddings ([ACTION] and [IMAGE]) and predict robot actions and future image states through an action decoder Pa and an image decoder Pv. respectively. Most importantly, we treat ReturnToGo as a novel modality of data and learn Return ToGo prediction token embedding [RTG]. We predict the maximized return given the language instruction I, image state o, and proprioception through the ReturnToGo decoder Ps:
Task progress. Considering that the impact of being divided into several sub-goal sequences on the overall trajectory is different. The later sequences are closer to the final target state. To reflect this, task progress reward 12 is designed:
m(se) K( HI'
r2
where n((+)) = [1, ... sequence the state s is in. The closer the sub-goal sequence is to the final goal state, the greater the task progress reward.
M(* }I} indicates which sub-goal
Behavior smoothness. To promote a smooth and natural movement trajectory. we mainly consider suppressing the joint velocity q and acceleration q of the robot arm move- ment and the rate of change of the action a,, thus punishing the trajectory movement that is too violent and stiff. The behavior smoothness reward r3 is:
LITG = B. [Im 1(4g < 0){(Ag)
(9)
with Ag = 9: - P. [To 0) 4)].
The loss function L of the ReinboT model comprises the Re- turnToGo loss LHTG, the arm action loss Carm. the gripper action loss Lgripper, and the future image loss Limage:
r3- = - -4| -1( -laa-aaa---1 aa-1 -a,l 2 -laaa aa-2- 22t--22222200 + 2 (6)
L = XLTrG + Carm + 0.01 Lgripper + 0.1 (10)
where is the ReturTTooo loss weight, and the loss weights for other modalities follow previous work (Li et al., 2025).
Task completion. For the visual language manipulation task, language instruction is regarded as goal that matches
Carm is Smooth-L, loss, Egripper is a cross entropy loss, and Limage is pixel-level MSE.
the robot's behavior. The task completion reward na is:
1 is successful).
When designing how to predict action a with feature infor- mation containing ReturnToGo, we make modular designs in the ReinboT network structure. Specifically, we first input the language instruction I, image state ot-u+1:t and proprioception 8t-u+1:; into the backbone network TG. and obtain the features I RTG - and ( corresponding to [RTG] and [ACTION] token embeddings:
Here, 1(-) is a binary indicator function used to indicate whether trajectory completes the instruction.
Based on these four main factors, the general dense reward captures the nature of the long-horizon visual-language ma- nipulation tasks is:
paction h -
Te (1, Otu,o, 81 St (11)
=
(8)
The feature 1 1 is then input into the ReturnToGo de- coder Pe obtain the last layer of hidden features ahidden
where {w, }== is the reward weight, which can ensure that each reward component maintains comparable order of
hideen = Pp RTG
(12)
4

---

ReinboT: Amplifying Robot Visual-Langus Manipulation with Reinforcement Learning
The hidden features glidden is concatenated with the ac- tion features h and are further input into the action decoder Pw predict the action -1:
However, blindly increasing m will lead the model's overly optimistic estimate of the maximum return that can be achieved in the training data distribution. Related theo- retical analysis is in Reinformer (Zhuang et al., 2024).
d = Pw (h Stst+k-1)
(13)
In the classic RL algorithm, maximizing the Q-value is uti- lized to achieve the best policy model. This implies that applying RL in VLA necessitates the introduction of an additional RL loss function. Such an addition may pose obstacles to the learning process of models like transform- ers (Mishra et al., 201 8; Parisotto et al., 2020; Davis et al., 2021). In contrast, our return condition maximization cir- cumvents the need to incorporate the RL-specific loss.
The modular design in ReinboT allows to obtain robot ac- tions with only single model inference, thus enjoying higher inference efficiency than the Reinformer model. The more benefit of this design is that during the inference phase, we do not need to manually set the initial value of ReturnToGo like the DT model. This is crucial for actual deployment because it greatly alleviates the tediousness of manual pa- rameter adjustment, and the actual deployment environment cannot directly obtain rewards to a large extent. The Rein- boT inference pipeline has been summarized in Alg. 1. The implementation details are in Appendix Sec. A.
5. Experiments
In this section, we explore how the proposed ReinboT model can effectively implement the RL principle of maximiz- ing return to enhance robotic vision-language manipulation tasks. To this end, our experiments aim to investigate the following questions: 1) Does ReinboT show better gener- alization ability and higher success rate when performing long-horizon tasks compared to baseline algorithms? 2) How important is the dense reward component the overall generalization performance of ReinboT? 3) What the characteristics of maximizing return prediction in ReinboT? 4) Can ReinboT complete few-shot learning and out-of- distribution (OOD) generalization in real-world scenarios?
Algorithm 1 ReinboT:: Test-time Execution
1: ReinboTT model a.. Pş. Pu. initial image state Op,tes initial proprioception to,test, language instruction lest and environment Env. // Return ToGo initialization value is not required.
2: 4-0 0
3: while & S Trest do
4: Calculate ReturnToGo and action features by Eq. 11
5: Calculate ReturnToGo last layer of hidden features
by Eq. 12
6: Calculate robot action 0 by Eq. 13 Interact with Env 01+1,et St+1nes + Env. Env.Step(â:)
7:
5.1. Generalization Evaluation on Mixed-quality Data
// Reward is not required.
Setting. We first construct a mixed-quality dataset based on CALVIN (Mees et al., 2022), which contains long-horizon manipulation tasks, to examine the performance of the pro- posed ReinboT and baseline algorithms. This dataset con- tains small amount of data with language instructions in CALVIN ABC (about 50 trajectories per task) and large amount of autonomous data without language instructions. In addition to the original data collected by human teleop- eration without language instructions in CALVIN (more than 20,000 trajectorics), the autonomous data also contains failure data generated by the interaction between the trained VLA behavioral policy RoboFlamingo (Li et al., b) and the environment CALVIN D (more than 10,000 trajectories). To promote data diversity, different degrees of Gaussian noise (0.05, 0.1, and 0.15) are added to the actions of the RoboFlamingo policy model during the interaction. We study training on this mixed-quality data, then fine-tune a small amount of data with language instructions, and finally test the generalization performance on CALVIN D. The sub- goal division and dense reward examples of mixed-quality training data are in Appendix Fig. 8 ~ 11. Tab. shows the success rate of each language instruction in the chain and the Average Length (AL) of the completed tasks.
8: 1+1+1 1+1
9: end while
4.3. Discussion and Analysis of ReinboT
Compared with common and-to-end VLA models, the most significant feature of proposed Reinbo'T is the additional introduction of Return loss (Eq. 9), and the action is affected by the ReturnToGo token ghidden (Eq. 13). We will subsequently analyze how this framework achieves RL return maximization, well the differences and advantages compared to return maximization in classic RL.
Take a date point (1, (o,s), 9t. 9t. oz) for example. The loss of Roto is implemented based on expectile re- gression, with a key parameter m. When m = 0.5, the expectile regression degenerates into MSE, and the pre- dicted ReturnToGo token 9, approaches the ground-truth St- At this point, ReinboT degenerates into the paradigm of imitation learning, also the common end-to-end VI.A models equipped with ReturnToGo token prediction. When m > 0.5, the expectile regression will predict the 9, greater than gt. which is called return maximization. This maxi- mized return guides the ReinboT to predict a better action.
5

---

ReinboT:: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning
Table Generalization performance comparison of models trained on CALVIN mixed-quality data test environment
No. of Instructions Chained
Algorithms
Avg. Length (t)
RoboFlamingo (annotated data)
0.55
0.19
0.07
0.02
0.00
0.83
GR-1 (annotated data)
0.67
0.37
0.20
0.11
0.07
1.41
PIDM (annotated data)
0.60
0.45
0.32
0.23
0.13
1.73
GR-1
0.62
0.31
0.18
0.14
0.10
1.36
GR-MG
0.65
0.35
0.24
0.11
0.05
1.41
RWR (sparse)
0.63
0.36
0.21
0.12
0.07
1.38
RWR (sub-goal, sparse)
0.71
0.46
0.27
0.19
0.11
1.73
RWR (dense, single)
0.75
0.52
0.27
0.18
0.11
1.82
ReinboT (sparse)
0.70
0.44
0.29
0.19
0.12
1.74
ReinboTT (sub-goal, sparse)
0.74
0.50
0.28
0.17
0.12
1.80
ReinboT (dense, single)
0.77
0.53
0.32
0.18
0.11
1.90
ReinboT (dense, full)
0.79
0.58
0.40
0.28
0.21
2.26
Table 2. Ablation experiments are conducted verify the necessity of designed reward components.
No. of Instructions Chained
Avg. Length (1)
3
4
5
ReinboT (dense, full)
0.79
0.58
0.40
0.28
0.21
2.26
W/o ReturnToGo
0.65
0.36
0.19
0.11
0.06
1.36 (-39.8%)
W/o sub-goal achievement r
0.72
0.50
0.32
0.20
0.12
1.87 (-17.2%)
W/o task progress r22
0.75
0.48
0.29
0.17
0.10
1.79 (-20.8%)
W/o behavior smoothness
0.73
0.50
0.32
0.21
0.14
1.90 (-15.9%)
W/o task completion
0.75
0.50
0.33
0.21
0.14
1.93 (-14.6%)
Baselines. To thoroughly evaluate the effectiveness of the proposed ReinboT model, we consider some representative baseline algorithms and reward design methods, including RoboFlamingo (Li et al., b), GR-1 (Wu et al.). PIDM (Tian et al., 2024) (three imitation learning types), GR-MG (Li et al., 2025) (hierarchical imitation learning type), and RWR (Peters & Schaal, 2007) (offline RL type). "anno- tated data" means that the model trained only on small amount of data with text annotations (about 50 trajectories per task). "sparse" means utilizing sparse rewards, that is, the reward of the last three steps of a successful trajectory is 1, and the rest is 0 (Nakamoto et al.). "sub-goal, sparse" means utilizing sparse rewards, that is, the reward of the last three steps of a successful trajectory and the sub-goal state 1, and the rest "dense, single" means utilizing the dense reward we proposed, and the final calculated single- dimensional scalar return is utilized when calculating the ReturnToGo loss. "dense, full" means utilizing the dense reward we proposed, and the predicted Return ToGo is a vector containing the calculated single-dimensional scalar return and each return component. The baseline details are introduced in Appendix Sec. A.2. Generalization performance comparison. Tab. shows that among the models trained only data with text notations, PIDM integrates vision and action into closed
loop and achieves better generalization performance than the baselines RoboFlamingo and GR-1. However, limited to the imitation learning type, the performance of PIDM (AL is 1.73) is lower than the offline RL type model RWR and ReinboT (AL is 1.82 and 2.26, respectively). Moreover, the performance of GR-1 (AL is 1.36), which can obtain all the training data information, is slightly lower than the performance of GR-1 trained only on training data with text annotations (AL is 1.41). GR-MG is also limited by the low-level policy based imitation learning. Therefore, the VLA model under the imitation learning paradigm only performs maximum likelihood on the original training data distribution, which is difficult to capture and fully utilize the characteristics of the mixed quality distribution, resulting in unsatisfactory performance.
For ReinboT and RWR, our dense reward improves perfor- mance better than sparse rewards. Predicting each compo- nent of ReturnToGo can further improve the generalization ability of ReinboT (AL increased from 1.90 to 2.26). There- fore, the proposed reward has deeper and more detailed representation of the data quality distribution, thus bring- ing denser supervision signals to the training of the VLA model. The ReinboT can effectively implement the idea of RI. to leverage dense return maximization enhance long- long-horizon visual- visual-language manipulation tasks.
6

---

ReinboT: Amplifying Robot Visual- Manipulation with Reinforcement Learning
(a)
0.01
0.005
-
0.001
0.0005
-K-
Average Achieved Task Langth
Frobabiity Densityy
0.8
80
Training Data
2.0
0.6
Teeling El Param, m=0.5
Success Rate 1%)
60
1.5
0.4
Testing Param. m=0
lesting ER Penam. m=0
Testing ER Faram. m-0.99
40
1.0
0.2
0.0
20
0.5
RetumbbGG Value
10
(b)
Task
Task
Avg.
0.5
0.7
0.9
0.99
Average Achieved TasS Length
Retum
Trining ann.
lesting ER Faram. m=0.5
80
Testing ER Param. m-0,7
2.0
4-
lesting ER Param. m=0
Success Rate (06)
Testing ER Param, 99
1.5
10
20
30
40 50
60
40-
1.0
Step
0.5
20-
Figure 3. (a) Distribution of ground-truth ReturnToGo of CALVIN mixed-quality training data and distribution of the maximized Return predicted by the ReinboT when interacting with the test enviromment D. (b) Comparison of ReturTTooo in the training data with text annotations in mixed-quality data and the maximized Return predicted by the ReinboT at the interaction time step. The impact of different values of the Expectile Regression (ER) parameter yn the CRTG loss function is investigaled.
Task
Task
Task
Task
Task
Avg. Len.
Figure 2. (a) Impact of different values of ReturnToGo LRTG loss
weight A. (b) Impact of different values of the expectile regression
parameter m the ReturnToGo LRTG loss function.
5.2. Ablation Study
Ablation of dense reward component. We conduct abla- tion experiments on ReinboT trained on CALVIN mixed- quality data and tested on environment D to evaluate the contribution of each reward component to the model gen- eralization (Tab. 2). If there is no ReturnToGo modal in- formation in ReinboT, the metric AL will drop sharply by 39.8%. If there is a lack of reward component in Rein- boT, its performance will be lost to varying degrees. The most significant impact is task progress r'2 (reduced by 20.8%), followed by sub-goal achievement 71 (reduced by 17.2%). The reward components behavior smoothness r3 and task completion Fa have similar impacts, decreasing by 15.9% and 14.6% respectively. Therefore, each reward component can help the model to deeply identify various aspects of data quality and has a significant impact on the robot's generalization performance.
Properties of the predicted maximized RL return. To an- alyze the underlying reasons for the performance improve- ment of the proposed RinboGPT model, we explore the properties of the predicted maximized RL return (Fig. 3). The sample size of the training data is in Appendix Fig. 7. The results show that the expectile regression parameter m increases, the ReturnToGGo distribution shifts towards a larger value. Therefore, ReinboT can effectively identify and distinguish the quality distribution of training data and predict the robot action that maximizes return in the cur- rent (and historical) state as much as possible. This means that when the robot performs an action, it will consider maximizing the long-horizon benefits in the future period, rather than only considering the current (and historical) state in the short term. This capability can effectively enhance the generalization performance of the Reinbo'T' model in long- horiznorizon manipulation tasks.
Performance impact of hyperparameters À and m. We further conduct ablation experiments on À and m intro- duced in ReinboT, trained on CALVIN mixed-quality data and tested on environment D, to explore their impact on model performance (Fig. 2). The hyperparameter À is uti- lized to make a trade-off between the model's 's prediction of ReturnToGo and other modalities. The expectile regression parameter is utilized to control the model's sensitivity to different expectation levels, thereby adjusting the model's fitting characteristics for the ReturnToGo distribution. Ex- perimental results show that when À = 0.001 and m = 0.9, ReinboT achieves the best performance, which is the default setting for our model unless otherwise specified.
Moreover, we can find that the model performance with parameter m 0.99 is lower than that with m 0.9 (Fig. 2(b)), although the former has higher ReturnTooo tendency (Fig. 3). This indicates that too large will cause ReinboT to overestimate the maximum ReturnToGo that can be achieved in the training data distribution. The ReinboT has difficulty responding to the predicted over-estimated Return ToGo, resulting in performance degradation.
7

---

ReinboT:: Amplifying Robot Visual-Langun Manipulation with Reinforcement Learning
Few-shot Learning
Out-of-distribttion Generalization
Unseen
Instruction
(a)
(d)
(
@
Figure Few-shot learning and OOD generalization evaluation scenarios real-world tasks. Few-shot learning evaluation scenarios include cup grasping (a), bowl grasping and placing (b), and plush toy grasping and placing (c). OOD generalization evaluation scenarios include unseon language instructions (d), desktop backgrounds (c), distractors (f), and manipulated objects (g).
ReinboT (Ours)
GM-1
Success Rote 6
50
Sample
Cup Graspiigg
Bewl Grasping and Placing
Plush Tay Grasping and Placing
Average
(b)
ReinboT (Ours)
RWR
GR:1
25
50
75
Succcess Rate (9)
Return ToGo Value
50
25
Figure 5. Distribution of successful realistic trajectories.
Simple
Unseen
Unseen Backgrounds
Unssen Distractors
Unseen Objects
Unseen Average
Instructions
5.3. Evaluation Real-world Tasks
Figure 6. (a) Comparison of few-shot learning performance. (b)
Settings. We conduct evaluations on real-world tasks amine whether the proposed ReinboT can perform effective few-shot learning and generalization in realistic scenarios. Specifically, we consider the picking and placing tasks of ob- jects such as cups, bowls, and stuffed toys on a robotic arm URS. The total number of successful trajectories collected approximately 530 (data distribution in Fig. 5), and the model first trained on these data. The sub-goal division and dense reward examples of successful training data real-world URS are in Appendix Fig. 12 ~ 15. For few- shot learning evaluation, we consider three object grasping and placing tasks (Fig. 4(a-c)). Each task contains only 30 successful trajectories, and the model is fine-tuned on these three tasks, For OOD generalization evaluation, we consider scenes with unseen instructions, backgrounds, distractors and manipulated objects (Fig. 4(d-g)).
Generalization comparison on simple and unseen tasks.
shot learning and OOD generalization performance in real- istic scenarios, and significantly outperforms the baseline methods. This is due to ReinboT's ability to effectively consider maximizing future returns. RWR performs on par with the GR-1 This may be due the overfitting of RWR to the training data and its reliance reweighting the data, which may lead to optimization problems when the data distribution is uneven or the amount of data is insufficient.
6. Conclusion
This paper internalizes the principle of maximizing return in RL into the VI.A framework, thereby enhancing the robot's long-horizon manipulation capabilities. The proposed Rein- boT can predict the maximum dense return that depicts important information of the manipulation task, thus hav- ing deep and detailed understanding of data quality. This ability allows robot to consider not only the current (and historical) state, but also the future dense benefits when taking decision actions. Compared with baselines, Rein- boT achieves excellent performance in both simulated and real-world visual-language manipulation tasks. Our work advances robot visual- langual manipulation capabilities, contributing to embodied general intelligence. promising work is to consider the scaling of models and data to cope with the rich and diverse robotic tasks the real world.
ReturnToGo distribution of successful trajectories. Fig. 5 shows the ReturnToGo distribution of successful tra- jectories in reality. The result shows that even if the training data are all successful trajectories, their quality distribution still uneven under the dense reward metric we proposed. Therefore, is necessary introduce RL ideas into the VLA model deeply identify the data distribution and guide the prediction of actions that maximize data quality.
Real machine comparison. The quantitative performance comparison of real-world tasks is in Fig. 6, and the qual- itative results are in Appendix Fig. 16. The experimental results show that the proposed ReinboT has excellent few-
8

---

ReinboT: Amplifying Robot Visual-Langunn Manipulation with Reinforcement Learning
Acknowledgments
Chebotar, Y., Vuong, Q.. Hausman, K., Xia, F,, Lu, Y., Irpan, A., Kumar, A., Yu, TT,, Herzog, A., Pertsch, K., al. Q-transformer: Scalable offline reinforcement learning via autoregressive q- g-functions. In Conference on Robot Learning, pp. 3909-3928. PMLR, 2023.
This work was supported by the National Science and Technology Innovation 2030 Major Project (Grant No. 20222D020880),, and NSFC General Program (Grant No. 62176215).
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P.. Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084-1509999 2021.
Impact Statement
This research introduces Reinforced robot GPT (ReinboT), a novel Vision-Language-Acttonn model that integrates re- inforcement learning principles to address the limitations of variable training data quality in robotic decision- making tasks. By predicting dense returns that capture the nuances of manipulation tasks, ReinboTT' achieves deeper under- standing of data quality distribution and generates more robust decision-making actions aimed at maximizing future benefits. The model demonstrates state-of-the-ar perfor- mance on the CALVIN mixed-qualit, dataset and superior few-shot learning and out-o--disti generalization ca- pabilities in real-world tasks. This advancement has the potential to significantly enhance the adaptability and effi- ciency of robotic systems in complex. real-world environ- ments, paving the way for more reliable and versatile robotic applications across various industries.
Chen, Y., Jha, D. K., Tomizuka, M., and Romeres, D. Fdpp: Fine-tune diffusion policy with human preference. arXiv preprint arXiv:2501.08253 2025.
Davis, J. Q., Gu, A., Choromanski, K., Dao, T., Re, C., Finn, C., and Liang. P. Catformer: Designing stable transform- ers via sensitivity analysis. In International Conference on Machine Learning, pp. 2489-2499, PMLR, 2021.
Ding, P., Zhao, H., Zhang, W., Song, W., Zhang, M., Huang, S., Yang, N., and Wang, D. Quar-vla: Vision-language- action model for quadruped robots. In European Confer- ence on Computer Vision, pp. 352-367. Springer, 2024.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations,
References
Aigner, D. J., Amemiya, T., and Poirier, D. J. On the estimation of production frontiers: maximum likelihood estimation of the parameters of a discontinuous density function. International economic review, pp. 377-396, 1976.
2020.
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Bai, S., Zhou, W., Ding, P., Zhao, W., Wang, D., and Chen, B. Rethinking latent representations in behavior cloning: An information bottleneck approach for robot manipula- tion. arXiv preprint arXiv:2502.02883. 2025.
Guo, Y., Zhang. J., Chen, X., Ji, X., Wang, Y.-.,, Hu, Y.. and Chen, J. Improving vision-language-e acction model with online reinforcement learning. arXiv preprint arXiv:2501. 16664, 2025.
Black, K., Nakamoto, M., Atreya, P., Walke, H. R., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manip- ulation with pre-trained image-editing diffusion models. In The Twelfth International Conference on Learning Representations.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000-16009, 2022.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishhan, K., Hausman, K., Herzog, A.,
Hejna, J., Bhateja, C. A., Jiang, Y., Pertsch, K., and Sadigh, D. Remix: Optimizing data mixtures for large scale imitation learning. In 8th Annual Conference on Robot Learning.
Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023.
Hu, S., Shen, L., Zhang, Y., Chen, Y., and Tao, D. On trans- forming reinforcement learning with transformers: The development trajectory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.

---

ReinboT:: Amplifying Robot Visual-Langue Manipulation with Reinforcement Learning
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, Perceiver: General perception with it- erative attention. In International conference on machine learning, pp. 4651-4666 1664. PMLR, 2021.
Parisotto, E., Song, F., Rae, J., Pascanu, R., Gulcehre, C., Jayakumar, S., Jaderberg. M., Kaufman, R. L.., Clark, A., Noury, S., et al. Stabilizing transformers for reinforce- ment learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
James, S. and Davison, A. J. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2):1612-161) 111199 2022.
Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space con- trol. In Proceedings of the 24th International Confer- ence on Machine Learning, ICML '07, Pp. 745-750, New York, NY, USA, 2007. Association for Comput- ing Machinery. ISBN 9781595937933. doi: 10. 1145/1273496.12735999 URL https://doi.org 10. 10.1145/1273496. 127335090
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. Ad- vances Neural Information Processing Systems, 2021.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Radford, A. Improving language understanding by genera- tive pre- training. 2018.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, (8):9, 2019.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint ar arXiv:2005.01643 2020.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Li, P., Wu, H., Huang, Y., Cheang, C., Wang. L., and Kong, T. Gr-mg: Leveraging partially-annotated data via multi- modal goal-conditioned policy. IEEE Robotics and Au- tomation Letters, 2025.
Li, X., Hsu, K., Gu, J., Mees, O., Pertsch, K., Walke, H. R., Fu, C.,, Lumawat, I., Sich, I., Kirmani, S., et al. Evaluating real-world robot manipulation policies in simulation. In 8th Annual Conference on Robot Learning, a.
Rublee, E., Rabaud, V., Konolige, K., and Bradski, G. Orb:
An efficient alternative sift surf. In 201 Interna- tional conference on computer vision, pp. 2564-2571. Ieee, 2011.
Li, X., Liu, M., Zhang. H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., et al. Vision-language foundation models effective robot imitators. In The Twelfth International Conference on Learning Represen- tations. b.
Shafiullah, N. M., Cui, Z., Altanzaya, A. A., and Pinto, L. Behavior transformers: Cloning & modes with one stone. Advances in neural information processing systems, 35: 22955-22968, 2022.
Shridhar, M., Manuelli, L., and Fox, D. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning. pp. 785-799. PMLR, 2023.
Mark, M. S., Gao, T., Sampaio, G. G., Srirama, M. K., Sharma, A., Finn, C., and Kumar, A. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint (tt:::2212.066668 2024.
Sobotka, F. and Kneib, T. Geoadditive expectile regression. Computational Statistics & Data Analysis, 56(4):755- 767, 2012.
Mees, O., Hermann, L., Rosete-Beas, E., and Burgard, W. Calvin: A benchmark for language-conditionee policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 733)7327773344, 2022.
Sohn, K., Lee, H., and Yan, X. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018
Sutton, R. S. Temporal credit assignment in reinforcement learning. University of Massachusetts Amherst, 1984.
Nakamoto, M., Mees, O., Kumar, A., and Levine, S. Steer- ing your generalists: Improving robotic foundation mod- els via value guidance. In 8th Annual Conference on Robot Learning.
Tian, Y., Yang. S., Zeng. J., Wang, P., Lin, D., Dong, H., and Pang, Predictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint arXiv:2412. 15 2024.
10

---

ReinboT:: Amplifying Robot Visual-Langun Manipulation with Reinforcement Learning
Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Neural Informa- tion Processing Systems, 2017. URL https://api. semant i wwcchlac.r......oo arr.l.rrrggg....oo......... wwt.aaa...r..oooooooa/aea 13756489.
Zhao, H., Zhang, M., Zhao, W., Ding, P., Huang, S., and Wang, D. Cobra: Extending mamba to multi-modal large language model for efficient inference. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 39, pp. 10421-10429, 2025b.
Vuong, Q., Levine, S., Walke, H. R., Pertsch, K., Singh, A., Doshi, R., Xu, C., Luo, J., Tan, L., Shah, D., et al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 2023.
Zhao, T. Z.,, Kumar, V., Levine, S., and Finn, C. Learning fine-grained bimanual manipulation with low-cost hard- ware. In ICML Workshop on New Frontiers in Learning. Control, and Dynamical Systems.
Zhao, W., Chen, J., Meng, Z., Mao, D., Song, R., and Zhang, W. VImpe: Vision-language model predictive control for robotic manipulation. arXiv preprint arXiv22407.09829, 2024.
Wu, H., Jing, Y., Cheang, C.,, Chen, G., Xu, J., Li, X., Liu, M., Li, H., and Kong, T. Unleashing large-scale video generative pre-training for visual robot manipulation. In The Twelfth International Conference on Learning Repre- sentations.
Zhuang, Z., Peng, D., Liu, J., Zhang, Z., and Wang, D. Reinformer: Max-return sequence modeling for offline rl. In International Conference on Machine Learning, pP- 62707-62722. PMLR, 2024.
Yamagata, T., Khalil, A., and Cantos-Rodríguez, R. Q- learning decision transformer: Leveraging dynamic pro- gramming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, 2022. URL https: http:/app...sst.aeeeeeeeesas org/ :.:orpuSID:222188000 sww.nt iisaannlc.cchc.......
Yang, S., Du, Y., Ghasemipour, S. K. S., Tompson, J., Kael- bling, L. P., Schuurmans, D., and Abbeel, P. Learning interactive real-world simulators. In The Twelfth Interna- tional Conference on Learning Representations.
Zhai, S. Bai, H., Lin, Z., Pan, J., Tong, P., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., et al. tine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances neural information processing systems, 37:1 109311095--19977 2024.
Zhang, H., Ding, P., Lyu, S., Peng, Y., and Wang, D. Gevrm: Goal-expressive video generation model for robust visual manipulation. In The Thirteenth International Conference on Learning Representations, 2025.
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. Siren's 's song in the ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.012. 002999 2023.
Zhang, Z., Zheng, K., Chen, Z., Jang, J., Li, Y., Wang, C., Ding, M., Fox, D., and Yao, H. Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:2411.191 2024.
Zhao, H.. Song, W., Wang, D., Tong, X., Ding, P., Cheng, X., and Ge, Z. More: Unlocking scalability in reinforcement learning for quadruped vision-lamm aaroo--lngaag--ctto models. arXiv preprint 2503. 08000, 2025a.

---

ReinboT: Amplifying Robot Visual-Languana Manipulation with Reinforcement Learning
A. Appendix
A.1. Inplementation Details
Network structure and training process, The network backbone of ReinboT adopts the GPT2 (Radford et al., 2019) structure, and the ReturnToGo decoder and image decoder adopt the Transformer structure. In terms of the action decoder of ReinboT, we follow previous work (Zhao al.) and predict the robot action trajectory through Conditional Variational AutoEncoder (CVAE) (Sohn et al., 2015; Kingma, 2013). Specifically, the eVAE encoder is utilized encode the action trajectory into style vector embedding. The style vector embedding, the output embedding of the [ACTION] token, and the k learnable token embeddings are concatenated together. Then we input them into the Transformer to predict the k-step action trajectory. The network design hyperparameters of ReinboTT are shown in Tab. 3.
Table Network hyperparameters configuration.
Parameter
Value
Number of action prediction horizon
5 for CALVIN; 64 for UR5
Number of image (proprioception) history stack
10
Latent dimension of action encoder
32
Hidden layer dimension of action encoder/decoder
128
Hidden layer dimension of RTG decoder
128
Visual feature dimension
768
Language feature dimension
768
Embedding dimension
384
Number of layers in backbone
12
Number of attention heads
12
Activation function
ReLU
To train the ReinboT model more efficiently, we initialize its weights with the pre-trained model weights, which are derived from the generated video pre-training on the Ego4d (Grauman al., 2022) dataset consistent with GR-1 The ReinboT model allows the use of training data without language instructions. In the specific implementation, we provide empty string as the language instruction input of the model. The training hyperparameters are shown in Tab. 4.
Table 4. Training hyperparameters configuration.
Parameter
Value
ReturnToGo loss weight À
0.001
Expectile regression parameter m
0.9
Gradient clip
1.0
Epochs
50
Warm-up epochs
1
Batch size
32
Learning rate
0.001
Weight decay
0.01
Dropout rate
0.1
Reward weight w Optimizer
0.1, 0.0,0,
Adam (81 0.9, B2 0.999) (Kingma, 2014)
A.2. Baselines Introduction
To evaluate the effectiveness of the proposed ReinboT model, some representative baseline algorithms and reward design methods are considered: 1) RoboFlamingo: RoboFlamingo (Li et al., b) is a VIA model that leverages pre-trained VI.Ms for single-step visual-language understanding and models sequence history information with an explicit policy head. 2) GR-1: GR-1 (Wu et al.) is a simple and effective imitation learning method that utilizes a pre-trained video model to enhance action generation. 3) GR-MG: GR-MG (Li et al., 2025) leverages partially- data by conditioning on text instructions and goal images, using a diffusion model to generate goal images during inference. 4) PIDM: PIDM (Tian et al.,
12

---

ReinboT:: Amplifying Robot Visual-Language Manipulation with Reinforeement Learning
2024) presents end-to-end paradigm that utilizes inverse dynamics models conditioned the robot's forecasted visual states to predict actions, integrating vision and action in closed loop for scalable robotic manipulation. 5) RWR: The RWR (Peters & Schaal, 2007) we reproduced is offline RL algorithm that aims to directly optimize the VLA policy model by maximizing the cumulative reward while bypassing the thorn RL value function estimation problem. The discounting factor used our experiment is set as 0.9. The update gradient for the VIA tasks we reproduced when calculating the action loss function Lu is:
T
VaLa
Σ v log m(all, (o, s)
E o i
r(1, (o, c))
(14)
2500
Training Data
2000
Sample Size
1500
1000
500
10
ReturnToGo Value
Figure Distribution of ground-truth ReturnToGo CALVIN mixed-quality training data.
0.04
2.75
0.09
Joint Velocities
Gripper Action
0.02
0.75
0.01
2.00
0.00
00.01
-0.50
-0.02
-3.75
-0.05
Figure The sub nob- division of long-hooe horizon tasks with language instructions of "slide the door to the left" CALVIN mixed-quality training data. The black vertical dashed line represents the sub-goal of the trajectory (the same below).
13

---

ReinboT: Amplifying Robot Visual-Langunn Manipulation with Reinforcement Learning
100
(b)
50A Iagge 03B
640
614
(e)
Soximage C0e 8
(f)
SEA TEP Portoo MSR
1.00
1:0 (h)
m
1.00 (1)
(K)
ex
000
MW
Ece
m))
230 (n)
(p)
100
130
10
18
26
10
(9)
Figure The dense reward and reward component of long-horizon tasks with language instructions of "slide the door the left" in CALVIN mixed-quality training data. SGA: Sub-goal Achievement. BS: Behavior Smoothness. Same below.
1.00
3.75
Joint Velocities
Gripper Action
3.25
-2.55
-2.50
-3.00
20
Figure The sub-goal division long nong-horizon tasks with language instructions "uunn on yellow lamp" CALVIN mixed-quality training data. The black vertical dashed line represents the sub-goal of the trajectory (the same below).
14

---

ReinboT: Amplifying Robot Visual-Languun Manipulation with Reinforcement Learning
100
(b)
144 (d)
50N Inage 038
0.00
6.0
020
614
610
(e)
(f)
SEA TCP Poc on NSS
(g)
[b))
0.00
(1)
MM
65 "OP Mnjle Weeke
(K)
(mg
1.0 (n))
MZNi Accomnttoo
(0)
[p)
Pappees
004
M1
m
644
26
10
99)
Rewa10
Figure 11. The dense reward and reward component of long-horizon tasks with language instructions of "nurn the yellow lamp" in CALVIN mixed-qualit training data.
10-
0.0
EEF Velocities
Gripper Action
0.02
0.6
-0.03
10
20
30
20
Figure The sub-goal division long- horizoorizon tasks with language instructions of "Pikk up the green cup fup me" in the real-world URS successful training data.
15

---

ReinboT: Amplifying Robot Visual-Langunn Manipulation with Reinforcement Learning
[b)
1.89
SGh tmaore cen
10ag MEE
2.4
b.00
0.00
50k hoge $3M5
0.800
2.00 (f)
5G6 Frt Teitronn VCE
35.0nw (h)
Recoleation
1:07
(o0n(())
0.00 ())
3:00
0.00
14
(m)
Acceccttion
(m)
uoa (0)
(p)
Frooress
0.00
2.0
0.0
Figure 13. The dense reward and reward component of Long-horizon tasks with language instructions of "icck up the green cup me" in the real-world URS successful training data.
10-
0.0
EEF Velocities
0.91
Gripper Action
0.6
-0.03
-0.30
Figure The sub-goal division long-horizon tasks with language instructions "Put the smaller blue into red bowl" in the worldworld URS successful training data.
16

---

ReinboT: Amplifying Robot Visual-Lann Language Manipulation with Reinforcement Learning
(6)
(d)
500 100000 M00
3.80
So4 Inaze con
5004- Irage
8.0
504 Unuge 50N
2:85 (f)
CGAL baltonn Vit
(ce((g)
Aapee 1e8
(h)
sas
2.4
(1)
(xa((k)
EE accolent
8:00
000
(m)
Action Acceestoe
(()
3:00
Proceee
[.) (1)
Figure 15. The dense reward and reward component of long-horizon tasks with language instructions of "Put the smaller blue bowl into the red bowl" in the real-world UR5 successful training data.
17

---

ReinboT: Amplifying Robot Visual- Language Manipulation with Reinforcement Learning
(a) Language Instruction: pick up the red cup for me
(b) Language Instruction: put the smaller blue bowl into the red bowl
Puple
Panlle
(c) Language Instruction: take the tiger out of the red bowl and put it in the green bowl
Figure Real machine deployment the proposed ReinboT model. ReinboT can effectively complete real-world pick-and- place tasks with few-shot learning.
18


Reviews:
[{"review_id": "1mGUkG65hw", "reviewer": "Reviewer_TPRg", "summary": "This paper introduces ReinboT, a model that integrates reinforcement learning (RL) concepts, specifically the idea of maximizing cumulative dense rewards, into a vision-language-action (VLA) framework for robotic manipulation. The authors propose a reward design that breaks down long-horizon tasks into several sub-goals and factors in elements such as sub-goal achievement, task progress, behavior smoothness, and task completion. By predicting ReturnToGo (RTG) – effectively a measure of future cumulative reward – ReinboT adjusts its policy to maximize these returns within a single end-to-end learning setup. The model is built on a GPT-style transformer backbone that encodes various modalities, including images, language instructions, and proprioceptive data. In experiments on both simulated (CALVIN) and real-world robotic tasks, ReinboT achieves higher success rates and better generalization capabilities than baselines that rely solely on imitation learning or more conventional RL approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your high affirmation and recognition of our work. We respond to the reviewer’s concerns point by point.\n\n**Q1**: I think the criteria for keypoints in demonstration can be sensitive to the quality of demonstrations; for instance, when demonstrations have mis grasps, there will be false positive keypoints. Some discussions on the sensitivity of quality of demonstrations are warranted for more effective real-world usage.\n\n**R1**: The designed rule-based densification reward contains four items: sub-goal achievement, task progress, behavior smoothness, and task completion. The task completion reward is result-oriented and is used to determine whether the entire trajectory is successful. The other rewards are process-oriented and are used to assist the robot's decision-making actions. In some cases, false positive keypoints appear when there are incorrect grasping behaviors in the demonstration. However, the behavior smoothness reward suppresses the velocity and acceleration of the robot's body state and actions, thus penalizing such incorrect behaviors of violent back-and-forth grasping. Therefore, the final densified reward can be resistant to such noise in the demonstration data to a certain extent and is not so sensitive to the quality of such demonstrations.\n\n**Q2**: Comparison against other VLA architectures and RL framework, such as OpenVLA with actor-critic, can help demonstrate the particular combination of the proposed approach.\n\n**R2**: It is currently an open problem to efficiently combine VLA models such as OpenVLA with the actor-critic framework in RL. This is because OpenVLA discretizes the action space, while the standard actor-critic algorithm models in a continuous action space. Moreover, in the context of multi-modal manipulation tasks, it is also a problem to efficiently achieve accurate estimation of the critic network and reduce training and memory overhead. In contrast, the proposed ReinboT circumvents the thorny critic network estimation problem and effectively implements the benefit of maximizing the return in RL.", "claims_and_evidence": "The main claims are that (1) incorporating dense reward information into a vision-language-action transformer allows the model to better recognize and leverage mixed-quality offline data, (2) predicting a “maximized” ReturnToGo guides the policy toward actions that yield benefits for long-horizon tasks. The evidence supporting these claims are extensive on the CALVIN benchmark; in particular, the paper does a good job demonstrating that ReinBoT is an effective approach for incorporating elements of RL into a large VLA architecture.", "methods_and_evaluation_criteria": "The paper focuses on success rate and generalization capabilities of ReinBoT. These evaluation criteria are appropriate for the problem: chaining instructions tests long-horizon robustness, and OOD scenarios reveal how well the model extrapolates from its training distribution. The design of dense rewards is argued to reflect real-world aspects of manipulation tasks (like smoothness and sub-goal achievement), which is a practical choice for multi-stage robotics tasks. The combination of these evaluation metrics provides a solid overview of how well the method performs in both standard and challenging conditions.", "theoretical_claims": "This paper does not introduce new theoretical claims.", "experimental_designs_or_analyses": "The experiments contain large-scale validation on the CALVIN dataset, real-world deployment, as well as various ablation studies. The experimental methodology seems sound and standard in the literature.", "relation_to_broader_scientific_literature": "The paper’s main contribution to the broader field is showing that a single end-to-end architecture can incorporate an RL optimization perspective (via expectile regression) without needing a separate Q-function or an online adaptation stage can be an effective way to build large-scale VLAs. While prior works on VLA are discussed, more in-depth experimental validation can be helpful.", "essential_references_not_discussed": "I think all important related works are discussed in the paper.", "other_strengths_and_weaknesses": "Weaknesses:\n1. I think the criteria for keypoints in demonstration can be sensitive to the quality of demonstrations; for instance, when demonstrations have mis grasps, there will be false positive keypoints. Some discussions on the sensitivity of quality of demonstrations are warranted for more effective real-world usage. \n2. Comparison against other VLA architectures and RL framework, such as OpenVLA with actor-critic, can help demonstrate the particular combination of the proposed approach.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "My questions and suggestions are stated above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VZ3DhLCBih", "reviewer": "Reviewer_rsGY", "summary": "This paper proposed an RL framework for finetuning the Vision-Language-Action model called ReinBoT, including automatic sub-goal division and reward densification, which is critical to applying RL in VLA training. The authors also use ReturnToGo to replace vanilla challenging value estimation. ReinBoT achieves SoTA performance in CALVIN benchmark.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your positive response and affirmation of our work. We respond to the reviewer's concerns point by point.\n\n**Q1**: Experiments: ReinBoT reported a SoTA performance in the CALVIN benchmark. I am not an experienced researcher in VLA area but wanna know why not include more benchmarks like as described in OpenVLA? I think this would much strengthen this paper.\n\n**R1**: In the field of VLA for general-purpose tasks, model training generally requires a large amount of training data and computing resources, as well as a hardware platform adapted for real-machine deployment. Therefore, conducting experimental comparisons on a wider range of benchmarks is a time-consuming and laborious task that is not affordable for general laboratories.\nMoreover, in order to provide a direct comparison with previous work (OpenVLA and Pi-0), we further fine-tuned OpenVLA and Pi-0 on more than 900 real-world ur5 robot arm grasping and placing trajectories (including small square wooden blocks, plastic cups, corn, green peppers and other objects). The training configuration is 8xA100 GPUs, batch size is 8 (for OpenVLA) and 32 (for Pi-0), 40000 steps of fine-tuning. The test results show that our proposed ReinboT has competitive performance with Pi-0, with a success rate of about 0.8, which is better than OpenVLA (Only about 0.35). Compared with ReinboT, Pi-0 is pre-trained on a wider and more diverse range of manipulation data and has more model parameters (Pi-0 model parameters are about 3B, while ReinboT has about 300M). Therefore, the capabilities of ReinboT deserve further development, and expanding its training data volume and network parameter scale is an important future work. On the other hand, we speculate that the main reason for the low success rate of OpenVLA is that it only uses a single image observation and a discretized action space. Local perception makes it difficult to grasp the spatial position of objects, and discretized actions also have serious error accumulation problems in long-horizon manipulation tasks, resulting in poor performance.\n\n**Q2**: For the behavior smoothness reward defined in Eq.(6), it seems to encourage to increase the velocity and acceleration. Why this design leads to smooth actions? Or did you miss a minus sign？\n\n**R2**: The reward weight $w_3$ corresponding to the reward $r_3$ in formula (6) is a negative number. Therefore, the overall reward is to suppress the velocity and acceleration of the robot manipulation, thereby enhancing the smoothness of the action.\n\n**Q3**: What are the weights of the four reward components?\n\n**R3**: The four weights are $\\frac{0.1}{T_{traj.}}$, $\\frac{0.1}{T_{traj.}}$, $\\frac{-0.01}{T_{traj.}}$ and $\\frac{0.1}{T_{traj.}}$, where $T_{traj.}$ represents the corresponding trajectory length. The reward weights are summarized in Appendix Table 4.", "claims_and_evidence": "1. Sub-goal division and reward densification are proved to be useful in Table.1.\n2. ReturnToGo estimation is proved to be critical in ablation studies in Table.2.\n3. ReinBoT did achieve SoTA performance in CALVIN as demonstrated in Table.1.", "methods_and_evaluation_criteria": "Methods:\nThe subgoal division is a rule-based specific design for the manipulation arms, which would restrain the generality of this method to other robotic multi-period tasks. The reward densification including subgoal achievement, task progress, and behavior smoothness is effective and general to many other tasks. Dense reward is important for reducing the difficulty for RL agents. The author also employ ReturnToGo estimation directly which alleviates the value estimation problem.\n\nExperiments:\nReinBoT reported a SoTA performance in the CALVIN benchmark. I am not an experienced researcher in VLA area but wanna know why not include more benchmarks like as described in OpenVLA? I think this would much strengthen this paper.", "theoretical_claims": "This paper doesn't include theoretical analysis.", "experimental_designs_or_analyses": "It would be better if they include more than one benchmark.", "relation_to_broader_scientific_literature": "ReinBoT could be a future mainstream RL framework in RL for VLA training.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. For the behavior smoothness reward defined in Eq.(6), it seems to encourage to increase the velocity and acceleration. Why this design leads to smooth actions? Or did you miss a minus sign？\n2. What are the weights of the four reward components?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgIh6RqAvS", "reviewer": "Reviewer_TPwh", "summary": "The paper \"ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning\" introduces an end-to-end Vision-Language-Action (VLA) model, ReinboT, which integrates reinforcement learning (RL) principles to improve robotic manipulation. The core idea is to incorporate a dense reward structure to address the issue of mixed-quality training data that limits the performance of existing VLA models based purely on imitation learning. The paper proposes a novel dense reward design that captures various aspects of robotic manipulation, such as sub-goal achievement, task progress, behavior smoothness, and task completion. Additionally, the model leverages a ReturnToGo (RTG) prediction mechanism to guide decision-making by estimating future cumulative rewards. Experimental results show that ReinboT outperforms state-of-the-art models on the CALVIN benchmark and demonstrates superior few-shot learning and generalization to out-of-distribution (OOD) scenarios in real-world robotic tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful comments, which can further enhance the completeness and thoroughness of our work. In response to the reviewer's concerns about more baseline comparisons, we have added more experiments. We respond to all concerns point by point.\n\n**Q1**: A direct comparison with Open-VLA and Pi-0.\n\n**R1**: We further fine-tuned OpenVLA and Pi-0 on more than 900 real-world ur5 robot arm grasping and placing trajectories (including small square wooden blocks, plastic cups, corn, green peppers and other objects). The training configuration is 8xA100 GPUs, batch size is 8 (for OpenVLA) and 32 (for Pi-0), 40000 steps of fine-tuning. The test results show that our proposed ReinboT has competitive performance with Pi-0, with a success rate of about 0.8, which is better than OpenVLA (Only about 0.35). Compared with ReinboT, Pi-0 is pre-trained on a wider and more diverse range of manipulation data and has more model parameters (Pi-0 model parameters are about 3B, while ReinboT has about 300M). Therefore, the capabilities of ReinboT deserve further development, and expanding its training data volume and network parameter scale is an important future work. On the other hand, we speculate that the main reason for the low success rate of OpenVLA is that it only uses a single image observation and a discretized action space. Local perception makes it difficult to grasp the spatial position of objects, and discretized actions also have serious error accumulation problems in long-horizon manipulation tasks, resulting in poor performance.\n\n**Q2**: How does ReinboT compare in terms of inference speed relative to prior VLA models like OpenVLA?\n\n**R2**: In our ur5 real machine deployment, the inference speeds of ReinboT, Pi-0 and OpenVLA models are about 0.09s, 0.12s and 0.2s respectively. In fact, ReinboT also uses action chunking technology to generate 64 future actions at a time on the ur5 manipulation task. If we execute multiple steps of action (currently only one step of action is executed by default), the inference efficiency of the model can be further improved. In addition, we did not make any additional designs like Pi-0. The ReinboT model is deployed on the local platform, and the OplenVLA and Pi-0 models are deployed on the remote server. Local deployment platform: CPU: 12th Gen Intel(R) Core(TM) i7-12700F; GPU: NVIDIA GeForce RTX 2060/PCIe/SSE2. Remote server platform: CPU: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz; GPU: NVIDIA A100-SXM4-80GB.\n\n**Q3**: How does Reinbot perform compared to imitation learning + data curation?\n\n**R3**: The core idea of combining imitation learning with data curation is to use a metric that measures the distance between source and target data to filter out high-quality data for imitation learning. Therefore, in our VLA problem setting, an intuitive data curation baseline approach is to use ReturnToGo as a trajectory quality metric and filter out high ReturnToGo data for imitation learning. The performance comparison of the proposed ReinboT and the data curation baseline method on CALVIN is shown in the following table. Experimental results show that ReinboT can achieve superior performance. Compared with data management methods that directly discard low-quality data, ReinboT can have a more fine-grained understanding of the distribution of mixed-quality data, thereby better guiding action generation.\n### Comparison on CALVIN Mixed-quality Data (Test D)\n| Algorithms                     | 1    | 2    | 3    | 4    | 5    | Avg. Length       |\n|--------------------------------|:----:|:----:|:----:|:----:|:----:|:-----------------:|\n| Imitation learning + data curation | 0.69 | 0.47 | 0.28 | 0.22 | 0.14 | 1.78             |\n| **ReinboT (Ours)**             | **0.79** | **0.58** | **0.40** | **0.28** | **0.21** | **2.26 (+27.0%)** |\n\n**Q4**: the reward engineering seems to require some insights from humans, is there any idea to automate the reward engineering?\n\n**R4**: Previous work used off-the-shelf multi-modal LLM to automatically label robot trajectories with rewards. However, the prompts of LLM still require time and effort to design. The hallucination problem of LLM also significantly affects the accuracy of the designed rewards. In contrast, the rule-based densified reward we proposed is based on the characteristics of long-horizon multi-modal manipulation tasks. It is naturally suitable for manipulation tasks, thus having a wider range of applicability. For new manipulation tasks, we only need to slightly adjust the velocity threshold as the key point criterion and the reward weight used to balance the reward value. This process does not impose an undue burden on researchers' time and energy.\n\nWe believe that the reviewer's concerns have been fully addressed. Please convey any additional concerns that hinder the reviewer from improving the score in a timely manner. Thank you again for the reviewer's detailed and thorough review comments.", "claims_and_evidence": "- **ReinboT improves robotic manipulation accuracy over existing VLA models.** This claim is supported by experimental results showing ReinboT’s superior performance on CALVIN mixed-quality data.\n\n- **Dense reward modeling enhances learning efficiency.** The ablation studies confirm that each component of the dense reward contributes to performance improvements.\n\n- **ReturnToGo prediction enables better decision-making.** The paper presents empirical evidence that RTG-based predictions enhance long-horizon planning.\n\n- **ReinboT generalizes better to unseen tasks.** The few-shot learning and OOD experiments support this claim.", "methods_and_evaluation_criteria": "The methodology appears well-structured and appropriate for the problem domain.", "theoretical_claims": "The paper does not heavily focus on formal theoretical contributions.", "experimental_designs_or_analyses": "The experimental design is comprehensive and well-executed. The paper includes:\n\n- Baseline comparisons with state-of-the-art VLA and RL-integrated models.\n\n- Ablation studies to examine the impact of dense reward components.\n\n- Real world experiments \n\nHowever, I wonder how Reinbot performs compared to OpenVLA [Kim et al., 2024], Pi-0 [Octo Model Team et al., 2024], or imitation learning + data curation work [Nasiriany et al., 2022; Du et al., 2023; Kuhar et al., 2023].\n\n[Kim et al., 2024] OpenVLA: An Open-Source Vision-Language-Action Model, arXiv 2024\n\n[Octo Model Team et al., 2024] Octo: An Open-Source Generalist Robot Policy, \n\n[Nasiriany et al., 2022] Learning and retrieval from prior data for skillbased imitation learning. CoRL 2022\n\n[Du et al., 2023] Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. arXiv 2023\n\n[Kuhar et al., 2023] Learning to discern: Imitating heterogeneous human demonstrations with preference and representation learning. CoRL 2023", "relation_to_broader_scientific_literature": "The paper builds upon previous works in:\n\n- Imitation learning for VLA models (e.g., GR-1)\n\n- Offline RL and return maximization techniques (e.g., Decision Transformer, Reinformer)\n\n- Reward modeling for robotic control\n\nReinboT integrates reinforcement learning (RL) principles to improve robotic manipulation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n- The integration of RL principles into a VLA model is novel and well-motivated.\n\n- The dense reward formulation is carefully designed and empirically validated.\n\n- The experimental results are strong, demonstrating both simulation and real-world efficacy.\n\nWeaknesses and Questions\n\n- A direct comparison with Open-VLA and Pi-0 would provide further clarity on ReinboT’s advantages.\n\n- How does ReinboT compare in terms of inference speed relative to prior VLA models like OpenVLA [Kim et al., 2024], particularly in real-time applications?\n\n- How does Reinbot perform compared to imitation learning + data curation [Nasiriany et al., 2022; Du et al., 2023; Kuhar et al., 2023]?\n\n- the reward engineering seems to require some insights from humans, is there any idea to automate the reward engineering?\n\n[Kim et al., 2024] OpenVLA: An Open-Source Vision-Language-Action Model, arXiv 2024\n\n[Octo Model Team et al., 2024] Octo: An Open-Source Generalist Robot Policy, \n\n[Nasiriany et al., 2022] Learning and retrieval from prior data for skillbased imitation learning. CoRL 2022\n\n[Du et al., 2023] Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. arXiv 2023\n\n[Kuhar et al., 2023] Learning to discern: Imitating heterogeneous human demonstrations with preference and representation learning. CoRL 2023", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cyaOy7JIF3", "reviewer": "Reviewer_bfnn", "summary": "The paper introduces ReinboT, a VLA model integrated with offline RL for robotic manipulation. It improves decision-making by predicting dense rewards, which guide the robot to maximize long-term returns. ReinboT outperforms baseline models in simulated and real-world tasks, demonstrating superior few-shot learning and generalization.\n\n## update after rebuttal\nI confirm my score. Authors addressed comments and added clarity and results to the original submission.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your positive and insightful response to our work. In response to the reviewer's concerns about more baseline comparisons and reward ablation, we have added more experiments. We respond to all concerns point by point.\n\n**Q1**: While the proposed dense reward function works well for certain tasks, it may not be flexible enough for more complex environments.\n\n**R1**: One of the core contributions of our work is to propose a sparse reward densification method for general visual language manipulation tasks. We believe that the design principles of these four rewards will greatly inspire the reward design for more complex environments. More refined and flexible reward design for complex environments is an important future work.\n\n**Q2**: Do experiments in a more complicative environment to test the robustness of ReinboT.\n\n**R2**: We compare ReinboT, OpenVLA and Pi-0 on more challenging real-world tasks. Please see our response (**R1**) to **Reviewer TPwh**.\n\n**Q3**: Do an ablation study on the reward $r_1$ and test it to see whether ReinboT needs the complicative reward design.\n\n**R3**: The ablation experiment of reward $r_1$ is shown in the table below. The experimental results show that the loss of any part of $r_1$ will lead to a decrease in model performance. Therefore, we do need to explore the characteristics of multi-modal manipulation tasks from various aspects of modal information to better characterize the data quality distribution.\n### Ablation Study on $r_1$\n| Design Choices                 | 1    | 2    | 3    | 4    | 5    | Avg. Length        |\n|--------------------------------|:----:|:----:|:----:|:----:|:----:|:------------------:|\n| **ReinboT (Ours)**             | **0.79** | **0.58** | **0.40** | **0.28** | **0.21** | **2.26**          |\n| W/o sub-goal achievement $r_1$    | 0.72 | 0.50 | 0.32 | 0.20 | 0.12 | 1.87 (-17.2%)     |\n| W/o proprioception state MSE   | 0.72 | 0.52 | 0.33 | 0.24 | 0.20 | 2.01 (-11.1%)     |\n| W/o image state MSE            | 0.75 | 0.50 | 0.33 | 0.27 | 0.22 | 2.07 (-8.4%)      |\n| W/o image state SSIM           | 0.71 | 0.53 | 0.31 | 0.19 | 0.18 | 1.92 (-15.4%)     |\n| W/o image state ORB            | 0.76 | 0.51 | 0.32 | 0.23 | 0.16 | 1.97 (-12.8%)     |\n\n**Q4**: Provide the readers with experience on how to set  $\\lambda$ and $m$.\n\n**R4**: The original imitation learning loss is still the main supervisory signal for model learning, while the ReturnToGo prediction is more of an auxiliary role. Therefore, the $\\lambda$ needs to be appropriately set smaller, and we recommend starting from around 0.001.\nOn the other hand, the $m$ is utilized to control the model’s sensitivity to different expectation levels, thereby adjusting the model’s fitting characteristics for the ReturnToGo distribution. After extensive ablation experiments, we recommend setting the $m$ from around 0.9.\n\n**Q5**: Why does ReinboT need to predict future image?\n\n**R5**: Predicting future images is just consistent with the baseline method (GR-1). The core contribution of ReinboT lies in the introduced reward densification method and the organic integration of the RL return maximization principle in VLA.\n\n**Q6**: Why applying RL in VLA may bring obstacles to transformers? Did you fine-tune VLA with PPO? These results may support the claim more convincingly.\n\n**R6**: Transformer architectures, especially large pre-trained VLMs, are sensitive to noisy gradients introduced by online RL. Directly fine-tuning the entire VLA model using RL loss can destabilize training. Unlike language tasks, robotics online RL requires exploration, which introduces high-variance updates that can corrupt the carefully tuned parameters of the VLM or even cause it to change catastrophically. Although we do not currently fine-tune VLA directly with PPO, recent research [1] confirms the above. Intuitively, in the online actor-critic RL learning setting, the exploration behavior of the policy is prone to produce some bad trajectory states, and the critic network that has just been trained will have difficulty accurately evaluating the value of these trajectory states, thus misleading the fine-tuning of the VLA policy model and causing performance degradation. In the context of multi-modal manipulation tasks, how to effectively achieve accurate estimation of the critic network and reduce training and memory overhead is also an open question. In contrast, our currently proposed ReinboT bypasses the thorny critic network estimation problem and effectively achieves the benefits of maximizing rewards in RL under the offline learning setting. Further extending ReinboT to the online setting is an important future task.\n\n[1] Guo, Y., Zhang, J., Chen, X., Ji, X., Wang, Y. J., Hu, Y., & Chen, J. (2025). Improving Vision-Language-Action Model with Online Reinforcement Learning. arXiv preprint arXiv:2501.16664.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The theoretical foundation of ReinboT is not formally established through rigorous proofs or mathematical derivations.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Dense reward design, return-to-go prediction.", "essential_references_not_discussed": "To the best of my knowledge, no", "other_strengths_and_weaknesses": "Strengths:\n- ReinboT’s dense reward mechanism and Return-to-Go (RTG) prediction provide a novel approach to improving long-horizon decision-making in robotic RL. These methods offer creative solutions to sparse reward problems.\n\nWeaknesses:\n- The model’s performance is highly sensitive to the choice of $\\lambda$ and $m$. If $m$ is too large, the model may become overly optimistic in estimating rewards, leading to poor adaptation and decreased performance. This sensitivity limits ReinboT’s generalization across tasks and datasets.\n- While the proposed dense reward function works well for certain tasks, it may not be flexible enough for more complex environments. A more refined reward design may be needed to better capture the nuances of complex tasks.", "other_comments_or_suggestions": "- Do experiments in a more complicative environment to test the robustness of ReinboT.\n- Do an ablation study on the reward $r_1$ and test it to see whether ReinboT needs the complicative reward design.\n- Provide the readers with experience on how to set hyperparameters $\\lambda$ and $m$ when adopting ReinboT to other scenarios.", "questions_for_authors": "- Why does ReinboT need to predict future image?\n- The paper mentioned that \"in the classic RL algorithm, maximizing the Q-value is utilized to achieve the best policy model. This implies that applying RL in VLA necessitates the introduction of an additional RL loss function. Such an addition may pose obstacles to the learning process of models like transformers.\". Why applying RL in VLA may bring obstacles to transformers? Did you fine-tune VLA with PPO? These results may support the claim more convincingly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes ReinboT, a framework for finetuning vision-language-action (VLA) models using techniques from reinforcement learning (RL). In an effort to address the challenges of imitation learning from mixed-quality demonstrations, ReinboT provides dense reward feedback to VLA models by decomposing long-horizon tasks into subgoal sequences that are used together with behavior smoothness and task completion to define a dense reward signal. ReinboT employs ReturnToGo (RTG), a predicted measure of the future cumulative reward, to further support the fine-tuning of VLAs. Evaluations on the CALVIN benchmark demonstrate that ReinboT outperforms contemporary baselines, while an ablation study reveals the contributions of the key components of the framework.

The paper was reviewed by four referees, who agree on a number of the paper's strengths and weaknesses. Among them, at least three of the reviewers find that the paper provides clear evidence regarding the benefits of the ways in which ReinboT uses RL methods to improve VLA training, including the advantages of subgoal decomposition and the use of the ReturnToGo objective, with ReinboT achieving state-of-the-art performance. The reviewers raised a few concerns with the paper as initially submitted, notably the importance of including comparisons to other VLA architectures and RL frameworks (e.g., OpenVLA with actor-critic). In their rebuttal, the authors noted that the incorporation of actor-critic methods is challenging since the actions are discrete, though it is worth noting the existence of work adapting actor-critic methods to discrete action spaces [Petros, 2019]. During the rebuttal, the addition of a comparison to OpenVLA and Pi-0 suggests that ReinboT is competitive, however more a more extensive evaluation is required before any conclusions can be drawn. That said, the paper provides interesting insights into the ways in which techniques from RL can be utilized to improve the training of VLAs, which would be of interest to many in the robot learning community.

The AC notes that TPwh, who provided the most critical review, did not engage in the discussion process, despite several attempts on the part of the AC.


Christodoulou, Petros. "Soft actor-critic for discrete action settings." arXiv preprint arXiv:1910.07207 (2019).
Decision: Accept (poster)