Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance
Authors: Mitsuhiko Nakamoto
Oier Mees
Aviral Kumar
Sergey Levine
Venue: CoRL 2024 (2024)
Keywords: generalist policies
value functions
robot reinforcement learning
TL;DR: We introduce Value-Guided Policy Steering (V-GPS), a novel approach that improves performance of pre-trained generalist robotic policies by re-ranking their actions according to a value function learned via offline RL at deployment time.

Abstract: Large, general-purpose robotic policies trained on diverse demonstration datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of different scenes, and for acquiring broad repertoires of manipulation skills. However, the data that such policies are trained on is generally of mixed quality -- not only are human-collected demonstrations unlikely to perform the task perfectly, but the larger the dataset is, the harder it is to curate only the highest quality examples. It also remains unclear how optimal data from one embodiment is for training on another embodiment. In this paper, we present a general and broadly applicable approach that enhances the performance of such generalist robot policies at deployment time by re-ranking their actions according to a value function learned via offline RL. This approach, which we call Value-Guided Policy Steering (V-GPS), is compatible with a wide range of different generalist policies, without needing to fine-tune or even access the weights of the policy. We show that the same value function can improve the performance of five different state-of-the-art policies with different architectures, even though they were trained on distinct datasets, attaining consistent performance improvement on multiple robotic platforms across a total of 12 tasks. Code and videos can be found at: https://nakamotoo.github.io/V-GPS

Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance
Mitsuhiko Nakamoto
Oier Mees
Aviral Kumar
Sergey Levine
UC Berkeley *Carnegie Mellon University
GGogge DeepMind
https://nakamotoo. gi ttthhbioo/V-GPS io/V-GPS
Large, general-purpose robotic policies trained on diverse demonstra- tion datasets have been shown to be remarkably effective both for controlling a variety of robots in a range of different scenes, and for acquiring broad repertoires of manipulation skills. However, the data that such policies are trained on is gen- erally of mixed quality - not only are human-collected demonstrations unlikely to perform the task perfectly, but the larger the dataset is, the harder it is cu- rate only the highest quality examples. It also remains unclear how optimal data from embodiment is for training another embodiment. In this paper, we present general and broadly applicable approach that enhances the performance of such generalist robot policies at deployment time by re-ranking their actions according to a value function learned via offline RL. This approach, which we call Value-Guided Policy Steering (V-GPS), is compatible with a wide range of differ- ent generalist policies, without needing to fine-tune even access the weights of the policy. We show that the same value function can improve the performance of five different state-of- the-art policies with different architectures, even though they were trained on distinct datasets, attaining consistent performance improve- ment on multiple robotic platforms across total of 12 tasks.
Abstract:
Keywords: Generalist Policies, Value Functions, Robot Reinforcement Learning
Test-time Policy Steering via Value Guidance
Generalist Robot Policy
Single V-GPS Value Function
1. Action Proposal
2. Re-Rank with Q-value
Execute High-Ranking Action
Frozen
Black-box
API only
Octo
OpenVLA
Rotata,, G
RT-X
[9, m. og) oifaa st.
aila.o.1 d
etc.
Qptst,
Figure 1: (V-GPS) We introduce Guided Policy Steering (V-GPS), novel approach that improves the performance of pre-trained generalist robotic policies by re-ranking their actions at deployment time based on value function Iearned via offline RI. The single V-GPS value function be combined with any off-the-shelf generalist policy in plug-and-play manner, without the need to fine-tune or access the policy's weights, improving downstream performance across multiple robotic platforms.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Large, high-capacity models trained diverse datasets are key the effectiveness of modern ma- chine learning methods [1, 2, 3, 4, 5]. However, this recipe presents major challenge in robotic learning: while large datasets collected from diverse sources have recently made the study of large- scale robotic learning feasible [6. 7], such data sources are typically of mixed quality, and recovering high-performing and fluent policies from such suboptimal data presents a major challenge. While state-of-the-an imitation learning methods can effectively replicate the distribution of demonstra- tions [8. 6]. the mixed quality of these datasets makes this distribution fall short of the perfor- mance we would like from our robotic systems. More concretely, generalist policies often fail due to imprecise manipulation, such as failed grasping or early dropping, despite their strong semantic generalization (as is evident from a number of existing results; see Appendix G of Brohan et al. [8] & Section 5.2 of Black et al. [10]). These issues become more severe when the policy encounters environmental distribution shifts, even if these shifts are extraneous in nature (e.g., changes in table texture, camera pose, and distractors; see Figures and 9 of Li et al. [11]).
How improve the precision, robustness, and proficiency of these generalist policies while still retaining the best benefits of their scale and generalization capabilities? While one intuitive method could involve refining the policy through fine-tuning, this approach is infeasible in the non-stationary real world. Not only would each adaptation cycle require costly human tele-operated or instrumented data collection, but also unintuitive hyperparameter tuning to prevent the model from losing its generalist capabilities. If we can instead devise an approach to preserve the generalist policy, but simply "steer" it in a way that improves precision and robustness upon deployment, that would be most desirable. What is good way to steer an off-the-shelf generalist policy? Our key insight is that re-ranking multiple action proposals from generalist policy using value funetion at test-time allows us to accomplish this. Re-ranking with some sort of value or reward functions is extremely effective at improving the reasoning capabilities of large language models (LLMs) [12, 13, 14] in single-step "bandit" problems. However, is yet to be shown effective for multi-step robotic manipulation problems with stochastic environment interaction from raw pixel observations, outside of simulated gym environments [15, 16].
In this paper, we build a recipe to train general robotic value function via offline reinforcement learning (RL) and show that despite the multi-step nature of robotic problems, value-function guided test-time action selection is an effective approach for improving generalist policies. As shown in Fig- ure 1, this approach enables us to directly improve the generalist policy on scenes and manipulation problems encountered at deployment time, unlike prior offline RL methods that use a value function on the training data and hence, still fail on shifts encountered upon deployment. In addition, using the value function only test-time is modular and plug-and-play with any generalist policy, and does not require tuning bells and whistles as conventional robotic offline RL pipelines [17, 18).
The main contribution of this paper is V-GPS, Value-Guided Policy Steering, a method that uses offline RL value functions to steer generalist robotic policies. We build a recipe to pre-train a value function on diverse robotic datasets, demonstrating that their use improves maneuvers of several open-source generalist policies. To our knowledge, this is the first work to leverage test-time ac- tion sampling for real-world robotic generalist policies. We conduct extensive evaluations in both simulated [1 [] and real-world environments [19], using two different embodiments, across total of 12 tasks, and on top of five state-of-the thethe-art open-source generalist policies including Octo [9], RT1- x [6], and VLe [20]. V-GPS achieves an improvement of +82% % in real-world manipulation tasks, and consistently enhances all five generalist policies across multiple embodiments.
2 Related Work
Large-scale Robotic Datasets and Policies. Many prior works have collected and open-source robotic datasets [7, 21, 19, 22, 23, 24, 25, 26, 27, 28, 29]. These datasets include various quality of data that have been collected in diverse ways, ranging from human tele-operations [7, 21, 19, 26] to autonomous collection with scripted random policies [22, 23]. Recent efforts to aggregate

---

these existing datasets [6] have made learning from large-scale, multi-source datasets more feasi- ble for the community. Thus, a number of works have leveraged these large-scale datasets to train general- purpose robotic policies, which have shown generalization to controlling multiple robot ma- nipulators with a single policy [9. 6], to unseen tasks [8, 30J, and to new language instructions or goals [31, 30, 9, 32, 33, 34J. RT-X [6] and Octo [9] have built and open-sourced robotic foundation models with high generalization capability by scaling the policy with Transformer architecture [35] and training with advanced imitation learning methods [36, 37J. However, these generalist policies often fail due to imprecise manipulation, especially when the policy encounters environmental dis- tribution shifts [10, 11]. Our work is broadly applicable to these off-the-shelf policies, aiming to improve their performance by seamlessly integrating a value function at test time.
Value-based Offline RL for Robotics. Prior works have suggested that offline can, in principle, recover more optimal behavior than imitation learning from mixed-quality data [38, 39]. For real- world robotic tasks, several studies have also shown offline RL to be effective for scaling with large datasets [17, 40, 19] and large models [18, 41]. RL policies trained with value functions [42] can naturally learn to predict actions that maximize long-term rewards [43]. However, these methods typically leverage standard model-free offline RL algorithms [44, 45, 46, 47, 48, 49] that require using value functions to train and update the policy, so that the policy models are often limited to Gaussian distributions. This makes it difficult to scale the policy state-of-the-ar expressive architectures or leverage pre-trained generalist robotic policies. In contrast, our method provides a more general and flexible way to leverage the value function that can be integrated with any off- the-shelf, black-box pre-trained policy.
Sampling-based Action Selection. Another way to leverage value functions to use them for sampling- based action selection. In this approach, multiple actions are sampled from the policy and then ranked using the value function, with the top actions selected and executed. For language models, sampling-basee -based action selection has been shown be effective in improving performance for tasks such as Q&A and summarization [12, 13, 14, 50, 51]. In the robotics domain, Brohan et al. [52] demonstrates the effectiveness of using a language-grounded value function and employs it to score high-level language commands in skill space. However, their focus to enhance high- level reasoning rather than improve low-level performance. Prior work [15, 53, 16] has also shown that training a value function to directly score low-level actions is effective on D4RL, simulation tasks [54]. However, this approach has not yet been applied to diverse, high-ddmensional, real- world robotic tasks. Our work is the first to show that training value functions real- world robotic datasets can effectively guide sampling-base based low-level action selection, leading to improvements in large-scale robotic foundation models.
3 Preliminaries and Problem Statement
We study problems where we wish to control robot through language instructions. We assume access to generalist, inguagg-conditionee robotic policy æ (a ss, Z), which can sample multiple actions ....... ax given the current state and language command l. Note that we do not assume access to the model weights of т,, allowing the policy to be completely black-box.
For building our approach, we will consider the formalism of a Markov decision process (MDP) M AS 7)) S, denote the state and action spaces, and P(s'|s, a) and r(s, a) denote the dynamics and reward functions. 7 E (0, 1) denotes the discount factor. The value function Q(s, a) represents the long-term discounted return Σ. 7 R (st, az). Our approach will prescribe recipe to leam this value function and then show that it is helpful in steering pre-trained generalist policy. In our setting, the dataset is a language-annotaan where each trajectory 7" consists of a sequence of states s/ E and actions a. E A along with a natural language command [" describing the task performed in the trajectory.
robot dataset D = {(r1, 77), (r², (7), ??)..... (rN, (r", )}²)
4 Analysis: Failure Modes of Generalist Policies
To motivate the failure modes of generalist policies and develop our approach, we begin by inves- tigating failure modes associated with generalist robotic manipulation policies. For this analysis, use the Octo-small-1.5 model [9], open-source transformer-based generalist robotic policy
3

---

trained on the OXE dataset and attempt to investigate some failure modes of this policy. The videos can be found at https://nakammn
github.ion io/VVGPS.
Case Failure of precise grasping. We first evaluate the Octo policy on a real WidowX robot platform for the task "put pepper in pot" (see Scene A in Figure 3). The surface of the plastic green pepper is slippery and presents an uneven curvature, often making it critical to choose the grasp point and magnitude of the gripper action appropriately for reliable grasp (see Figure 2). Even when policies can grasp the green pepper, imperfect grasp points or gripper actions often lead to the object falling off the gripper while the task is being executed.
Pre-maturely timed at- tempts to complete the task. We conduct an additional study on the task "put mushroom on cloth," as shown in Figure 2. Unlike the green pepper, the mushroom is relatively casier to grasp because it's a soft ob- ject. In our evaluation, we found that the Octo policy is indeed able to suc- cessfully grasp the object and move it towards the cloth. However, it tends to drop the mushroom pre-maturely, such that the mushroom does not land on the cloth. In addition to such pre- mature attempts to complete the task, also observe cases where Octo does not release the object in timely manner. Often, the target obtains and remains stuck in the gripper, and arbitrary arm drifts during this period eventually cause the object to fall outside the target container. For example, in the task "put sushi in pot" in Scene C (see Figure 3), the model tends to hold onto the sushi for too long, resulting in it being dropped outside of the container, as shown in Figure 2.
Case 2:
"put pepper por"
"put mushroom an cloth
"put sushi pot"
Figure 2: (Failures of Octo) Octo policy encounters failures such as imprecise grasping (first row), dropping the object prematurely (second row), and holding onto the ohject for too long (third row).
5 V-GPS: Value-Guided Policy Steering
While these failure cases may vary among different policies and scenarios, they highlight the room for improvement in the precision and robustness of generalist robotic policies. In this section, we will describe our approach V-GPS which utilizes a value function to improve generalist policies to avoid these failures. The key insight behind V-GPS is to use value function "re-rank" multiple actions sampled from the generalist praited policy and execute the action that the value function thinks is most likely to succeed. We achieve this by first training language-conditione value function on robotic dataset and then combining it with generalist policies at test time. We describe each of the phases of V-GPS below.
5.1 Training: Learning Value Function via Offline RL Pre-training
The main component of V-GPS is a language-connditioned Q function Qa a, l), where 8 is the state, a is the action, and / is the language instruction. To train such a value function, we first need to obtain a reward function that can be used to supervise the value function training. Recall that in our problem setting, we are only provided with a dataset D of each entry in D consists of trajectory 7¹ and language instruction 1 To convert this dataset into a form that is amenable training a Q-function, we annotate the last H transitions of this trajectory with sparse binary reward value of +1 to indicate completion of the task specified by the language instruction. The reward values for all other transitions in this trajectory are marked as O. In our experiments, we set the value of H = 3 following Kumar et al. [17], which utilized an analogous scheme for learning value functions but with one-hot task descriptors.
robotic data, where
Equipped with this reward function, in principle, one use any offline RL or policy evaluation algorithm to fit Qg. Each algorithm will fit the value function of different policy: while algorithms

---

such as SARSA [55] will fit the value function of the behavior policy, "full" offline RL methods such as conservative Q-learning (CQL) [46], calibrated Q-learning (Cal-QL.) [56], and implicit Q- learning (IQL) [47] attempt to find the optimal value function supported on actions sampled from the behavior policy. Our intended approach, which uses value functions for re-ranking, presents some unique requirements: (1) since we only utilize the value function once to re-rank actions from the generalist policy (as opposed to iterative re-ranking),
need this value function to be close as possible to the optimal value function; and (2) since we wish use one single value function for steering multiple generalist policies that are trained on large-scale diverse datasets, we want our value function to be robust to -itto---iittitttionn (OOD) actions. Given these requirements, we choose to utilize Cal-QL [56] as our main algorithm for training the value function, as attempts to approximate the "best in support" approximation of the optimal value function while being robust noisy actions due its conservative objective. While we use Cal-QL for our main algorithm, we demonstrate that IQL is also effective for V-GPS in Appendix A.
Formally. Cal-QL trains Q function Qo a, I) with the following objectives, where 8* Q2 is the backup operator applied to delayed target Q-network Qa. Q" I) is the Q-value of a reference policy μ., and a is a hyperparameter to control the conservative penalty. The conservative regularizer aims to penalize the learned Q-function for OOD actions while compensating for this pessimism on actions seen in the training dataset.
(0( ar (Bo,tDD,an [max (Qo a, (s, a, 1)) II (Qa(s, a, a,1)))
Calibrated conservative regolarizer R(0))
1 + 2
B3 -B
(1)
Implementation details. In our experiments, had utilize several design choices and im- plementation details to obtain a good value function. While a complete set of hyperparameters is provided in the Appendix B, here we discuss some central design choices.
(i) Reward function. As discussed above, since our offline datasets do not specify reward annota- tions, we label the last H steps of demonstration rollout with a reward. Following the binary reward scheme from Kumar et al. [17], where choose H 3. In addition, instead of utilizing reward values 0 and 1, we found it better to utilize shifted reward values of 0 and -1
(ii) Model architecture. Our language-conditioned value function Qa(s, a, /) uses a ResNet-34 [57] image encoder with FiLM language conditioning. While this architecture has been shown to be effective for learning language-conditinne behavior cloning policies [19, 58, 59], we find it also effective for learning value functions. The language instructions are first processed by frozen MUSE encoder [60], and then passed into every block in ResNet with FiLM conditioning [61].
5.2 Deployment: Test-Time Action Re-Ranking
Once we obtain a value function, we can use it steer any generalist policy æ upon deployment. A simple idea for doing this would be to use the value function for "re-ranking" multiple action didates sampled from the generalist policy. Specifically, at any given moment during deployment, given the current observation 8 and the language prompt I, we first sample K actions {aI..... ax } from the generalist policy ग, and query the value function Qo to get scores for each action can- didate. Given these scores, one can choose which actions to select by either acting greedily as a: = arg max Q (s, a;), or sample the action from "re-ranked" categorical distribution obtained a,.==
by computing a temperature-controlle softmax over Q-values:
(Qo (1))
Qo (st,
at Softmax
3
where 3 is temperature parameter that controls the sharpness of the distribution, making the sam- pling process more and more greedy as - O. This hyperparameter makes method more flexible, as it allows to strike balance between how much we trust the policy and how much
5

---

Algorithm V-GPS: Test-Time Action Selection & Execution
Require: Language-condtioned policy и (a St, 1), Q-function (s,s, a, 1), initial state s0x, lan- guage command fest, maximum time step T, number of actions sample K, temperature 3 0
1:
2: while S do
3: Sample {a1...., a ~ (a | .1)
Propose K actions from policy Re-rank and select high-value action
4: Select a, Softmax (Qaloyai) (o((.ax)
5: Execute a,
6: S(+] +- New observation
7: ++1+1
8:
end while
rely the value function. Further details of our design choices and implementations be found in the Appendix B. Pscudocode for test-time action re-ranking and control is provided in Algorithm 1.
Experimental Evaluation
The goal of our experiments is to evaluate the effectiveness of V-GPS improving the robustness and precision of number generalist policies for open-worke language- -uided robotic manipula- tion problems. To this end, we aim answer the following questions:
1. Can V-GPS improve the downstream performance of a number of off-the-shelf' generalist policies across different embodiments?
2. What kind of failures of generalist policies does V-GPS address?
To answer these questions, we conduct evaluations both simulated and real-world environments, using two different embodiments, across total of 12 tasks, and top of five state-of-the. open-source generalist policies. Note that embodiment data for all policies across both real-world and simulated tasks.
the same single value function trained on cross-
6.1 Experimental Scenarios and Comparisons
Training dataset: To apply V-GPS on cross-embodiment tasks, we trained single value function on a mix of Bridge V2 dataset [19] and Fractal dataset [24, 6). The Bridge V2 dataset consists of 45K language-annotated manipulation demonstrations collected 24 environments at 5Hz. The Fractal dataset a collection of open-world manipulation demonstrations, comprising 130K episodes that cover than 700 tasks collected on the Google Robot.
WidowX Robot
Google Robot
Scene
Scene
Spoon Towel
Carrot Plate
Pick Can
groon poppor por" "uu sweet polaro on cleti"
mushromo "put muirrom pat"
Stack Blocks
Eggplant Basket
Ypus "put spoon pas"
Real World
SIMPLER
Figure (Experimental setup) We evaluate our method 12 tasks total. the real-world WidowX robot platform, we study 6 tasks across 3 different scenes. In the SIMPLER simulated evaluation suite, we study tasks on the WidowX platform and 2 tasks on the Google Robot.

---

Real-world setup and tasks: We conduct our real-robot evaluations 6 DOF WidowX250 robot arm. Our evaluations were carried out across 6 tasks in 3 scenes as shown in Figure 3.
Simulation setup and tasks: Our simulated experiments are performed in the SIMPLER environ- ment [11]. SIMPLER is real-to-sim evaluation suite designed specifically for real-robot policies, as it can accurately reflect real-world performance. We evaluate 6 tasks two robot embodiments: 4 tasks on the WidowX arm and 2 tasks on the Google Robot, as shown in Figure 3.
Generalist policies: We evaluate V-GPS on top of five different general-purposs robotic policies:
Octo-small [9], 27M parameter open-source generalist policy pre-trained on mix of 25 dif- ferent datasets from the Open-X-Emmoddmmen (OXE) dataset [6]. The policy uses a transformer backbone based on ViT-S [62], followed by a diffusion action head to model expressive action distributions. While the model can take either language instruction an image as goal, we its language-conditionel feature for experiments.
Octo-base [9], the larger version of Octo, with 93M parameter based on ViT-B [62] backbone.
Octo-small-1.5 [9], the updated version of the Octo-small model. The same architecture is trained with augmented language instruction via rephrasing from GPT-3.5 and repeating the language tokens at every context window, aiming for improved language understanding.
RT1-X [6], 35M parameter transformer policy pre-trained on the OXE dataset.
Open [20], 7B parameter vision-language-attie action model, trained 970k episodes of robotic demonstration from OXE dataset [6]. The policy was fine-tuned pre-trained vision language model, Prismatic [63].
We provide further details about the baselines and the evaluation setup in Appendix C and D.
6.2 What Kind of Failure Modes of the Generalist Policy Does V-GPS Address?
Real-world results. We present the perfor- mance real-world tasks in Table 1. V-GPS consistently improves Octo-small-1.5 in all tasks, with notable improvements of +55% in Scene A, +92% in Scene B, and +100% in Scene Qualitatively, V-GPS successfully re- solved the failure modes discussed in Section 4. For example, the "put pepper in pot" task in Scene A, which requires precise grasping, V-GPS doubles the success rate from 15% to 35%. Observe Figure that the rohot can grasp the slippery pepper more reliably, leading improved performance. Furthermore, V-GPS largely addresses the pre-mature and untimely release of objects in Scenes B and C. As ex- ample, on the "put mushroom on cloth" task, incorporating the value function accurately up- weights the gripper close action until the mushroom is over the cloth, allowing the generalist policy to deviate from its default behavior of releasing the mushroom too early, as shown in Figure 4. This alone doubles the performance on this task. On the "put sushi the pot" task in Scene C, Octo suffers from late dropping issue, while V-GPS triples the performance on this task. This suggests that our value function rank and select the critical action to drop the sushi at the right time.
"put pepper por"
"UU sushi por"
(Qualitative visualizations) V-GPS im- proves the precision of grasping the slippery object (first row), prevents the policy's s default behavior of re- leasing
Figure 4:
object too early (second row) holding the object for too long (third row). More qualitative re- sults and vidcos can be found at https: /tap::///aaaiee /naaamttooo.oo..o githhb.io/V--e 1o/V-GPS
6.3 Can V-GPS improve various generalist policies across different embodiments?
To answer this question, we now evaluate V-GPS on top of five generalist policies Octo-small, Octo-bate Octo-small-1.5,, RT1-X, and VLe in SIMPLER simulation environments. As shown in Table 2, our method improves all five policies across multiple embodiments on average. In
7

---

Task
Octo-small-I.5
V-GPS (Ours)
Improvement
Green pepper pot
0.15
0.35
Scene A
Sweet potato on cloth
0.30
0.35
Average
0.23
0.35
+55.6%
Mushroom on cloth Mushroom in pot
0.35
0.70
Scene B
0.30
0.55
Average
0.33
0.63
+92.3%
Sushi in pot Spoon in pot
0.10
0.30
Scene C
0.25
0.40
Average
0.18
0.35
+100%
Total
Average
0.24
0.44
+82.8%
Table 1: (Real-world performance) V-GPS consistently improves the success rates of Octo the board, achieving an 82.8% improvement on average. This demonstrates that using our value function re-rank the actions can enhance the generalist policy.
Task
Octo-s
Octo-s +Ours
Octo-b
Octo-b +Ours
Octo-s.1.5
Octo-s.1.5 +Ours
RT-1-X
RE-1XX
OpenVI.A
OpenVL.A
+Ours
+Ours
Spoce on lowel
0.52
0.46
0.25
0.21
0.01
0.06
0.01
0.01
0.00
0.00
Carmt on plate
0.15
0.16
0.18
0.24
0.00
0.00
0.06
0.07
0.06
0.04
WidowX
Stack blocks
Fggplant basker
0.07
0.49
0.07
0.84
0.00
0.01
0.00
0.44
0.02
0.00
0.01
0.00
0.00
0.02
Average
0.30
0.38
0.17
0.20
0.01
0.13
0.02
0.03
0.05
0.07
Pick Can
0.31
0.38
0.29
0.24
0.05
0.43
0.19
0.29
0.72
0.82
Google
0.16
0.44
Robot
Average
0.22
0.27
0.17
0.14
0.07
0.29
0.32
0.36
0.62
0.69
Total
Average
0.27
0.34
0.17
0.18
0.02
0.18
0.12
0.14
0.24
Table 2: (SIMPLER [11] performance) V-GPS improves the success rates of all five generalist policies across
multiple embodiments using the same single value function.
particular, V-GPS improves the "put eggplant in basket" task by large margin for all policies. As shown in Figure 3, unlike the other tasks that are open-space tabletop manipulation, this eggplant task is unique because it presents a vertical height difference and an obstructing wall between the target basket and the sink. As result, any policy that is not careful would likely hit the wall and fail to complete the task. In addition, the slippery surface of the eggplant requires precise grasping locations and carefully modulated grip to execute the task effectively. Therefore, the empirical finding that V-GPS produces the biggest benefits on this eggplant task aligns with our discussion in Section 4. In addition, Octo-small-1.5 performs surprisingly poorly in SIMPLER compared to its previous Octo-small model. Nonetheless, combining V-GPS with Octo-small-1.5 can largely miti- gate its performance degradation. This might suggest that V-GPS effectively makes the policy robust against performance variations caused by changes in checkpoints or training recipes. In addition, We also show that V-GPS preferred over fine-tuning the generalist policy itself in Appendix E.
7 Discussion and Future Work
In this paper, we presented V-GPS, an approach that utilizes value functions for steering general- ist robot policies upon deployment. V-GPS does not require altering or fine-tuning the generalist policy, and can even operate effectively with black-box access to a pre-trained policy. Via thor- ough evaluation in both simulation and the real world, we show that V-GPS significantly improves the robustness and precision pre-trained policies. Despite these promising results, there are still limitations. First, while V-GPS can, principle, be combined with any policy, the policy must be able to sample actions stochastically rather than deterministically generate diverse action candi- dates. Second, since V-GPS utilizes a separate value function, it does increase the computational and time expenses during deployment. While this is not significant issue in our experiment (see Appendix G), might limit its applicability in high-frequency tasks. Future work could explore achieving a coompute-optimall balance between using the policy and querying the value function. Finally, while V-GPS can improve generalist policies on in-distribution tasks and environmental changes (e.g., table texture, height, etc.), its ability to handle completely unseen languages and ob- jects is limited, as the value function is trained on data from only two robotic embodiments. Scaling up value function architectures and using more diverse data is promising direction for future work.
8

---

Acknowledgments
We thank Zhiyuan Zhou for his help and suggestions on the implementation, and Seohong Park and Pranav Atreya for their informative discussions. We also thank Homer Walke, Karl Pertsch, and Xuanlin Li for providing details about Octo, OpenVLA, and SIMPLER. Additionally, we thank Noriaki Hirose for his feedback on the teaser figure.
This research was supported by the AI Institute, NSF FRR IIS IIS-2150826, ONR N00014-20-1-2388, and AFOSR FA9550-22-1-0272 The computation was supported by the Google TPU Research Cloud (TRC) program through their TPU donations.
References
[1] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354- 2017.
[2] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 1821- PMLR, 2021.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[4] Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint vrXiv:2011008:1 2020.
[5] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bemstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprini arXiv: 10l.:222088775888 2021.
[6] O. X.-E. Collaboration, A. O'Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee,
A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan,
A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi,
A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burrges-Limerick, B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalash- nikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F, V. Frujeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng. G. Schiavi, G. Berseth, G. Kahn, G. Yang, G. Wang. H. Su. H.-S. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Fu- ruta, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, Leal, J. Liang, J. Abou-Chakra, Kim, J. Drake, J. Peters, J. Schneider, Hsu, Bohg, J. Bingham, Wu, J. Gao, Hu, Wu, Wu. J. Sun, Luo. J. Gu, Tan, J. Oh, J. Wu. J. Lu, Yang. J. Malik, Silvério, Hejna, J. Booher, J. Tompson, J. Yang, Salvador, J. J. Lim, J. Han, K. Wang K. Rao, K. Pertsch, K. Hausman, K. Go. K. Gopalakrishnan, K. Goldberg, K. Byme, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto,
L. Fei-Fei, L. Tan, L. J. Fan, L. Ott, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip. M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. Kim, N. Kanazawa, N. Hansen, N. Heess, N. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbecl, P. Sundaresan, Q Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi,

---

R. Mart in- In-Mart'ing in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Men- donca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, Nasiriany, S. Schaal, S. Welker, S. Tian, Ramamoorthy, S. Dasari, Belkhale, S. Park, S. Nair, S. Mirchandami, T. Osa, I. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davehev, T. Z. Zhao, T. Armstrong. T. Darrell, T. Chung, V. Jain, V. Van- houcke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang. X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X-Embodiment: Robotic learning datasets and RT-X mod- cls. https:/ hoops://arxive wwagiii.o/..........oo.o wwt//ogg/////eeeeeeoe aat///ab///.././////// 08864,886448 2023.
[7] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, al. Droid: A large-scale wn-the-wild robot manipulation dataset. arXiv preprint arXiv:2403. / 12945, 2024.
[8] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-actioo models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.18815 2023.
[9] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, Luo, T. Kreiman, Y. Tan, L. Y. Chen, Sanketi, Q. Vuong, Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024.
[10] K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2024.
[11] X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kir- mani, S. Levine, J. Wu, C. Finn, H. Su, Q Vuong, and Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05944 2024.
[12] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14.668,,,, 2021.
[13] A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06643 2024.
[14] S. Han, I. Shenfeld, A. Srivastava, Y. Kim, and P. Agrawal. Value augmented sampling for language model alignment and personalization. In ICLR 2024 Workshop on Reliable and Re- sponsible Foundation Models, 2024. URL https://arxiv: http://aaxxx..oo//a///////a wtg/ 06639
[15] H. Chen, C. Lu, C. Ying, H. Su, and J. Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference Learning Repre- sentations, 2023.
[16] P. Hansen-Estruuch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies, 2023.
[17] A. Kumar, A. Singh, F. Ebert, M. Nakamoto, Y. Yang, C. Finn, and S. Levine. Pre-training for robots: Offline enables learning new tasks from a handful of trials. Proceedings of Robotics:
Science and Systems, Daegu, Republic of Korea, 2023.
[18] Q-transformer: Scalable offline reinforcement learning via autoregressive fufunctions. In 7th Annual Conference Robot Learning, 2023.
10

---

[19] H. Walke, K. Black, A. Lee, M. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruce Q. Vuong, A. He, V. Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
[20] M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and
C. Finn. Openvla: An open-source vision-language-action model. arXiv preprint, 2024.
[21] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and
S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. Science and Systems, 2022.
Robotics:
[22] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, pages 885-897, PMLR, 2020.
[23] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Ay- tar, D. Budden, M. Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv: /909. /2200, 2019.
[24] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Ju- lian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-
1: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2212.06817, 2022.
[25] S. Dasari, O. Mees, S. Zhao, M. K. Srirama, and Levine. The ingredients for robotic diffu- sion transformers. arXiv preprint arXiv:2410. 10088, 2024.
[26] H.-S. Fang, H. Fang, Z. Tang, J. Liu, J. Wang, H. Zhu, and C. Lu. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023.
[27] G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. Hatch, A. Jain, T. Yu, P. Abbeel, L. Pinto, C. Finn, and A. Gupta. Train offline, test online: A real robot learning benchmark. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9197-9203, 2023. doi: (D00/ICRA48891.202...00
[28] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07649 2020.
[29] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imita- tion. In Conference on Robot Learning, pages 879-893, PMLR, 2018.
[30] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learn- ing, pages 991-1002, PMLR, 2022.
[31] Y. Jiang, A. Gupta, 7. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. 7hu, and L. Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth Interna- tional Conference on Machine Learning, 2023.
[32] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1201-110112211 11212, 2022.

---

[33] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditioned conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315. PMLR, 2022.
[34] O. Mees, J. Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Au- lomation (ICRA), London, UK, 2023.
[35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[36] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:230.113705, 2023.
[37] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[38] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644; 2020.
[39] A. Kumar, J. Hong. A. Singh, and S. Levine. Should i run offline reinforcement learning or behavioral cloning? In International Conference on Learning Representations, 2022. URL https://openreviex www./neeunuee... Apt AP1KKT37J.
[40] C. Bhateja, D. Guo, D. Ghosh, A. Singh, M. Tomar, Q. H. Vuong, Y. Chebotar, S. Levine, and A. Kumar. Robotic offline rl from internet videos via value-function pre-training. 2023, URL oww htts//oopuII::222217725
https://aggs
swmanticscoolarr http::://pp...mmmmiicchollla
[41] J. Farebrother, J. Orbay, Q. Vuong, A. A. Taiga, Y. Chebotar, T. Xiao, A. Irpan, S. Levine, P. S. Castro, A. Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.039550 2024.
[42] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[43] E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task ag- nostic offline reinforcement learning. In Proceedings of the 6th Conference on Robot Learning (CoRL), Auckland, New Zealand, 2022.
[44] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. arXiv preprint 1812::1812.020000 2018.
[45] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019. 2019.
[46] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning, arXiv preprint arXiv:2006.0477 2020.
[47] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. 2021.
[48] A. Nair, M. Dalal, A. Gupta, and S. Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0933 2020.
[49] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage Advantage-weighted weighted regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint arXiv:1910.0011 2019.
[50] K. Li, S. Jelassi, H. Zhang. S. Kakade, M. Wattenberg, and D. Brandfonbrener. Q- probe: A lightweight approach to reward maximization for language models. arXiv preprint 2xXiv:2402.14688 2024.
12

---

[51] S. Verma, J. Fu, M. Yang, and S. Levine. Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204. 08426, 2022.
[52] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do can, not say; Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[53] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[54] Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. In arXiv, 2020. URL /tttt:://aaxiiii wwwaaaiii....opapaeeeee org/ /2004 07219.
[55] D. Brandfonbrener, W. F. Whitney, R. Ranganath, and J. Bruna. Offline rl without off-policy evaluation. arXiv preprint arXiv:2106.08909 2021.
[56] M. Nakamoto, Y. Zhai, A. Singh, M. S. Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine. Cal-QL: Calibrated offline pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36, 2024.
[57] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[58] V. Myers, A. He, K. Fang, H. Walke, P. Hansen-Estruch, C.-A. Cheng, M. Jalobeanu, A. Kolobov, A. Dragan, and S. Levine. Goal representations for instruction following: A semi-supervised language interface to control. arXiv preprint arXiv:2307.00117, 2023.
[59] N. Hirose, C. Glossop, A. Sridhar, D. Shah, O. Mees, and S. Levine Lelan: Learning a language-conditioned navigation policy from m-tthe-wild video. In Conference on Robot Learning, 2024.
[60] Y. Yang. D. Cer, A. Ahmad, M. Guo. J. Law, N. Constant, G. H. Abrego, S. Yuan, C. Tar, Y.-H. Sung, et al. Multilingual Universal Sentence Encoder for Semantic Retrieval, July 2019.
[61] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. FiLM: Visual Reasoning with General Conditioning Layer, Dec. 2017.
[62] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenhorn, Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- for image recognition at scale. arXiv preprint arXiv:2010. 19999, 2020.
[63] S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh. Prismatic vims: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07886 07865 2024.
[64] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation eπrо in actor- critic methods. In International Conference on Machine Learning (ICML), pages 1587-1596, 2018.
13

---

A V-GPS with IQL
While we used Cal-QL train our value function in the main paper, one can use any offline RL or policy evaluation algorithm to fit Qs. To demonstrate this, further show the results with IQL value function in this section. Formally, IQL. trains a Q function (s a, 1) and a state-value function (s, with the following objectives, where LI is the expectile loss L5 (u) = |T 1(u < 0)ju², and Q7 represents the target Q-network, delayed soft average of the current Q-network:
Lv(v)= ((aa,.)> [L (Qp(s, a, Vs(s
(3)
(2) (( AND (r(s,a,1)+*W)) 7V= (5',1) Qo Qo(s,
(4)
We evaluated V-GPS using the IQL value function in SIMPLER. As shown in Table 3, using an IQL value function for V-GPS also effective for improving the success rates of all five generalist policies across multiple embodiments.
Task
Octo-s
Octo-s +Ours
Octo-b
Octo-h +Ours
Octo-s-1.5
Octo-s-1.5 +Ours
RTI-X RTI-X +Ours
OpenVL.
OpenVL.
+Ours
Spoon
0.25
0.16
0.07
Carrot on plate
0.15
0.18
0.18
0.20
0.00
0.00
0.06
0.07
0.06
0.06
WidowX
Stack blocks
0.07
0.09
0.00
0.00
0.00
0.02
0.00
0.00
0.00
0.00
Eggplant basket
0.49
0.59
0.28
0.37
0.01
0.07
0.01
0.01
0.14
0.54
Average
0.34
0.17
0.18
0.04
0,02
0.03
0.05
0.15
Pick Can
0.31
0.30
0.29
0.30
0.05
0.47
0.19
0.32
0.72
0.78
Google
0.17
0.04
Robot
Average
0.22
0.23
0.17
0.18
0.07
0.18
0.32
0.37
0.62
0.61
Total
Average
0.27
0.31
0.17
0.18
0.02
0.14
0.12
0.15
0.24
0.31
Table 3: (V-GPS with IQL) Using an IQL value function for V-GPS is also effective for improving the success rates of all five generalist policies across multiple embodiments.
B V-GPS Implementation Details
In this section, we provide the implementation details V-GPS for value function pre-training, and test-time action re-ranking. The hyperparameters listed in Table 4.
B.1 Value Function Training
Our language-conditioned function (s a, uses a ResNet-34 image encoder with FiLM lan- guage conditioning shown in Figure 5. The image observation is first passed through the ResNet- 34 encoder, while the language instruction, processed by frozen MUSE encoder, is applied to every block in ResNet using FiL.M conditioning. The 7-dimensional actions are concatenated with the fi- nal output from the ResNet, then passed through two 256-unit hidden layers, and finally, a scalar Q value predicted, For both Cal-QL and IQL, we trained the value function using a mixture of the Bridge and Fractal datasets with batch size of 512 on a single v4-8 TPU VM. We used a discount factor y 0.98, clipped double Q-learning [64], and shifted reward values of 0 and -1 We signed the final 3 steps of each trajectory as positive rewards O, and the rest as negative rewards -1. We use the Adam optimizer with learning rate of 3e-4. During training, we augment the image observations with random cropping and color jitter. The Cal-QL value function is trained using an alpha of a = 5.0 for IM steps. The IQL value function is trained using an expectile of T = 0.7 for 200K steps.
B.2 Test-Time Action Re-Ranking
During test-time, we sample action proposals from the base policy æ at each time step, and then re-rank the proposed actions using the Q function with Equation 2. In the real-world evaluations with the Cal-QL value function, we used K 50 and we found selecting the action greedily by setting 3 - leads to satisfactory results. In simulation, we swept over K {10, 50) and B {0, 0.1, 1.0} and report the best result for each policy.
14

---

MLP layers
ResNet34
Qe
FiLM
Camera Observation
Language Prompt
MUSE (Frozen)
"put sushi in pot"
Action Vector a
Figure 5: (Model Architecture.) Our value function ResNet-34 image encoder with FiLM language conditioning.
Cal-QL
5.0
IQL expectile т discount factor learning rate
0.7
0.98
3e-4
positive reward steps H
3
number of actions to sample K
{10, 50}
softmax temperature /
0.1, 1.0)
Table 4: (V-GPS hyperparameters)
C Baseline Implementation Details
For (small base, small-1.5), we used the publicly released checkpoints from https: For RTI-X, we used the publicly released JAX check- point from https://github. http://goooeeeeeeeiooooop.p..eae used their public checkpoint openvla-v01-76 from https://bugginnfacce.omm cotpeceviac//peelallaa.aacc openvla-v01-7b. To combine OpenVLA with method, had to iterate the forward pass K times at each time step to sample multiple actions, since it does not yet support batch infer- ence. Our real-world evaluation is implemented on top of the evaluation codes provided from https://githir com/octoomodels//ooo and the simulated evaluation based on https: // //github.. com/ wwm/simpler--eeeeemmmmemeeeeeeee www.eerreenriiimmeieeee.
For OpenVLA,
D Experimental Setup
(Real world) We conducted our real-world evaluations on 6 tasks across different scenes as shown Figure We provide the language instructions we used for each task Table We conduct 20 trials per task and report the average success rates in Table 1. We randomize the configurations and orientations of each object for each trial.
(SIMPLER) We conducted the simulated evaluations on 6 tasks in the SIMPLER environment, including tasks on the WidowX robot platform and 2 on the Google Robot platform as shown in Figure 3. We used the default language instructions for each task shown in Table For RT1-X and Octo- (smaa, base, sma)l-1.5), conducted 100 trials for each of three different random seeds. For Open VLA, conducted 50 trials per task due to its slower inference speed, as it does not yet support batch inference. The average success rates are reported in Table 2.
15

---

Language Instructions
Language Instructions
put the green pepper in the pot put the sweet potato on the cloth
put the spoon on the towel
Scene A
put carrot on plate
Widow
stack the green block the yellow block put eggplant into yellow basket
put the mushroom on the cloth put the mushroom the pot
Scene B
Google Robot
pick coke can
put the sushi in the pot put the green spoon the pot
Scene C
move {objectl} near {object2)
Table 5: (Real-world scenes and tasks) We eval- uate V-GPS tasks across different real-world scenes.
Table 6: (SIMPLER scenes and tasks) We eval- V-GPS in 6 tasks
different embodi- ments in SIMPLER environment.
(Further Details) Our goal is to use a value function to improve the pre-trained generalist policies, and we do not assume access to any additional data beyond what the generalist policies were pre- trained on, All the generalist policies are pre-trained on the OXE dataset, which fully open-sourced and public. Our experiments are designed study the following cases:
SIMPLER: clearly in-distribution, with the same combination of objects in the same scene
Scene A: seen tabletop, with different combinations of objects
Scene B: seen tabletop with a lower table height of I inch than usual (to distribution shift), and with different combinations of objects
Scene C: unseen tabletop, with different combinations of objects
Given that V-GPS improves upon the pre- rre-trained policy in SIMPLER and in Scenes A, B, and C, method has proven to be effective in both in-distribution cases and domain shifts, including changes in table height and unseen tablctops or backgrounds.
E Additional Comparisons to Policy Fine-Tuning Approaches
A common question is: Why is re-ranking with Q-values preferred over fine-tuning generalist poli- cies? There are several reasons why re-ranking with Q-values might be more effective. First, large generalist models might be closed- and available via API only (such as RT-2-X), which hin- ders fine-tuning. Second, as these models increase in size, fine-tuning becomes increasingly com- putationally expensive. Fine-tuning Open VLA, for instance, requires 8 x A100s.
Furthermore, since the generalist policies are pre- pre-trained on the OXE dataset, which already contains the datasets for the downstream tasks we studying (the Bridge dataset and the Fractal dataset), further fine-tuning the generalist policy on these individual datasets does not necessarily improve performance. To demonstrate this, we conducted additional experiments to compare V-GPS to three different policies:
Octo-finetune: Octo-small model pre-trained on OXE + fine-tuned on bridge dataset
2. Octo-scratch: Octo-small model trained on bridge dataset from scratch
3. Resnet-DP: Diffusion Policy [37] with a Resnet34 encoder, which a state-of-the-arr ar- chitecture for imitation learning, trained on bridge dataset from scratch
As shown in Table 7, neither fine-tuning the generalist policy nor training a policy solely on single dataset improves performance over Octo, and V-GPS the only method that achieves better performance than the generalist policy. This clearly highlights the benefit of re-ranking with Q- values preferred over fine-tuning the generalist policies.
F Ablation Over the Size of Dataset
To investigate how the size of the offline dataset impacts performance, we trained IQLvalue func- tions smaller datasets Bridge Dataset subsampled to 50% and 10% and evaluated the perfor- mance SIMPLER's ' eggplant task. As shown in Table reducing the dataset size to 50% still
16

---

Task
Octo-small
Octo-finetuuee finetuned
Octo-scratch
Resnet-Di
Ours (IQL)
Ours (Cal QL)
Spoon on towel
0.52
0.28
0.01
0.05
0.50
0.46
Carrot on Plate
0.15
0.12
0.01
0.18
0.15
Stack blocks
0.07
0.06
0.00
0.06
0.09
0.07
Eggplant basket
0.49
0.41
0.00
0.37
0.59
0.84
Average
0.30
0.22
0.01
0.12
0.34
0.38
Table 7: (Comparison to fine-tuning generalist policies or training the policy from scratch.) V-GPS is the only method that achieves better performance than the generalist policy.
achieved the same improvement, and reducing it 10% resulted in slightly worse performance, but it still improved over Octo-small. This shows that even value function trained on small amounts of data can be effective in guiding generalist policies at test time.
Model
Success Rate
Octo-small (baseline)
0.49
Ours- 100%
0.59
Ours-50%
0.59
Ours-10%
0.55
Table 8: (Ablation over the size of datasets.) Even a value function trained on small amounts of data can be effective in guiding generalist policies test time.
G Analysis of the Overhead in Inference Time
We conducted an analysis of the inference time per time step using the Octo-small model. As shown in Table 9, using = 10 results in 1.28 times slower inference, and using K = 50 results in 1.59 times slower inference compared to the baseline, which we did not find to a significant slowdown in practice. The analysis conducted on the inference machine that we used for real- world evaluation. Furthermore, this level of overhead will not be an issue in real-world WidowX tasks. This is because the WidowX environment typically uses blocking control with 0.2-second interval, meaning actions are predicted every 0.2 seconds [19, 9].
Method
Inference time Overhead
Octo-small
0.0752
1.00
Ours K 10
0.0963
1.28
Ours K 30
0.1096
1.46
Ours K 50
0.1196
1.59
Ours K 100
0.1596
2.12
Table (Analysis of the overhead inference time.)
H Ablation Over the Number of Actions K
We conducted an ablation study on K using the WidowX eggplant task and the Google Robot pick-coke task. As shown in Table 10, we found that the IQL value function performs best with K 10, and increasing K leads to the exploitation of the value function, resulting in performance degradation. In contrast, the Cal-QL value function is more robust to K, and using larger value improve performance.
Task
Eggplant IQL.
Pick Coke
Offline RI. method
Cal-QL.
IQL.
Cal-QL.
Octo-small (baseline)
0.49
0.49
0.31
0).11
Ours K 10
0.59
0.77
0.30
0.38
Ours =30 -30
0.47
0.81
0.37
0.38
Ours 50
0.42
0.84
0.31
0.38
Ours = 100
0.35
0.63
0.37
0.36
Table 10: (Ablation over K.) Cal-QL. is robust to K than IQL.
17

---

I Comparisons to the Actors of Cal-QL & IQL
We evaluated the IQL and Cal-QL actors in the SIMPLER tasks, but shown in Table 11, we found that they were unable to successfully complete them, consistently achieving a zero success rate. Interestingly, the common failure case for these actors was their inability to learn the gripper's proper opening and closing actions, consistently outputting the open open-gripper action. We also tried roll the actors in the real-world setup and observed the same issue. This is common problem when training Gaussian (or Tanh-squashed Gaussian) actor on manipulation datasets such as Bridge Data, since the modes of the gripper's open/close distribution are too extreme (0 for close and for open), and the actor is too simple to model them effectively. This highlights the benefit of our method, which combines the value function with the pre-trained generalist policies, allowing us to enjoy the advantages of both the critic and the ofatc-of-the-ar expressive imitation learning policies.
Task
IQL actor
Cal-QL actor
Spoon on towel
0.00
0.00
Eggplant basket
0.00
0.00
Table 11: (Comparisons to actors Cal-QL IQL.) The actors of Cal QL and IQL consistently achieve a zero success rate. This highlights the benefit of our method, which combines the value function (critic) with the pre-trained generalist policies.
J Comparison to Using a Random Policy or Random Action Selection
To prove that both parts of V-GPS - the value function and the generalist policy are the specific reasons for improvement, we compared the following two methods on the eggplant task:
1. Random-selecting: Octo-small policy + randomly selecting actions
2. Random-policy: Random policy V-GPS value function
The results are shown in Table 12. As expected, Random-selecting performs similarly to the naive Octo-small model, showing no improvement. This highlights the benefit of using the value function for action selection. Furthermore, Random-policy fails to perform the task and consistently achieves a zero success rate. This is also expected, as if the policy generates nonsensical action proposals, then using the value function will help. In short, both parts of V-GPS the pre-trained policy and the value function are crucial for improvement, and combining them both together leads the best performance.
Method
Success Rate
Octo-small (baseline)
0.49
Random-se -eleccinn
0.49
Random-polle -oolicy
0.00
V-GPS (ours)
0.84
Table 12: (Comparison to using a random policy or selecting the actions randomly.) Using a random policy random action selection does not improve performance over the generalist policy, demonstrating that V-GPS is the specific reason for the improvement.
Model
Num Params
K Details of the Network Size
Q Network (Ours)
25.6M
Octo-small
27M
We provide the number of parameters for our value function and the generalist policies in Ta- ble 13. Our value function is a ResNet-34- based network with 25.6 million parameters. This smaller than all the generalist policies we studied, specifically 27% the size of the Octo-base model and 0.3% the size of Open- VLA.
Octo-base
93M
Open
7B
RTI-X
35M
Table 13: (Network size.) Our critic network is smaller than all the generalist policies we studied, specifically 27% the size of the Octo-base model and 0.3% the size of VLA.
18


Reviews:
[{"review_id": "WCXShyN1Jc", "reviewer": "Reviewer_Gdoc", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper present, V-GPS, a value-based reranking method to improve the quality of the pretrained generalist policy. This approach is to fix the policy failures due to imprecise manipulation at the testing time. The proposed approach leverages offline RL to learn the Q-value of the test domain. Then using the Q-value to rerank the top K actions. The experiments showed big improvements in simulated and real robot tasks.", "summary_of_recommendation": "The overall idea is interesting and can have good impact for the generalist policy deployment. As discussed in weaknesses and questions, some questions still need to be answered to support the claim.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- To better support the claim of policy improvements, it will be helpful if the authors can split the experiment into different cases such as in-distribution, distribution shifts, and different embodiments to answer when and how the proposed method can help the generalist policies. \n- The assumption about a good offline dataset may not be realistic, it will be helpful to see an experiment on how the size of the offline dataset could impact the performance improvements.\n- The current experiments show good improvements in WindowX settings. However, it doesn’t seem to help Google robot that much, especially negative impact in the OpenVLA case. Why is there a gap between different embodiments?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u16SHUzXEM", "reviewer": "Reviewer_AHDo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a fairly unique approach (V-GPS) to improving pre-trained foundation models for manipulation. The method first trains a value function with IQL. V-GPS then ranks sampled actions from a pre-trained foundation model by the estimated q value and then employs those actions. The paper shows that this procedure improves the performance of the foundation model, without having to pass policy gradients through it.", "summary_of_recommendation": "The paper is novel and well written, providing ablations will strengthen it further.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "What was the size of the Critic Q network used? Does scaling this model lead to better results? If the Q network is much smaller than the pre-trained foundation model, then this enables researchers with lesser compute to train these models instead of fine-tuning much larger foundational models. This fact can strengthen the paper if explicitly mentioned.\n\nWhat exactly was the overhead added to the inference time in VGPA compared to the regular Octo/VLA models? This is mentioned as a limitation, but knowing the exact number might be useful. Please club this with the ablation over K mentioned above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZUdqcc12x", "reviewer": "Reviewer_jKak", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Generalist robot policies perform sometimes have issues with closing the gripper at the right time, releasing the gripper on time, etc. The authors train a value function to re-rank multiple sampled actions and execute the action that the value function thinks is likely to succeed. They do this by training a language-conditioned value function that they train. This value function is agnostic to the weights of the full policy and can be applied to any robot policy. It does so by re-ranking actions and choosing the best one.", "summary_of_recommendation": "Although the paper has good real-world deployments, I would recommend this paper in its current state for a weak reject due to the lack of comparisons and limited discussion of any other kinds of methods. The idea of test-time action re-ranking is interesting. However, by not evaluating any potential alternatives, this work does not sufficiently show that it is better than other approaches. I am open to discussion on these points and increasing my score.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- The authors' real world experiments were on the Bridge data. Was Bridge included in the training data for the generalist policies used? Does the SIMPLER simulation replicate environments that are found in the training data used in the generalist policies?\n- What is the relationship of this kind of work to methods like LOReL's visual model predictive control [1] which samples M different actions, predicts a reward for each action sequence with a language-conditioned visual reward function, then executes the best one which they find by using the Cross Entropy Method (CEM). In your case, it appears that a large generalist model provides a few initial actions proposals instead, then a similar approach to LOReL ranks the action proposals.\n- The authors use the term \"generalist value functions\". In the real-world experiments, only Bridge data is used, and in the SIMPLER simulation experiments a mix of Bridge and Fractal data is used. Are two embodiments sufficient to be a generalist value function? I did not put this question under weaknesses as I do not think the robotics community has converged to what it means to truly be a \"generalist\" policy, so I am curious what the authors think.\n- More questions are in the weaknesses section above.\n\n[1] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. CoRL 2021.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths
- Experiments show significant improvements over baselines
- Can be combined with different pre-trained policies
- Innovative idea

Weaknesses
- Unclear in which cases the method is helpful
- Reported results don't match with original papers
- Large offline dataset might be unrealistic
- Some missing details
- Experiments insufficient to prove that V-GPS was the specific reason for improvement

# After discussion phase:

The authors successfully managed to address the most critical concerns of the reviewers. They are now all voting "weak accept" (but some couldn't update their original scores). All reviewers find the method interesting and novel. The improvement over the SOTA is not massive (also see below), but the idea behind the paper definitely deserves to be published as opens avenues for future research without requiring massive compute.

During the reviewer and AC discussion phase one more issue popped up that needs to be fixed: Reporting relative improvement is probably not the best option for this paper. This seems to be totally skewed by octo-s-1.5, where performance increases from 1% to 7% leads to 700% relative improvements (marginally improving a bad generalist policy) that brings the overall relative improvement to a high number, while absolute improvement is a lot more modest.
Decision: Accept