Vision Language Models are In-Context Value Learners
Authors: Yecheng Jason Ma
Joey Hejna
Chuyuan Fu
Dhruv Shah
Jacky Liang
Zhuo Xu
Sean Kirmani
Peng Xu
Danny Driess
Ted Xiao
Osbert Bastani
Dinesh Jayaraman
Wenhao Yu
Tingnan Zhang
Dorsa Sadigh
Fei Xia
Venue: ICLR 2025 Spotlight (2025)
Keywords: robot learning
vision-language model
value estimation
manipulation
Primary area: applications to robotics, autonomy, planning

Abstract: Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estimator, or temporal value function, across different tasks and domains requires both a large amount of diverse data and methods which can scale and generalize. To address these challenges, we present Generative Value Learning (GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Naively asking a VLM to predict values for a video sequence performs poorly due to the strong temporal correlation between successive frames. Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks across diverse robot platforms, including challenging bimanual manipulation tasks. Furthermore, we demonstrate that GVL permits flexible multi-modal in-context learning via examples from heterogeneous tasks and embodiments, such as human videos. The generality of GVL enables various downstream applications pertinent to visuomotor policy learning, including dataset filtering, success detection, and value-weighted regression -- all without any model training or finetuning.

Published as a conference paper at ICLR 2025
VISION LANGUAGE MODELS ARE IN-CONTEXT VALUE LEARNERS
Yecheng Jason Ma Joey Hejna' 13, Ayzaan Wahid', Chuyuan Fu', Dhruv Shah JJacky Liang', Zhuo Xu' Sean Kirmani', Peng Xu', Danny Driess', Ted Xiao', JJnathan Tompson', Oshert Bastani², Dinesh Jayaraman', Wenhao Yu', Tingnan Zhang', Dorsa Sadigh', Fei Xia 'Google DeepMind, Uniiversity of Pennsylvania, Stanford University Correspond to: jasonyma seas. upenn edu, xiafei xiafei@googll @googgel com Website and Interactive Demo: http::/generativve value leeee-eerrnnnng gittinn.ggggg............. io
ABSTRACT
Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estima- tor, or temporal value function, across different tasks and domains requires both large amount of diverse data and methods which can scale and generalize. To address these challenges, we present Generative Value Learning (GVL), a universal value function estimator that leverages the world knowledge embedded in vision- language models (VLMs) to predict task progress. Naively asking VLM to predict values for a video sequence performs poorly due to the strong temporal correlation between successive frames. Instead, GVL poses value estimation temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMS more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions. Without any robot task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks across diverse robot platforms, including challenging bimanual manipulation tasks. Furthermore, we demonstrate that GVL permits flexible multi-modal in-context learning via examples from het- erogeneous tasks and embodiments, such as human videos. The generality of GVL enables various downstream applications pertinent to visuomotor policy learning, including dataset filtering, success detection, and advintage-weighted regression all without any model training or finetuning.
INTRODUCTION
Predicting temporal progress from visual trajectories is important task for embodied agents that interact with the physical world. A robot capable of generalizable progress estimation can in principle discern desirable and undesirable behaviors learn visuomotor skills in new environments. This is most often studied reinforcement learning literature (Schaul et al., 2015), where progress estimation equivalent universal value learning under specific choices of reward function. However, universal value estimation comes with a number of key challenges: (1) broad generalization to new tasks and scenes, (2) the ability accurately estimate state in partially observed environments, and (3) temporal consistency (i.e. satisfying the Bellman equation) over long horizons. Most existing methods trained on relatively small amounts of vision-only data (Chen et al., 2021; Ma et al., 2022: Ahn et al., 2022) lack the semantic, spatial, and temporal understanding needed to ground task progress in the space-time manifold of video, preventing generalization. Moreover, they often reason over single frames, inducing a high-degree of uncertainty in partially observed environments which in turn can effect the consistency predictions for poorly estimated states. However, these challenges not insurmountable: modern vision language models (VLMs) exhibit marked generalization and reasoning capabilities, potentially making them useful for value estimation.
Though not often considered as candidates for value estimation, VLMs excel at its aforementioned core challenges. First, state-of- the-art VLMs have exhibited strong spatial reasoning and temporal understanding capabilities across various vision tasks (Nag al., 2022; Chen al., 2024; Hong et al.,

---

Published as a conference paper at ICLR 2025
"fold the black dress"
Step
Step
Step
Step
Step
Input Shuffling
Vision Language Model
Autoregressive Prediction
100%
79%
24%
0%
52%
Generative Value Learning
Applications
Dataset Filtering
Success Detection
Policy Learning
predictions
Few-shot predictions
50 OXE datasets
challenging bimanual tasks
Figure I: GVL auto-regressively predicts task completion percentage shuffled frames, enabling impressive in-context value learning. GVL can effectively zero-shot and few-shot predict task progress diverse and challenging real-world tasks; these capabilitics enable expansive set of downstream applications, including dataset filtering, success detection, and policy learning.
2023; Gao et al., 2024), allowing them to generalize to novel scenarios. Second, large transformer- based VLMs have the requisite context window (GeminiTeam et al., 2024) to reason over large amounts of historical information to accurately estimate state from observation sequences when predicting task progress. Finally, VL.Ms make predictions auto-regressivelly, meaning they commit to their own outputs inputs for subsequent predictions, imposing consistency constraints on long generations. For example, a VLM is unlikely to estimate that task is 50% completed it already has 50% completion prediction in context. However, how exactly VLM should be used predict values is unclear. Empirically, we find that simply placing a video in-context and prompting the model to return progress predictions for each frame fails our analysis suggests strong temporal correlations between successive frames often cause VLMs to produce uninformative monotonic values that disregard the actual quality of the trajectory and differences between frames (Section 4) and different approach is needed.
To effectively leverage the broad knowledge of VLMs, we introduce Generative Value Learn- ing (GVL), a universal value estimation method enabled by long-context VLMs, which crucially operates over shuffled frames. At its core, GVL asks frozen state-of-the-art VLMs, such as (GeminiTeam et al., 2024), to auto-regressively predict the completion percent- age of a task specified in natural language for sequence of shuffled input video frames; see Fig. 1. Perhaps surprisingly we find that simply shuffling the frames of the input video effectively overcomes the strong implicit temporal bias found in video, enabling VLMS to generate meaningful values. While GVL is capable of generating values in zero-shot manner. GVL scales with examples via multi-modal modal in-context learning. Providing more examples visual "unshuffling" in context increases performance, irrespective of the target embodiment. For example, human videos can improve GVL's performance on predicting robot task progress.
Gemini-1.5-22
find that the performance of
To facilitate large-scale value prediction evaluation, additionally introduce new evaluation metric, Value-Order Correlation (VOC), measuring how well predicted values correlate with the ground-truth timestep order in expert videos; we will show, VOC is also useful metric for
2

---

Published as a conference paper at ICLR 2025
measuring dataset and trajectory quality, which allows GVL to be used for applications beyond value-based policy learning such as data quality estimation and success detection. We first evaluate GVL's value prediction quality with VOC large suite of real-world robotics datasets, spanning 51 datasets, 20 embodiments, and more than 300 tasks. This includes 50 datasets from Open-X (OXE) dataset (Padalkar et al., 2023) in addition to our own bimanual manipulation dataset containing 250 challenging real-world tasks on an ALOHA platform (Zhao et al., 2023), which are considerably longer horizon and more fine-graines than those in the OXE dataset. In aggregate, GVL exhibits strong zero-shot value prediction capabilities with highly positive VOC scores on most datasets; its performance further improves with various types of multi-modal in-context examples. Using GVL, we demonstrate scalable foundation model supervision for robot learning at various data abstraction levels. Specifically, GVL can help measure dataset quality in OXE. Second, it can be used for success detection, enabling imitation learning on mixed-quality datasets. Finally, the raw value estimates from GVL can be used for advantage-weighte regression for real-world offline reinforcement learning (Peters & Schaal, 2007; Peng et al., 2019).
In summary, our contributions are
1. Generative Value Learning (GVL), a universal value prediction framework via VLM in- context autoregressive value estimation on shuffled video frames.
2. An extensive evaluation real- weal-world datasets demonstrating GVL's zero-shot scalability and multi-modal in-context learning capabilities.
3. Demonstration that GVL can be used in downstream applications including dataset quality estimation, success detection, and adeintege-weiighed regression for real-world control.
2 RELATED WORK
Reward and value foundation models. Several works have tried to learn transferable reward and value functions from diverse data. Early works learned models using robot (Sermanet et al., 2016) or even human videos with diseriminators (Chen et al., 2021), I), contrastive learning (Baumli al., 2023) or offline RL (Ma et al., 2022; 2023a; Bhateja et al., 2023) to guide manipulation tasks. With the advent of recent language and vision foundation models, several works have integrated them into vari- ous robotic applications such as semantic planning (Ahn et al., 2022; Huang et al., 2023b; Singh et al., 2023; Zhang al., 2023; Ding et al., 2023), imitation learning (Brohan al., 2023; Szot et al., 2023), and symbolic programming (Tang et al., 2023; Liang et al., 2023; Singh et al., 2023; Wang et al., 2023; Huang et al., 2023a; Liu al., 2023; Silver et al., 2023; Ding et al., 2023; Lin et al., 2023; Xie et al., 2023b). Most related to our work, LLMs and VLMs have been used as reward models. Kwon et al. (2023b); Mahmoudich et al. (2022) use language models to provide reward values for RL agents, while Klissarov al. (2023); Wang et al. (2024); Kwon al. (2023b) use them to provide preference feed- back. Ma et al. (2023b); Yu et al. (2023); Xie et al. (2023a) even have LLMs generate their code. These works use only the language capabilities of foundation models. More recent works directly VLMMs as zero-shot reward models (Rocamonde et al., 2023) or success detectors (Du et al., 2023; Guan et al. 2024). Critically, in these works the VLM acts only as an (often sparse) reward function which predicts success, and not value function that predicts task progress. Though some works use chain- of-thought prompting (Venuto et al., 2024) or active learning (Kwon et al., 2023a), they generally do not make use of the autoregressive, long-contens context, or in-context learning capabilities of state-of-an art VLMs. As consequence, they often evalaute reward prediction only on simple and simulated tasks. To our knowledge, we are the first to demonstrate that VI.Ms are capable of generalizable per-frame value estimation on real world tasks which can be used for downstream tasks like dataset selection.
In-context learning for robotics. In-context learning has been explored in the robot learning literature, primarily focusing on action generation (Duan et al., 2017: Finn et al., 2017; Dasari & Gupta, 2021; Xu et al., 2022; Di Palo & Johns, 2024; Liang et al., 2024; Fu et al., 2024). However, all these prior works require explicit, and often extensive training, on their robot tasks in order to realize in-context learning capabilities, and generalization is achieved only on narrow distribution of tasks. In contrast, we demonstrate that visual value estimation already enjoys flexible multi-modal in-context learning from pre-trained VLMs without any robot specific fine-tuning.
3

---

Published as a conference paper at ICLR 2025
3 GENERATIVE VALUE LEARNING
In this section, we introduce Generative Value Learning, GVL. At a high level. GVL frames value estimation as an autoregressive next-token prediction problem in which VLM is tasked with outputting the task progress for a batch of shuffled trajectory frames.
Problem setup. We model robotics tasks goal conditiond partially observed Markov decision processes (Puterman, 2014): M(Ø)= == (0, A, R, P,T,, jr, G) with observation space O, action space A, reward function R. transition function P, task horizon T,, initial state distribution su(o), and goal space G that specifies the task semantically. Conditioned on a task an agent π O - A aims to maximizes its value function, or the expected cumulative reward over the task horizon, V (o1:g) = (y.x,.(((or;g) + + r(or: 9)]. However, reward and value functions can be difficult to define for robotics applications given their heterogeneity. Given this, a popular universal notion of value task progress (Sermanet et al., 2016; 2018; 8; Eysenbach et al., 2020; Tian et al., 2020; Lee et al., 2021). This kind of temporal value function maps an observation and goal specification to a real number between O and 1: V: O x G - [0, IJ. where initial observations of the environment have value 0 and goal- satisfying observations have value 1. Under this definition, an expert trajectory T = (0I..... oT) ~ TE. has value function V* (ot: 9) 4. In this work, our goal is to obtain such temporal value function V that can predict such task progress v1,... By for each frame of video 01,-..,,0 Or.
Though seek to leverage priors imbued in large foundation models, shown in Section 4 simply prompting a VLM with video frames fails to produce meaningful estimates. To make VLMs amenable to value prediction, we propose three key components that comprise the GVL method: 1) autoregressive value prediction, 2) input observation shuffling, and 3) in-context value learning.
1. Autoregressive value prediction. Traditionally, value functions V(+): O self-consistent by enforcing the bellman equation
R. are trained to be
(1) When parameterizing value function as a feed-forward neural network, this is typically done by minimizing the mean mean--quared error of the equality above. As values for different observations within the same trajectory consistent even if we query it with only a single observation. VLMs on the other hand are not inherently trained with any consistency objective. Thus, if we independently query VLM with different observations from the same trajectory it is likely to produce inconsistent values. Our insight that providing the entire trajectory as input instead of just single observation offers VLMs greater opportunity to generate self-consistent value estimates. Concretely, given language description of the task lunk we ask the VLM to ruto-regressively generate values given the entire video as context:
(o.) R(o:) + E,,r [V(o+1)]-
related via the bellman equation, the resulting value function remains
2 VLM(o,,.... ,0;; UI,..., Ut lunk), Vt E [2,
(2)
We abbreviate this auto-regressive prediction process as ..,.... DT = VLM(O)..... OT; lusk). This simple mechanism allows the VLM to attend all previous predictions and frames when making the next value prediction, enabling it to produce globally consistent estimates over long-horizon sequences without needing to be trained like classical feed-forward value functions. Though this design choice enables VI.Ms to produce consistent values, it doesn't 't necessitate that the values are meaningful. Naively prompting VL.M in this manner tends to produce linear, monotonic value functions for every single video, regardless of optimality.
2. Input observation shufffing. Empirically we find that when presented a choronological sequence of frames VLMs discover the short-cut solution of outputting monotonically increasing values, often ignoring the task description or the actual quality of the trajectory. One hypothesis is that as VLMs trained ordered video frames for captioning and question answering, the chronology itself is cue for downstream tasks unrelated to value prediction. As a consequence, model naive prompting results in unfaithful low-quality value predictions. To break this temporal bias, we propose randomly shuffling the input frames. In this manner, GVL forces the VLM to pay attention to each individual frame and output faithful value predictions using all information provided in context. Concretely, GVL prompts a VLM as:
Vi.... Cn.. = VLM Opi lank, 01), where (i...., = permate(I,.....
where the permute operator randomly shuffles the temporal indicies. Note however, that we cannot shuffle every frame. If we do so, then the arrow of time in the original video can be ambiguous
4

---

Published as a conference paper at ICLR 2025
Le., in many cases, the reverse video is also physically plausible, making is the ground- ground-truth order impossible to predict. Thus, as in the above equation we condition the VLM on the first input frame allow it to use the first observation as anchor point for all other shuffled frames.
3. In-context value learning. While auto-regressive prediction and shuffling are enough to obtain good performance, GVL can perform even better by leveraging the appealing properties of VLMs. Notably, large models often exhibit in-context learning, where tasks can be learned by simply providing examples (Brown, 2020). This enables flexible and versatile in context value learning, by which GVL's predictions can steadily improve by providing examples at test time without any model finettuning In particular, we can simply prepend shuffled videos and their ground-trute task progress as in in-context examples to boost the value prediction quality via few-shot learning:
Vig = VLM (0i,-.., O'. lank permute ((01, v1), (02, v2),....,,.))) UM)))
(4)
As we show in Section 4, GVL benefits from flexible forms of in-context examples, including videos from unrelated tasks and even humans. Though GVL zero-shot is already effective across broad range of tasks and robots, in-context learning can still realize substantial improvement on the most difficult bimanual dexterous tasks.
EXPERIMENTS
We conduct large scale experiments assessing GVL's is value prediction generalization and in-context learning capabilities. Specifically, we study the following questions:
1. Can GVL produce zero-shot value predictions for broad range of tasks and embodiments?
2. Can GVL improve from in-context Icarning?
3. Can GVL be used for other downstream robot learning applications?
In all our experiments, we use Gemini- -1. Prro (GeminiTeam et al., 2024) as the backbone VLM for GVL; we ablate this model choice and find GVL effective with other VLMs well. After thorough study GVL's 's value prediction capabilities, we study several downstream applications in visuomotor policy learning, aiming to improve data quality at dataset, trajectory, and individual transition levels.
Evaluation metric. Our goal is to evaluate GVL value estimation scale as many robot datasets as possible, holistically testing its generalization capabilities and understanding its limitations. This makes it difficult use traditional evaluation metrics for value functions, such as observing downstream learned policy performance, as they require value functions that are specifically trained or finetuned for individual tasks and embodiments. This quickly becomes very expensive for universal value functions that are intended for use across large set of diverse real-world tasks and robots, many of which the practitioner may not have access to. Prior works on large-scale value learning have resorted to visually observing the smoothness of the value curve on expert trajectories qualitative "eye-test" for model generalization (Ma et al., 2022; 2023a; Karamcheti et al., 2023), but such evaluation is conducted only few selected videos. We formalize and scale up this intuitive approach and introduce a lightweight, yet predictive method for evaluating value models: Value- Ordder Correlation (VOC). This metric computes the rank correlation between the predicted values and the chronological order of the input expert video:
vOC = rank Paank-coree correlorroo.oo.o in (argsort(u)..... ( a arange(T)):
(5)
VOC ranges from -1 to where indicates that the two orderings are perfectly aligned. Expert quality demonstrations, by construction, have values that monotonically increase with time, and thus good value model should have high VOC scores when evaluated on expert videos. On the other hand, fixing good value model, low-quality trajectories should have low VOC scores. This because sub-optimal trajectories often contain high repetition of visually similar frames due to the presence of redundant, re-attempt actions poorly-placed cameras. As such, the values along the trajectories should not be monotonic, resulting in low correlation with the ground -tuth timestep order. As will show in our experiments, this value rank correlation metric has strong predictive power for the quality of the values as well as downstream policy learning performance, validating its usefulness as standalone evaluation metric for value predictions.
5

---

Published as a conference paper at ICLR 2025
OXE Value-Order Correlation Histogrem [Text Goal)
OXE Value-Order Correlation Histogram (mmage Goal!
GM.
2.5
CM.
Dersity
2.0
Dersity
2.0
1.5
1.0
1.0
0.3
0.5
-1.00
-0.75
-0.50
-0.25
0.25
1.00
-0.50
-0.25
0.00
0.25
0.75
2000
Correlation
Value
Correlation Value
Figure 2: Zero-shot value predictions on OXE datasets. Left: GVL significantly outperforms LIV on datasets with language goals. Right: GVL. still outperforms LIV on datasets with image goals despite solving the more difficult task of frame re-shufiting.
4.1 LARGE-SCALE REAL-WORLD EVALUATION
To study GVL's zero-shot value prediction capability, evaluate its VOC on two large expert robotics datasets.
Open X-Embodiment dataset. First, consider the Open X-Embodiment (OXE) dataset (Padalkar et al., 2023). an aggregation of trajectory data from 50 standalone academic robot datasets that consists of diverse tasks, robots, and camera viewpoints. For each of the 50 datasets, we randomly sample 20 trajectories and evaluate GVL zero-shot on each of the sampled trajectories. Note that not all OXE datasets have language task annotations, so we use the last frame of the trajectory as the goal specification when text annotation is not provided. To better contextualize GVL's 's value prediction quality, we compare a sate-of-the-aa multi-modal value model LIV (Ma et al., 2023a), a contrastive vision- vision-language model (Radford et al., 2021) fine-tuned with value learning objective videos for in-the-wild value estimation. LIV predicts the temporal value of an input observation by computing its embedding distance the embedding of the goal image or task description.
human
For evaluation, we plot the histogram of all 1000 (50x20) Value Order Correlation (VOC) scores in Fig. 2, split by goal modalities. Given that most OXE datasets contain collenee expert demonstrations, good value models should have high VOC scores; however, we acknowledge that there are sub-optimal trajectories within OXE that can introduce noise in our results; in Fig. 10 in Appendix F, we further evaluate the two methods on a subset of OXE datasets that is delegated to be high-quality corroborate Fig. 2. After we first establish GVL as an effective universal value model. we will present how GVL can be used to detect low-quality data in Section 4.3. As shown in Fig. 2, on both goal modalities, GVL consistently generates VOC scores that heavily skew to the right, indicating that it is able to zero-shot recover the temporal structure hidden in the shuffled demonstration videos, i.e., coherent value predictions. GVL's performance is also markedly better than LIV on language goals (Fig, 2 left). Here, LIV's predictions are random, suggesting that its embedding space does not contain sufficient knowledge for predicting dense values for arbitrary unseen robot videos. On image goals, LIV's prediction problem is arguably simpler because an embedding space that simply captures image similarity can result in ascending values that correlate with timesteps. Even then, GVL generates better quality value predictions as judged by slightly higher VOCs (Fig. right). In summary, GVL can indeed effectively utilize the world knowledge afforded by the backbone VLM to achieve effective value predictions zero-shot for the breadth of real-world robotic tasks and datasets.
Challenging bimanual datasets. OXE datasets primarily focus on simpler. short horizon single-arm tasks. To further stress test GVL, we evaluate on a new diverse dataset of 250 distinct household tabletop tasks on the bi-manual ALOHA systems (Zhao et al., 2023; Team et al., 2024a). This dataset includes highly challenging, long horizon skills, such as removing three gears sequentially from NIST board, folding a dress in eighth-fold, hanging a t-shirt on a cloth rack. See the bottom right of Fig. 1 for representative ALOHA tasks. For each task, we evaluate on 2 human teleoperated demonstrations to evaluate GVL and LIV zero-shot. The aggregate histogram over all 500 (250 x 2) VOC scores is illustrated in Fig. 3. As shown, GVL is capable of generating value predictions that are positively correlated on more than 60% of them with median VOCs of 0.12. This is promising, but worse than the performance the OXE datasets. GVL and LIV exhibit similar qualitative trends, indicating that both methods now struggle with the complexity of this dataset. In the next section, we extensively explore how to improve GVL on this dataset using in-context learning techniques.

---

Published as a conference paper at ICLR 2025
Histogram (ALOMA-250)
GVL In Context Improvement (ALOHA-13)
0.7
Zero-shot
Value- Order Corracation
LIV
0.6
Frequancy
One Shot Modian
Median
0.4
0.2
0.3
0.0
-1.00
-0.75
-0.50
-0.
25
0.00
0.25
0.50
0.75
1.00
0.2
Value- Value-Drdee Order Correlation
Number In-Context Trajectories
Figure 3: scales up 250 ALOHA bi-manual tasks and improve with in-context examples.
GVL Cross Embodmmen One-Shot Learning (ALOHA-13)
0.6
Value Droee Corneation
0.5
0.64
0.4
0.36
(a) In-context examples from human videos
0.3
0.2
0.1
Zero-Shot
Ore-Shot
One-Shot
(b) Target prediction robot videos
Figure 4: GVL benefits from cross-embodinent in-context learning capability: its value predictions be improved by examples from human vidcos.
4.2 MULTI-MODAL IN-CONTEXT VALUE LEARNING
As the diverse ALOHA dataset is significantly more challenging, we explore whether GVL can benefit from in-context learning, where additional shuffled observation-valus pairs are presented in the VIM context window (Fq. 4).
Few-shot in-context learning. First, we collect an additional demonstration for each of the 250 tasks and use its shuffled value-observation pairs as context for one-shot GVL value prediction for the same set of 500 evaluations. As seen Fig. 3, with one in-context trajectory, GVL's performance substantially improves with 90% positive VOCs and median VOC of 0.37. We further investigate whether performance can improve with more in-context examples on a represented subset of 13 tasks for which have more than 500 demonstrations; we evaluate few-shot GVL on 500 distinct trajectories per task with up to in-context examples. The average VOCs tasks and trajectories is shown in Fig. 3 (Right). We that GVL. demonstrass appealing in-context scaling as the average score steadily improves as we increase the number of in-context examples. Even with 5 in-context trajectories, meaning 150 total shuffled images, GVL is able to utilize its full context and exhibit strong generalization. This result demonstrates how state-of-art songccontett-window VLMs, such Gemini- -1.5 Prro, can be re-purposed to make for general-purpose purpose value functions with impressive test-time time improvement capability, quickly mastering value predictions with minimal supervision.
refer to this subset ALOHA-13. For these tasks,
Cross-embodiment in-context learning. Examples in-context are not limited to robot demonstra- tions. One advantage of GVL is that it can still benefit from in-context learning even when the demonstrations come from a different embodiment. Specifically, we record humans performing the same tasks as the ALOHA robot demonstrations and then use these human demonstrations as in-context examples for value prediction. As shown in Fig. 4, GVL with one cross-embodiment in-context example can effectively improve over its zero-shot counterpart. the Appendix, we also show that GVL similarly benefit from pross-task in-context learning. In conclusion, GVL presents a versatile framework for in-context value learning that can scale up to even the most challenging manipulation tasks.
7

---

Published as a conference paper at ICLR 2025
4.3 GVL APPLICATIONS
As GVL can generate high-quality value estimates, it can be applied to a number of downstream tasks including dataset quality estimation, success detection, and weighted imitation learning.
Dataset Quality Estimation. Robotic action models are increasingly trained on large mixtures of datasets (Padalkar et al., 2023; Team et al., 2024b; Kim et al., 2024) and selecting the right mixture is critical for policy performance (Hejna et al., 2024). However, dataset mixing is often done in an ad-hoc fashion by visual inspection (Team et al., 2024b). Having validated that GVL is an effective zero-shot value model, we investigate whether we can in turn use GVL's VOC scores to determine dataset quality within OXF To this end, for each OXE dataset in Fig. 2, we compute the average correlation scores for its sampled trajectories and present the ranking of the average score in Appendix C. In Table 1, we present subset of selected representative large-scale datasets in OXE. We see that datasets have large spread in their VOC scores, but these scores are interpretable and match human intuitions. Specifically, datasets collected from human teleoperators with relative fixed camera placements, such as RT-1 (Brohan et al., 2022), Dobb-E (Shafiullah et al., 2023), and Bridge (Ebert et al., 2021; Walke et al., 2023), have high VOC scores, despite their diversity in scenes and tasks. In contrast, datasets with autonomous data collection via scripted motions or motor babbling. such as QT-OPT (Kalashnikov et al., 2018) and RoboNet (Dasari et al., 2019), contain high number of suboptimal trajectories that do not exhibit smooth temporal structure to be re-shuffled.
Interestingly, DROID (Khazatsky et al., 2024), a recent large household manipulation dataset is ranked very low, consistent with prior works (Kim et al., 2024) that found that removing DROID from large action model training im- proved final performance. After inspecting tra- jectories from DROID with a low VOC score from GVL we found that many have poor cam- era angles that do not capture robot motion or have the arm or manipulated objects heavily oc- cluded, These observations indicate that GVL VOC can be indicative of dataset quality. In Ap- pendix G, we show that GVL dataset quality estimation can effectively be used to generate effective co co-training datasets from a raw, mixed -aked-yuality dataset such as DROID for real-world imitation learning.
Dataset
Avg. VOC
RT-1 (Brohan al., 2022)
0.74
Dobb-E (Shafiullah et al., 2023)
0.53
Bridge (Walke et al., 2023)
0.51
QT-ÖPT (Kalashnikov al., 2018)
0.19
DROID (Khazatsky et al., 2024)
-0.01
RoboNet (Dasari al., 2019)
-0.85
Table
Average GVL VOCs on selected OXE datascts.
Success detection and filtered imitation learning. Next we consider more granular intra-dataset quality control by investigating how GVL can be used as success detector for trajectory filtering, enabling filtered imi- tation learning on mixed quality datasets. As discussed, good value models should return low VOC scores on unsuccessful trajectories; in particular, it is difficult for GVL to re-shuffle frames within sub-optimal trajectories which often contain irregular repetitive behavior.. Thus, we can use GVL for success detection by filtering trajectories that have VOC scores below certain numerical threshold; we refer to this procedure as GVL-SD. We evaluate GVL-SD on six simulated bimanual dexterous manipulation tasks on the ALOHA system (see Fig. 7). Simulation is well-suited for this experiment because we can naturally control for data quality and reproducibility. More specifically, for each task, we construct a mixed quality dataset by rolling out a pre-trained policy of roughly 50% success rate for 1000 episodes, mirroring real-world autonomous data collection settings with high failure rate (Kalashnikov et al., 2018). We compare to SuccessVQA (Du et al., 2023), which poses success detection as a Visual- QuesstQn Answering problem. To ensure that the same amount of infor- mation is provided, we feed the full video sequence to the VLM; therefore, this baseline tests whether the VLM is equipped with video understanding capability good enough for out-of-the-bol success VooccessVAA-CA
Method
Accuracy
Precision
Recall
GVL-SD (Zero-
0.71
0.71
0.71
GVL-SD (One Shot)
0.75
0.85
0.70
SuccessVOA (Du al., 2023)
0.62
0.33
0.73
VOCCosSVAA-CoT
0.63
0.44
0.68
Table Comparison of VLM success detectors.
detection. In addition, we consider et al., 2022) to encourage the VLM to output intermediate textual reasoning outputs before providing the final success answer. Unless otherwise stated we use VOC threshold of 0.5 for GVL-SD.
which uses chain-of- -hought prompting ( (Wei
For all methods, we report the accuracy, precision, and recall in Table 2. GVL-SD consistently outperforms matches Success VQA on all classification metrics. In particular, Success VQA has
8

---

Published as a conference paper at ICLR 2025
GVL VOC Histogram (ALOHA-SIM.
Shuffling Histogram (11--
600
Success
2000
Success
500
Failure
Frequency
1500
300
1000
200
500
1.00
-0.75
-0.50
0.25
0.00
0.25
0.50
0.75
1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Value Comeelation
Correlation
Figure 5: GVL for Success Detection. Left: GVL behaves qualitatively differently successful and failed trajectories. Right: GVL (No-Shuffling) loses discriminability failure trajectories.
Comparison GVL One-shot Detection Fitered Behacior Cloning
Aloha Sim Score GVI. One-shot Threshold
0.50
0.47
GVL One- hot
1.20
T
Task Average
Nermalizeed Scere
64
0.37 0.37
2.15
Succecs
0.31
0.31 0.32
0.31
0.3
0.24
9.26
2.10
0.36
2.05
0.04
bawi-on-rack dcuble-insertion glass-on-rack mug-on-plate plate-on--ack single-irerttnn
Correlation Thrieshold
Figure 6: Success-Filtered Imitation Learning on ALOHA Simulation Tasks, Left: Using GVL-SD for success-filtered BC substantially oulperforms SuccessVQA. Right: GVL-SD is not sensitive the VOC threshold for improving imitation learning.
low precision, indicating that the base VLM systematically biases towards outputting failure. Adding in-context demonstration further improves GVL's performance across all metrics.
In Fig. 5 (Left), we also visualize the histogram of the VOC scores GVL, produces on success and failure trajectories. As expected, GVL failure trajectories renders uniform distribution when the task is unsuccessful, indicating the model's inability to uncover the original temporal order success and failure trajectories have distinct distributions over the correlation values indicating that GVL can adequately separate them. Fig. 5 (Right) shows that the histograms without shuffling are largely the same independent of success failure. This shows that by forcing the VLM perform the more difficult prediction task over shuffled frames, GVL can elicit better zero-shot values.
Now, we use the above success de- tection methods for filtered imitation learning; for all methods, we use Ac- tion Chunking Transformer (ACT) as the imitation learning algorithm (Zhao et al., 2023); ACT hyperparameters are tuned for ACT on the success-only subset and are fixed for all methods. Given the noisiness model check- points performance, we report the av- erage success rate of the last 10 model training checkpoints. Results for the six simulation tasks are shown to the right of Fig. 6, where GVL-SD's improved success detection leads to better performance over Suc- cess VQA. In fact, Success often hurts performance, likely because of its low precision which causes the policy (Right) show the effeect of varying the VOC threshold in {-1.0, O, 0.25, 0.5, 0.75) in comparison to training on all the data with ACT; note that this is the same using the lowest threshold, - 1.0 as it is lower bound on the VOC metric. As seen, GVL consistently outperforms ACT regardless of threshold values; when the threshold value is too high, i.e., 0.75, we see slight dip in performance when the overall dataset size becomes too small.
Real World ALOHA Tasks GVL+ DP
DP
Avg. VOC
bowl-in-ran banana-handover close-laptop open-drawer
7/10
6/10
0.57
7/10
5/10
0.73
9/10
6.5/10
0.59
4/10
6/10
0.09
remove-gears
4.67/10
7/10
0.19
pen pen-handover
1.5/10
0/10
fold-dress
7/10
7/10
Table 3: Real-World ALOHA Policy Learning Results. AWR with GVL (One-Shot) outperforms imitation learning baselines when the value predictions have high VOCs.
train on a high number of false positive (i.e. failure) trajectories. In Fig.
Aevintage-weighted regression for real-world visuomotor control. Finally, we illustrate how GVL assign importance weights to individual transitions within trajectories at fine-grained

---

Published as a conference paper at ICLR 2025
level akin to offline reinforcement learning. For these experiments data collected by human teleoperation on bi-manual ALOHA robot setups. Unlike simulation, our datasets only contain successful task executions but can be sub-optimal and multi-modal. Thus, directly utilize GVL's values with advantage weighted regression (AWR) (Peters & Schaal, 2007;
real-world demonstration
Peng et al., 2019); in which we weight each individual transition by the estimated advantange, or GVL value difference for that step:
[()) == -E lexp (T (16+1- -2)) log no (ax [x)]
(6)
We use diffusion policy (DP) as the policy backbone (Chi et al., 2023) for each task, and compare training diffusion policies with GVL (One-Shot) advantage weighting or lack thereof. We evaluate on 7 tasks with 10 trials per task and report success rate in Table 3. As can be seen, on majority tasks, GVL-DP outperforms DP and we see a clear correlation between improvement over DP and the VOC score. That is, when the value predictions are of high quality as judged by VOC, policy learning can benefit from GVL value weighting. On open-drawer and remove-geare rs. the top-down view does not provide sufficient resolution to distinguish task progress (see Fig. 8), as a consequence, the value predictions can be noisy, which can hurt policy learning. However, given the in-context learning results, we believe that it is possible to improve policy learning even on difficult tasks with non-ideal camera viewpoints.
4.4 ABLATIONS
Finally, we ablate key algorithmic design choices of GVL to validate their necessity. In the Appendix, we additionally demonstrate that GVL's performance robust the choice of backbone VLMs as well as input camera viewpoint.
Is autoregressive value prediction necessary? We consider an ablation that simply asks the VLM to predict values of input observations one by one without GVL's autoregressive batch prediction mechanism. This ablation, which we refer to as VLM (Single Frame), essentially poses value estimation VQA problem. In Tab. 4, we compare this ablation to GVL on subset of RT-1 dataset as in Section 4.1; the average VOC for VLM (Single Frame) is a mere -0.08, a significant drop from GVL's 0.74 on RT-1 dataset. As seen, pre-trained VLMMs by themselves are poor value estimators, generating inconsistent values that are too noisy to be used in practice.
Is input observation shuffling necessary? As discussed, we find that removing shuffling collapses ICV's predictions into generating degenerate values; that is, regardless of the quality of the provided trajectory, GVL. tends to predict monotonically increasing values, resulting inflated VOC scores that cannot be used to discriminate successful and failure trajectories., see Fig. 5 (Right). To further qualitatively illustrate this phenomenon, in Fig. 11 in the Appendix, we overlay raw GVL value predictions with frame shuffling and lackthereof to understand the spread of the value curves. We see that the overlay for original GVL looks "messy", suggesting that GVL outputs varied value curves that better capture the heterogeneity of the queried video qualities. contrast, without frame shuffling, GVL predictions indeed collapses onto few lincar ascending patterns.
RT- 1 dataset
GVL
Single Frame
0.74
-0.08
Table 4: Using VLMs to pre- dict values frame-by- -ramme signif- icantly underperforms GVL's au- toregressive mechanism.
5 CONCLUSION
We have introduced Generative Value Learning (GVL), universal value function via VLM autore- gressive value prediction on shuffled video frames. GVL can zero-shot output dense and high-quality value predictions for diverse and challenging real-world robotic tasks, spanning various robot embod- iments and task categories. With few-shot learning from the same task, different task, or different embodiment, GVL performance steadily improves. We have demonstrated several use cases of using GVL to perform dataset, trajectory, and transition selection to improve downstream policy learning performance and generalization. We believe that GVL takes an important step in using foundation models supervision for robot learning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We thank Jie Tan, Pannag Sanketi, Oliver Groth, and the rest the Google DeepMind Robotics team for helpful discussions and providing feedback on the paper. Yecheng Jason Ma supported by the Apple Scholars in AI/ML PhD Fellowship and OpenAl Superalignment Fellowship. This project was supported in part by NSF CAREER 2239301, NSF Award 2331783, DARPA TIAMAT HR001 HR0011249044 12490421.

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01699 2022.
Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan, Gheorghe Comanici, Sebastian Flennerhag, Maxime Gazeau, Kristian Holsheimer, Dan Horgan, Michael Laskin, et al. Vision- language models source rewards. arXiv preprint arXiv:2312.09187, 2023.
Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh, Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline rl from internet videos via value- function pre-training. arXiv preprint arXiv:2309. 13041, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:22
2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15811 2023.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. /4/65, 2020.
Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103. 16177, 2021.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain lchter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing lasounlanguage models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465 2024.
Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 2:XY::23033.4137, 2023.
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. In Conference on Robot Learning, pp. 2071-2084. PMLR, 2021.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv: /215, 2019.
Norman Di Palo and Edward Johns. Keypoint action tokens enable in-context imitation learning in robotics. arXiv preprint arXiv:2403. 19578, 2024.
Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06247, 2023.
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. arXiv preprint arXiv:2303.07280, 2023.
Yan Duan. Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho. Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danilidiss, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generulization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 1339669..11....... 2021.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909, 2020.
12

---

Published as a conference paper at ICLR 2025
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning In Conference on robot learning, pp. 357-368. PMLR, 2017.
Letian Fu, Huang Huang, Gaurav Datta, Lawrence Yunliang Chen, William Chung-Ho Panitch, Fangchen Liu, Hui Li, and Ken Goldberg. In-context imitation learning via next-token prediction. arXiv preprint ar arXiv:2408. 009400.......... 2024.
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12462-12469. IEEE, 2024.
GeminiTeam, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions tokens of context. arXiv preprint arXiv:2403.0555 2024.
Lin Guan, Yifan Zhou, Denis Liu, Yantian Zha, Heni Ben Amor, and Subbarao Kambhampati. task success" is not enough: Investigating the use of video-language models as behavior critics for catching undesirable agent behaviors. arXiv:2402.04210. 2024.
Joey Hejna, Chethan Anand Bhateja, Yichen Jiang, Karl Pertsch, and Dorsa Sadigh. Remix: Optimiz- ing data mixtures for large scale imitation learning. 8th Annual Conference on Robot Learning. 2024. URL https::////. wwtp/peeeeeeeeeee//e/ee (
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 14220482-20499 2023.
Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305. / 176, 2023a.
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint arXiv:2303. 00855, 2023b.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002, PMLR, 2022.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pp. 651-673. PMLR, 2018
Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint arXiv:2302.12766, 12766 2023.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403. 12945, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint 2.Xii:2406.09246: 2024.
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Fierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:23 10.00166, 2023.
13

---

Published as a conference paper at ICLR 2025
Minae Kwon, Hengyuan Hu, Vivek Myers, Siddharth Karamcheti, Anca Dragan, and Dorsa Sadigh. Toward grounded social reasoning. arXiv:2306.08651/ 2023a.
Teyun Kwon, Norman Di Palo, and Edward Johns. Language models as zero-shot trajectory generators. arXiv preprint arXiv:2310.
2023b.
Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph J Lim. Generalizable imitation learning from observation via inferring goal proximity. Advances in neural information processing systems, 34:161 18-1613, 2021.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
Jacky Liang, Fei Xia, Wenhao Yu, Andy Zeng, Montserrat Gonzalez Arenas, Maria Attarian, Maria Bauza, Matthew Bennice, Alex Bewley, Adil Dostmohamed, et al. Learning to learn faster from human feedback with language model predictive control. arXiv preprint arXiv:2402. 1450, 2024.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303. /2/53, 2023.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang. Joydeep Biswas, and Peter Stone. Lim+ P: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304. 11 1477,, 2023.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint S.Xiv:2210000033 2022.
Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pp. 23301-23320 PMLR, 2023a.
Yecheng Jason Ma. William Liang, Guanzhi Wang. De-An Huang, Osbert Bastani, Dinesh Jayaraman. Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. /2931, 2023b.
Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In International Conference Machine Learning, pp. 14743-14752 PMLR, 2022.
Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xiang. Zero-shot temporal action detection via languale prompting. European Conference on Computer Vision, pp. 681-697. Springer, 2022.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.0880 2023.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 1910.00177 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by rewiht- regression for operational space control. In Proceedings of the 24th international conference Machine learning, pp. 745-750, 2007.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
14

---

Published as a conference paper at ICLR 2025
Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310.12921, 2023.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference machine learning, pp. 1312-1320. PMLR, 2015.
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. arXhv:1612.06699 2016.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Timeecontrastive networks: Sepe-supe learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. On bringing robots home. arXiv preprint arXiv:2311.16099 2023.
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. Generalized planning pddl domains with pretrained large language models. arXiv preprint arXiv:2305.11014. 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference Robotics and Automation IEEE, 2023.
(ICRA), pp. 11523-1 1723--1530 1530.
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Natalie Mackraz, Devon Hjelm, and Alexander Toshev. Large language models as generalizable policies for embodied tasks. arXiv preprint arXiv:2310.17722 2023.
Yujin Tang, Wenhao Yu, Jie Tan, Hciga Zen, Alcksandra Faust, and Tatsuya Harada. Saytap: Language quadrupedal locomotion. arXiv preprint arXiv:2306.07580, 2023.
ALOHA 2 Team, Jorge Aldaco, Travis Armstrong, Robert Baruch, Jeff Bingham, Sanky Chan, Kenneth Draper, Debidatta Dwibedi, Chelsea Finn, Pete Florence, Spencer Goodrich, al. Aloha 2: An enhanced low-cost hardware for bimanual teleoperation. arXiv preprint arXiv:2405.02299 2024a.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv: 12213, 2024b.
Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. bosed- visual planning with self-supervised functional distances. arXiv preprint 2tt.:200/2.533333 2020.
David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup. Sherry Yang, and Ankit Anand. Code reward: Empowering reinforcement learning with vims. arXiv preprint arXiv:2402.00 0444444444 2024.
Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pp. 1723-1736. PMLR, 2023.
Huaxiaoyue Wang, Gonzalo Gonzalez- Purrarez- Yash Sharma, and Sanjiban Choudhury. Demo2code: From summarizing demonstrations to synthesizing code via extended chain-of- thought. arXiv preprint arXiv:2305. 16744, 2023.
Yufei Wang. Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RI-vim-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402. 03681, 2024.
15

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. th-tout prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24877 2022.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309.11433 2023a.
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023b.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pp. 24631-2466 PMLR, 2022.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huch Lee, Montse Gonzalez Arenas, Hao-Tics Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2300. 0.6477 2023.
Jesse Zhang. Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim. Bootstrap your skills: Learning to solve tasks with large language model guidance. arXiv preprint arXiv:2310.10021, 2023.
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304113705 2023.
16

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE WORK
GVL most suited for offline setting, in which full trajectories are available for inference. In the online setting, GVL must re-query each new sub-trajectory from every step; this may be computationally expensive. Furthermore, we have not investigated whether pre-trained VLMs can be fine-tuned to perform better value predictions. In addition, though test diverse camera view- points, we have not yet investigated whether multi-view observations can improve value prediction quality. In addition, tasks for which there exists unique ordering of frames from an expert demonstration. Tasks such as wiping stirring may be hard discern. Though these limitations present avenues for future work, we believe GVL is a step towards improved in- mn-the-will wild value estimation.
evaluation metric Value-Order Correlation is most suitable for a-periodic
B PROMPT
In this section, we provide the full prompt provided to the VLM for GVL predictions. The same prompt is used for all OXE datasets.
You are an expert roboticist tasked to predict task completion percentages for frames of robot for the task of mask_description)
The task completion percentages are between 0 and 100, where 100 corresponds to full task completion. etion. We provide several examples of the robot performing the task at various stages and their corresponding task completion percentages. Note that these frames are in random order, so please pay attention to the individual frames when reasoning about task completion percentage.
Initial robot scene:
In the initial robot scene, the task completion percentage is 0.
Now, for the task of baakk_esscriptonn)) output the task completion percentage for the following frames that are presented in random order. For each frame, format your response as follow: Frame {i}: Frame Description: (), Task Completion Percentages:
Frame {
C GVL OXE DATASET VOC BREAKDOWN
In this section, we provide the full list of average VOC score for each OXE dataset. In Table 5, we provide the VOC scores GVL with Gemini- 5 Pro as the backbone VLM. In Table we provide the VOC scores for GVL with 4o as the backbone VLM.
D SIMULATION TASKS
Figure 7, we illustrate six simulation tasks used for the success detection and filtered imitation learning experiment. For each task, use VR teleoperation to collect 500 trajectories for initial policy training. After the policy converges, we rollout the last checkpoint for 1000 imtes, resulting in naturally balanced mix-quality datasets of about half success and half failure trajectories.
E REAL-WORLD TASKS
In Fig. 8, we provide the top-down view of the 7 ALOHA tasks used for real-world policy learning experiments.
17

---

Published as a conference paper at ICLR 2025
Dataset
VOC Score
utokyo pr2 opening fridge converted externaly to rlds
0.8095
utokyo xarm bimanual converted externally to rids
0.7955
Slokyo_xarm_pice www.appaac.cooeeeeeeeeeeeeeeeeee fractal20220817_daa
_tt to_rsds
0.7880
0.7385
httpallllccaattttooooveeeeeeeeee berkeley waweeleyyautllb..... _ur5
0.7260
0.7185
wyu.door.opening appeningsuupprisig wwwsssuiiiggeeeeeiiiiiiiii effectiveness
0.6685
utokyo_pt pr2 tabletop manipulation converted externally to rids
0.5875
utaustin mutex
0,5810
haaplaaccmuppickupinneeeee hrgggug.inneettooooeveeeeeeeeeer
0,5585
fmb
0,5555
uesd kitchen dataset converted externally to rlds
0.5295
dobbe
0.5295
toto
0.5270
bridge
0.5145
austin sirius dataset converted externall rourdds
0.5100
asu_table_rtt top_converted converted_exterrallly externally to to_rlds Biakkleyyppt..coneetted eetettedllxtemlllly... berkeley cable routing
0,5055
0.4835
0,4470
use cloth sim converted externally to rlds jaco_play
0.4410
0.4205
bc_z
0,4065
viola
0.4035
wttp:/ccoooeeeeeeeeteeeeee rlds roboturk
0,3900
0.3545
austin buds dataset converted externaly to rlds stanford hydra dataset converted extemnlly rids srryyo__lsmo_coneee wew.ereteddettrnnlllyy.... _to_rlds frkkeley_fnucc. canuuummanpulttionn cmu stretch
0.3415
0.3325
0.3140
0.2685
0.2625
uesd pick and place converted externally rids kuka
0,2410
0.1915
dlr sara para converted externalyy to rids taco play
0.1600
0.0945
dlr dir_eddan_shardd_comteee wtthaaneeoooooooooooveeeeeeeeete
0.0855
droid stanford robocook converted externally to rlds
-0.0060
-0.0690
imperialcollege sawyer wrist cam
-0.1225
mhiis_nopprehensie..coomm wwwpieessilleecooeeeeeeeeeee wwwwwoowweeeeeeooeo externall to_rlds
-0.1310
austin sailor www.:eeeoooeeeeeeeeeeeeeeeeeeeee
ro.ldd
-0.1715
cmu unnuplay fusion
-0.3445
stanford kukaakkknnnn nuka_multimodal talttiimodll_datasttt...mm cetpisttecoovereeeeeeee stanford
rour
-0.3770
_to to.rrdds
-0,4505
nyu franka play dataset converted externally to rlds
-0.4555
uiue d3field
-0.7025
cmu_franks -eenkkkatxoootiionn dataset dattset.comeerteeexxeemaa... columbia.cairlat www.:haaaialaaapushtteeeeeeee reaht_real
-0.7395
-0.7625
robo_net
-0.8485
dlr lir_sara_grid www.:rrrriddddddeeeeeeeaa httppppp.ooeeeeeeeeeeeallal.o...
-1.0000
Table 5: GVL (Gemini-1. OXE Dataset VOC Scores
18

---

Published as a conference paper at ICLR 2025
Dataset
VOC Score
0.883
utokyo pr2 opening fridge converted externall to rlds
0.864
berkeley mvp converted externally to rlds
0.8285
utaustin_mutex
0.813
fractal202208 17_data data
0.803
utokyo_xarm_pie
0.7665
berkeley.autolee _ur5
0.755
utokyo_xang Bickyy__arm__bbe
0.749
utokyo pr2 tabletop manipulation externally to rlds
0.734
austin sirius dataset converted externally to roddds toto
0.7235
0.713
dlr_edan_sharee coddan.hardddccntttoocommmmm wow.tooooooooottto.....eelaalll. bridge
0.6595
0,6445
berkeley fanuc wtw.fafaaaanaaaeeeeeaeaaaaaaaade manipulation
0.6295
berkeley sereey-pppttooeee converted externally to fo.rlds
0.6235
uesd kitchen dataset converted converted_extermally rlds
0.603
roboturk
0.57
jaco_play
0,5615
0.557
uiue d3field
0.5395
cloth sim converted externally to rlds
0.5355
asu_table www.uaablettp.cooveteeeeeede externaaly _t_rlds maniskill www.eereeeeeeeeeealllll.......e. _t rordds kaist_nonprehensil... www.eronneeeeeeeeeeeeieeeeeeeeee to rlds viola
0.5025
0.499
0,492
0.4605
austin_ cmu play fusion
eet.reelleeeetllyeeeecco cottmalllyttorrree rlds
0.454
0.4235
tokyo Ismo converted externally rlds
0.3875
austin cestii_sailor_.. dlataset_cooveetteeeeeeeeeeeeeee
tl_rlds
0.3015
ucsd cosd_pick_and_ waw.eppacee...mmm .oaveret. wet.reralleeeeteeeeeeaeaa -o berkeley_cale wwaceley_ccbleeruuuue routing
0.2675
0.255
dlr-sara, airrssaa_pouur wtw.pppuroorrreeee converted externally
.laterall.etternallly to to.rlds
0.252
imperialcollege seerrialoolleeesaweer www.saawwwwwsssseeeeeees wwrsst.rrimmmmmm robo net
0.239
0.237
stanford hydra dataset converted externally to rlds cmu_stretch
0.205
0.1895
bc_z
0.176
nyu franka play dataset converted externally to rlds stanford robocook converted externally to rids kuka
0.1735
0.16
0.132
stanford Lanffrrd_aas.... wttpvvitcccvvee converede externally to tourdds
-0.173
stanford kuka multimodal .laaaset.ttttccccaaaaaa datasee..oonverted convertee..etterallll.. externally to rlds
-0.1785
Dolumbia_caiilaa Secaarrabaussatt..._ _uuht_reell
-0.1815
cmu mmu.ffanna..cmm htt.:/ottoooaattttttc.oooetccttd
-0.2075
taco play
-0.7705
eth agent affordances dlr war.saaaarrrdddlaaeeer
-0.279
-1
Table GVL (GPT-40) OXE Dataset VOC Scores
19

---

Published as a conference paper at ICLR 2025
(a) single.insertion
(b) double.
(c) mug-on.plate
(d) boxl.on.rack
(e) place.o..rack
(I) glass.on.rack
Figure 7: Simulated task setups of dexterous manipulation the ALOHA robot.
(a)
(b) bowl-in-rach
(c) Allose-laptor
(d) fold-dress
(c) open-drawer
(f) pan-hanoover
remove-gears
Figure 8: Real-world experiment setups for dexterous manipulation the ALOHA robot.
F GVL VISUALIZATION
In this section, we visualize several raw GVL predictions on our ALOHA datasets. Samples are chosen from diverse camera viewpoints for which we evaluated GVL. The visualizations are Fig. As shown, on diverse viewpoints, GVL remains effective for diverse tasks. We do observe that GVL sometimes would predict "spike" values that are not consistent with the rest of the predictions. We hypothesize that this could be due partial observability of the task that particular timestep as well as inherent stochasticity in the shuffling order; we leave to future work for more systematic investigation of these types of errors.
G ADDITIONAL RESULTS
In this section, we present additional results and analysis.
GVL and LIV comparison on subset of high-quality OXE datasets. Given that not all OXE datasets are necessarily demonstration data most suitable for being used for VOC-based evaluation. we have delegate few datasets to be of "high-quality" to use VOC original results as in Fig. 2. To this end, we have selected RT-1 (Brohan et al., 2023), Bridge (Ebert et al., 2021; Walke et al., B02B), (Shafiullah et al., 2023), and BC-Z (Jang al., 2022) datasets because these datasets have successfully been used in prior works for large-scale imitation learning and include diverse tasks and camera viewpoints. The results illustrated in Fig. 10. As
metric and re-create the
20

---

Published as a conference paper at ICLR 2025
0.4
0.2
Fou Brtl uit Dr s1
Chord Tutts Nomtor
(a) fo di dress from the top-down view.
lome
Stuffed Farrai
14
Grand Ttank Rarna Saur
(b) pen panddveeover from table view.
na polped lagg
(c) close lapt op from the right wrist camera.
(d) remove -gaar from the left wrist
Figure Example GVL predictions on diverse ALOHA tasks.
21

---

Published as a conference paper at ICLR 2025
OXE Value-Ordee Order Correlation Histogram (Bridge, BC-Z, Dobbe, RT-1)
33
Density
05
05
Value-Order Correlation (VDC)
Figure 10: On subset of high-quality OXE datasets, GVL consistently outperforms LIV, mirroring the
comprehensive OXE results in Fig.
seen, on this subset of high-quality and diverse OXE datasets, GVL consistently generates highly positive VOCs; in contrast, L.I generates VOCs close uniform distribution, indicating that struggles to accurately predict language-conditioned task progress on unseen robot videos.
GVL's qualitative behavior task failure and repetition. In supplementary material submis- sion, we have included two videos in which repetition or failure is present. As shown, in both cases, when the robot gripper pulls back from the object of task interest, GVL's task progress estimates decrease, and when the gripper recovers and makes progress again, GVL's estimates increase. These results demonstrate GVL's ability to accurately estimate values videos when repetition present.
GVL and No-Shuflling ablation qualitative comparison. As shown in Fig. 11, GVL generates value predictions that are varied over time; in contrast, without frame shuffling, the predictions all collapses onto a few monotonic patterns.
CVL Valuu Pred cions Chasllattive Compansson
GV, Viue Precedamm Gual talve Compersin
Shufte Irue
Sh.llhl
0.6
0.8
Luak 7r0gtes
Tesk
6.
03
10
15
25
25
30
13
Fammo
Fromo
Figure 11: GVL without shuftling produces uninformative monotonic values regardless of trajectory quality.
DROID co-training dataset filtering results. We use GVL to filter a subset random demonstra- tions from the DROID dataset (Khazatsky et al., 2024). We first use GVL to construct VOC scores for each of the two external cameras for 1243 demonstrations. Then, we filter the demonstrations according the average VOC score across both camera images. As policies trained on DROID do not perform well zero-shot, we follow prior works (Khazatsky et al., 2024; Hejna et al., 2024) and co-train policies with handful of in-domain demonstrations asses the quality of the filtered data. We include 15 demonstrations for each of three different tasks: 1) "Purple Bowl", where the robot places a purple bowl in a dish rack, 2) "Blue Pot" where the robot places blue pot the dish rack, and 3) a more challenging "Bowl Flip". where the robot has to first flip over large bowl, and then place the dish rack upside down. We train Diffusion policies (Chi et al., 2023) with a data mixture comprised of 60% demonstrations from DROID, filtered according to different methods, and 40% 15 in-domain demonstrations for each task. Full results can be found in Table 7. Similar to Khazatsky et al. (2024), we find that co-training with more data helps performance. GVL notably attains the same performance training with only 947 demonstrations as training on the entire subset of 1243 demonstrations. However, training hit reducing average success rate from 20/40 to only 7/40. All policies generally struggle more with the harder flipping task, especially with the blue bowl.
random subset of 947 demonstrations incurs performance
22

---

Published as a conference paper at ICLR 2025
Method
Demos
Purple Bowl
Blue Pot
Flip Bowl Orange
Flip Bowl Blue
Total
All Data
1243
9/10
6/10
4/10
1/10
20/40
Random Subset
947
2/10
2/10
2/10
1/10
7/40
GVL, VOC 0.0
947
10/10
6/10
3/10
1/10
20/40
Table 7: Policy Learning results using GVL filter the droid dataset.
Comparison of GVL, Zero shot and Success Detection for filtered Behavior Cloning
Noha sim Score vs GVL Zero shot Threshald
0.
0.50
Zerovshot
Rlate
04
0.38, 0.37
0.43
Success VGA
Nermatized Score
0.33 0.22
0.3
0.2
0.17
1.00
0.1
-
Average
0.95
0.0
baw--on-rack double-insertion glass-on-rack mug-on-plate
plate-on-rakk single-iderrten
-1.0
-0.5
0.0
0.5
Task
Value Correlation Threshold
Figure 12: Success-Filtered Imitation Learning on AT.OHA Tasks. Left: Using GVL-SD for success-files filtess- BC substantially outperforms SuccessVQA Right: GVL-SD is not sensitive to the VOC threshold for improving imitation learning.
Zero-Shot Aloha Sim Results, Fig. 12 we include results for GVL-SD zero-shot instead of one- shot. The results are qualitatively similar, where GVL-SDD consistently outperforms SuccessVQA. and different VOC threshold values all provide performance gain.
Different VLM backbone. We additionally consider GPT-4o as the backbone VLM to better understand GVL's performance in relation to the backbone VLM model. For evaluation, we plot the histogram of all 1000 (50x 20) Value Order Correlation (VOC) scores across all trajectories in Figure 13. As shown, GVL, independent of the backbone models, consistently generates VOC scores that heavily skews the right, indicating that it able to zero-shot recover the temporal structure hidden in the shuffled demonstration videos, i.e., coherent value predictions.
GVL OVE Value Order Correlation Hstogram
Frequancy
407
-027
-0.
3/
226
617
Corrolation
Figure 13: GVL. has comparable performances with different backhone VLMs;; the main difference is in the backhone model's 's refusal rate and conforming to the response template, which is reflected in the tall har at -1.0.
Cross-task in-context learning. We investigate whether examples from other tasks can also unlock GVL's in-context learning capability, On the previous ALOHA-13 13 tasks, we randomly pair up tasks, where we draw one demonstration from one task as the one-shot in-context example for another. Then, we compare VOCs with the original same-task one-shot setup. The results are shown in Figure 14. We see that providing examples from different task still beneficial, though the improvement not much as same-task examples. This is be expected intra-task examples still provide clue on the output format as well as a generic notion of task progress, but such information is not specific to the target task. That said, cross-tass ICL enables the flexibility of enabling foundation model guidance on a task without any tpee-fi prior.
Does GVL work different camera viewpoints? On ALOHA setup. collected all demon- strations using four camera viewpoints. Besides the top-down view reported in the main experiment above, we test whether GVL. remains performant when using alternative viewpoints, especially
23

---

Published as a conference paper at ICLR 2025
GVL Cross-Task One-Shot Learning (ALOHA-13)
0.6
Value Order Correlation
0.5
0.49
0.4
0.33
0.3
0.27
0.2
0.1
0.0
Zero-Shot
X-Task One-Shot
One-Shot
Figure 14: GVL demonstrates cross-task in-contexi learning capability: its value predictions can be improved by value examples from different tasks.
Owathood
Waisa
GVL In Context improvement by Camera (ALDHA-13)
one shot
Corrlation
0.5
0.3
0.1
Overhead
Table
Left Wrst
Right Wrist
Figure 15: GVL works better in-distribution table view, but one-shot improvement benefits all camera views.
gripper views that are likely more out-of-distribution with respect to the natural images used for VLM training. The aggregate zero-shot and one-shot results are shown in Figure 15. As seen, on average, GVL zero-shot works best on the Table viewpoint. This is not surprising. as images taken with the front facing table camera are arguably visually closer to naturally captured images used for all camera viewpoints.
VLM training. Yet, with in-context examples, GVL consistently improves In practice, this means that GVL is robust to camera viewpoints even when a camera viewpoint is determined to be sub-optimal post-hoc, practitioners make up for that by simply providing few in-context examples.
Does GVL pay attention to the task specification? To validate that GVL is merely recovering the temporal coherence in the shuffled input video but actively tracking visual progress according to the task language command, we compute the VOC scores for every combination of task input video and language description in the ALOHA-13 13 split. The heatmap visualization of the average VOC for every pairing is illustrated in Fig. 16 and Fig. 17 for GVL and the no shuffling ablation. On 9 out of 13 tasks, GVL achieves the highest VOC when the input video and the task description matches; in many unmatehed cases, the model simply refuses output value predictions, stating that the frames and the language description are not related. In contrast, when we do not shuffle the input frames, the quality dramatically drops.
24

---

Published as a conference paper at ICLR 2025
ShuffleTra True Task v. Description Haatmap
laptop
C.75
open laptop
9..33
dose shoo box-
remove-geees
+0.99
-0.12
<0.35
-0.24
0.28
3..00
+0.24
0.40
0.23
0.14
0.23
0.07
opes drower
-1.00
+1.00
0.24
-1.00
0.30
-1.00
0:36
0.71
-1.00
+1.00
+2,00
0.74
teke-hone-oot
1.00
1.00
1.00
3.00
0.32
1.00
3.00
1.00
1.00
1.50
0.00
apen-jandid
0.76
0.13
0.36
0.30
handover
bowe-ttaatk
1.00
1.20
0.70
0.05
-0.50
cup ii rack
1.00
1.000
1.50
1.00
3.00
4..00
3.04
1.00
0.05
0.10
0.24
00.12
banana- hancover
-1.00
+1.00
4:11
-1.00
0
4.000
11000
+1.00
-1.00
-1.00
0:32
40.17
-0.73
handover
0.30
0.31
0.33
0.33
1.00
0.36
C00
031
0.30
0.4
0.29
0.59
laptop
Shops Drc
open Wrannrr
take phoneeeeee otuf
ci Warkk
lananca kandoverr
Pert wwi-ttanddowet
Figure 16: GVL VOC for video and language description pairs. Shufflng enables GVL to pay attention to the language task description order faithfully predict observation values.
25

---

Published as a conference paper at ICLR 2025
ShuffleFalse Task v. Deser pttion Heatmap
laptop
open laptop
dose shoo box-
remove-geers
0.24
0.84
0.86
0.93
0.94
2.88
0:99
0.2
0.92
0.34
0.91
0.88
opes drower
-1.00
-1.00
0.79
0.93
0.01
4.011
0:56
0.57
-1.00
+1.00
0.97
0.83
0.25
teke-hone-oott
1.00
1.00
1.00
3.00
2.07
1.00
0.01
1.00
1.00
0.02
apen-jandid
0.07
0:00
0.88
0.96
handover
bowe-ttaatkk
0.28
0.07
0.50
cup in rack
1.00
11.00
-1.00
1.00
0.90
-1.20
C.00
11.00
0.83
0.97
0.02
0.38
banana- hancover
-1.00
+1.00
2.04
-1.00
D0
3.00
C.5.1
4..0
0.90
-3.08
0.91
1-0.75
handover
0.05
0.00
0.47
0.9
1.00
0.05
6.8
0.54
0.02
0.08
0.84
0.91
laptop
lake phoree otuf
6U Waakk
lananaa kandoover
pery wor-taanddoe
open
Figure 17: No-shuffling ablation VOC for video and language description pairs. Removing shuffling makes VLM output high VOC's independent task descriptions.
26


Reviews:
[{"review_id": "GFaWbsWyRF", "reviewer": "Reviewer_RQdG", "summary": "This paper presents Generative Value Learning (GVL), a novel technique that utilizes Vision-Language Models (VLMs) to serve as a value function. In particular, by shuffling the order of frames within the video and utilizing few-shot prompting, GVL exceeds the performance achievable using VLMs without shuffling and without temporal dependence. Furthermore, the authors demonstrate that, by comparing the GVL value estimates with the true frame order, it is possible to identify and filter out low quality video demonstration data. The authors demonstrate that GVL can be used to train more effective policies via both data filtering and more accurate value estimation.", "questions": "- It would be great to see LIV’s performance on the ALOHA dataset. I assume that, since GVL outperformed LIV on OXE, it would do so again on ALOHA, but it would be nice to see these results formalized.\n- What is ALOHA-13? I see the descriptions of ALOHA-250, but I don’t see the same for ALOHA-13 for some reason.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Although the proposed technique is simple, it seems (based on myriad experiments) to be effective. Figure 2 demonstrates a clear superiority of GVL over LIV, the baseline that the authors chose to test, for text-based goals. Figure 4 demonstrates that GVL can be decently effective at predicting values correctly over more complex videos. Finally, GVL was effective at training agents via both (1) data filtering for imitation learning and (2) advantage estimation for advantage weighted regression.\n\nBreaking things down by category:\n- Originality: Due to the simplicity of this technique, I think the originality is somewhat low, however the method is effective, especially in comparison with other simple alternatives.\n- Quality: With the exception of the potential circularity I mention below, the experiments performed seemed to be of high quality, and they seem to demonstrate the quality of this technique.\n- Clarity: There were some minor questions that I have detailed below, but for the most part, the paper was well-written and easy to understand.\n- Significance: The authors provide solid motivation for the significance of this technique by demonstrating how it could be used for model training and dataset quality checks.", "weaknesses": "The use of Value Order Correlation (VOC)—the correlation between GVL and frame order—seemed somewhat circular. It is used on the OXE dataset to judge GVL’s performance, a choice that assumes that the contents of OXE are of sufficient quality. (If, for instance, OXE contained many sub-optimal trajectories, an ideal value estimator would show low VOC.) Later, there is a discussion about the use of GVL+VOC to identify sub-optimal datapoints/subsets within OXE. This seemed confusing to me since it seemed to contradict the earlier assertion that OXE was of sufficient quality for judging GVL. \n\nIt would be good to see more analysis regarding GVL’s ability to accurately estimate values in videos where repetition is present. Currently, the existence of such an ability has to be inferred from downstream effects (e.g., the ability to use GVL for data filtering helps during imitation learning).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZkePIebo4K", "reviewer": "Reviewer_1PKJ", "summary": "The paper presents a method that enables existing VLMs to understand the task progress/completion, that could help in the use cases like dataset filtering, success detection and visuomotor control/policy learning. The authors also propose an autoregressive value prediction instead of the traditional bellman returns.", "questions": "Overall, I enjoyed reading the paper, but I'm just concerned with the strength of the experiment section. I'm willing to consider changing my score if the experiments section is made stronger.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros:\n1. The high-level of the paper is easy to understand because of the effort that the authors took to portray it on the figures.\n2. The experiments that were performed to evaluate the method are easy to see how the proposed method is better than the baseline/(baselines in some cases).", "weaknesses": "Weakness\n1. Currently, the block diagram only provides a high-level idea of what the paper is about, which is good. But it would to nice to have the language component of the model. For example, I get that you are trying to predict the value estimate of a frame, but without the task description, it’d be impossible to predict whats the value estimate.\n2. The experiments are very limited, although the use cases are diverse enough. In each of these use cases, however, the comparisons are quite restricted. My understanding is that you are using a policy that is learnt on the downstream task. There are so many representation learning works that do that and I urge the authors to find more recent baselines for the use cases, especially, for the visuomotor control\n3. Why are the authors not comparing with LIV on the visuomotor tasks?\n4. Is there a plot/table that shows how much incorporating observation shuffling helps or using autoregressive value prediction helps than the traditional bellman returns? I think this is important to understand the efficacies of the 2 proposed incorporations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r1T0cF7zTG", "reviewer": "Reviewer_tm1A", "summary": "This paper presents Generative Value Learning (GVL) a value function estimator for predicting task progress from visual trajectories. They show that without any task-specific training, their approach can in-context zero-shot and few-shot predict values for more than 300 real-world tasks across different embodiments. The approach is simple: autoregressive prediction, shuffle inputs, and provide in-context examples of task progress.", "questions": "- Can GVL be applied for online RL training?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Well motivated and set desideratas for a “universal value estimator”: accurately estimating state and consistency, highlighting the limitations of prior works\n- Interesting finding; shuffling the video frames help mitigate the temporal bias found in videos ⇒ better value estimations\n- Introduces a new evaluation metric: Value-Order Correlation which measures how well predicted values correlate with the ground-truth timestep in expert videos.\n- Extensive evaluation of GVL’s value estimates across a large suite of real-world robot datasets with varying tasks and embodiments\n- GVL can be applied to a wide variety of tasks such as dataset quality estimation, success detection, and weighted imitation learning.\n- Good analysis and ablation studies to verify the importance of each component in the GVL approach and robustness to different environment factors.", "weaknesses": "- The real-world results are not too impressive. Applying GVL to compute values only leads to a small improvement over the based diffusion policy baseline. It would be interesting to see more results showing how the value estimations of GVL help with downstream policy learning.\n- Method itself is technically simple, but still well-motivated and supported with empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YVbKbKtqdA", "reviewer": "Reviewer_nm81", "summary": "This paper casts universal value estimation of video frames as a temporal ordering problem over shuffled video frames. To address this, the authors introduce Generative Value Learning (GVL), which leverages the world knowledge embedded in vision-language models (VLMs) to estimate task progress. Here, the value for each frame is represented as the VLM's prediction of task completion rate. Experimental results demonstrate that GVL can be effectively applied to various downstream tasks, using a Value-Order Correlation (VOC) metric that measures the rank correlation between the rank of estimated values and the original frame order.\n\nOverall, I believe this paper is suitable for publication at ICLR, though there is room for further development, especially in the clarity of the writing.", "questions": "- In Figure 1, some arrows are bidirectional. Replacing these with unidirectional arrows could improve consistency.\n- In the Dataset Quality Estimation paragraph of Section 4.3, the reasoning feels incomplete. It may strengthen the experiment to directly train models with and without specific low-quality datasets and compare the resulting performance outcomes.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Framing value estimation as a temporal ordering problem over shuffled videos is a compelling approach.\n- The proposed rank correlation metric is thoughtfully designed and well-suited to this problem setting.\n- The experiments are extensive, covering a wide range of real-world tasks.", "weaknesses": "- My primary concern is with the method's applicability. As I understand, calculating the value for each frame requires access to a complete video, which could limit its use in online policy learning without an offline dataset or expert demonstrations. This restricts the downstream tasks to scenarios where offline datasets are available, while the compared method [1] remains applicable in online policy learning.\n- The manuscript contains several typos and minor errors that should be addressed. I recommend a careful proofreading. A non-exhaustive list includes:\n    - Line 163: missing comma.\n    - Line 178: missing space after “2).”\n    - Line 197: \"meaningfully\" should be \"meaningful.\"\n    - Line 393: \"ataset\" should be \"dataset.\"\n\n[1] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pp. 23301–23320. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)