Vision Language Models are In-Context Value Learners

## Abstract

Abstract Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estimator, or temporal value function, across different tasks and domains requires both a large amount of diverse data and methods which can scale and generalize. To address these challenges, we present Generative Value Learning (GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Naively asking a VLM to predict values for a video sequence performs poorly due to the strong temporal correlation between successive frames. Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks across diverse robot platforms, including challenging bimanual manipulation tasks. Furthermore, we demonstrate that GVL permits flexible multi-modal in-context learning via examples from heterogeneous tasks and embodiments, such as human videos. The generality of GVL enables various downstream applications pertinent to visuomotor policy learning, including dataset filtering, success detection, and advantage-weighted regression – all without any model training or finetuning.

## 1 Introduction

Predicting temporal progress from visual trajectories is an important task for embodied agents that interact with the physical world. A robot capable of generalizable progress estimation can in principle discern desirable and undesirable behaviors to learn visuomotor skills in new environments. This is most often studied in reinforcement learning literature [], where progress estimation is equivalent to universal value learning under specific choices of reward function. However, universal value estimation comes with a number of key challenges: (1) broad *generalization* to new tasks and scenes, (2) the ability to *accurately estimate state* in partially observed environments, and (3) temporal *consistency* (i.e. satisfying the Bellman equation) over long horizons. Most existing methods trained on relatively small amounts of vision-only data [, , ] lack the semantic, spatial, and temporal understanding needed to ground task progress in the space-time manifold of video, preventing generalization. Moreover, they often reason over single frames, inducing a high-degree of uncertainty in partially observed environments which in turn can effect the consistency of predictions for poorly estimated states. However, these challenges are not insurmountable: modern vision language models (VLMs) exhibit marked generalization and reasoning capabilities, potentially making them useful for value estimation.

Figure: Figure 1: Result highlights. GVL can effectively zero-shot and few-shot predict task progress on diverse and challenging real-world tasks; these capabilities enable expansive set of downstream applications, including dataset filtering, success detection, and policy learning.
Refer to caption: x1.png

Though not often considered as candidates for value estimation, VLMs excel at its aforementioned core challenges. First, state-of-the-art VLMs have exhibited strong spatial reasoning and temporal understanding capabilities across various vision tasks [, , , ], allowing them to *generalize* to novel scenarios. Second, large transformer-based VLMs have the requisite context window [] to reason over large amounts of historical information to *accurately estimate state* from observation sequences when predicting task progress. Finally, VLMs make predictions auto-regressively, meaning they commit to their own outputs as inputs for subsequent predictions, imposing *consistency* constraints on long generations. For example, a VLM is unlikely to estimate that a task is 50% completed if it already has a 50% completion prediction in context. However, how exactly a VLM should be used to predict values is unclear. Empirically, we find that simply placing a video in-context and prompting the model to return progress predictions for each frame fails – our analysis suggests strong temporal correlations between successive frames often cause VLMs to produce uninformative monotonic values that disregard the actual quality of the trajectory and differences between frames (Section 4) – and a different approach is needed.

Figure: Figure 2: Method overview. Generative Value Learning (GVL) generates values by auto-regressively predicting task completion percentage over shuffled frames, enabling impressive in-context value learning.
Refer to caption: x2.png

To effectively leverage the broad knowledge of VLMs, we introduce Generative Value Learning (GVL), a universal value estimation method enabled by long-context VLMs, which crucially operates over shuffled frames. At its core, GVL asks frozen state-of-the-art VLMs, such as Gemini-1.5-Pro [], to auto-regressively predict the completion percentage of a task specified in natural language for a sequence of shuffled input video frames; see Fig. 2. Perhaps surprisingly we find that simply shuffling the frames of the input video effectively overcomes the strong implicit temporal bias found in video, enabling VLMs to generate meaningful values. While GVL is capable of generating values in a zero-shot manner, we find that the performance of GVL scales with examples via multi-modal in-context learning. Providing more examples of visual “unshuffling” in context increases performance, irrespective of the target embodiment. For example, human videos can improve GVL’s performance on predicting robot task progress.

To facilitate large-scale value prediction evaluation, we additionally introduce a new evaluation metric, Value-Order Correlation (VOC), measuring how well predicted values correlate with the ground-truth timestep order in expert videos; as we will show, VOC is also a useful metric for measuring dataset and trajectory quality, which allows GVL to be used for applications beyond value-based policy learning such as data quality estimation and success detection. We first evaluate GVL’s value prediction quality with VOC on a large suite of real-world robotics datasets, spanning 51 datasets, 20 embodiments, and more than 300 tasks. This includes 50 datasets from Open-X (OXE) dataset  [] in addition to our own bimanual manipulation dataset containing 250 challenging real-world tasks on an ALOHA platform [], which are considerably longer horizon and more fine-grained than those in the OXE dataset. In aggregate, GVL exhibits strong zero-shot value prediction capabilities with highly positive VOC scores on most datasets; its performance further improves with various types of multi-modal in-context examples. Using GVL, we demonstrate scalable foundation model supervision for robot learning at various data abstraction levels. Specifically, GVL can help measure dataset quality in OXE. Second, it can be used for success detection, enabling imitation learning on mixed-quality datasets. Finally, the raw value estimates from GVL can be used for advantage-weighted regression for real-world offline reinforcement learning [, ].

In summary, our contributions are

- 1.
Generative Value Learning (GVL), a universal value prediction framework via VLM in-context autoregressive value estimation on shuffled video frames.
- 2.
An extensive evaluation on real-world datasets demonstrating GVL’s zero-shot scalability and multi-modal in-context learning capabilities.
- 3.
Demonstration that GVL can be used in downstream applications including dataset quality estimation, success detection, and advantage-weighted regression for real-world control.

## 2 Related Work

Reward and value foundation models.
Several works have tried to learn transferable reward and value functions from diverse data. Early works learned models using robot [] or even human videos with discriminators [], contrastive learning [] or offline RL [, , ] to guide manipulation tasks. With the advent of recent language and vision foundation models, several works have integrated them into various robotic applications such as semantic planning [, , , , ], imitation learning  [, ], and symbolic programming [, , , , , , , , , ]. Most related to our work, LLMs and VLMs have been used as reward models. Kwon et al. [], Mahmoudieh et al. [] use language models to provide reward values for RL agents, while Klissarov et al. [], Wang et al. [], Kwon et al. [] use them to provide preference feedback. Ma et al. [], Yu et al. [], Xie et al. [] even have LLMs generate their code. These works use only the language capabilities of foundation models. More recent works directly use VLMs as zero-shot reward models [] or success detectors [, ]. Critically, in these works the VLM acts only as an (often sparse) reward function which predicts success, and not a *value* function that predicts task progress. Though some works use chain-of-thought prompting [] or active learning [], they generally do not make use of the autoregressive, long-context, or in-context learning capabilities of state-of-art VLMs. As a consequence, they often evalaute *reward* prediction only on simple and simulated tasks. To our knowledge, we are the first to demonstrate that VLMs are capable of generalizable per-frame value estimation on real world tasks which can be used for downstream tasks like dataset selection.

In-context learning for robotics. In-context learning has been explored in the robot learning literature, primarily focusing on action generation [, , , , , , ]. However, all these prior works require explicit, and often extensive training, on their robot tasks in order to realize in-context learning capabilities, and generalization is achieved only on narrow distribution of tasks. In contrast, we demonstrate that visual value estimation already enjoys flexible multi-modal in-context learning from pre-trained VLMs without any robot specific fine-tuning.

## 3 Generative Value Learning

In this section, we introduce Generative Value Learning, GVL. At a high level, GVL frames value estimation as an autoregressive next-token prediction problem in which a VLM is tasked with outputting the task progress for a batch of shuffled trajectory frames.

Problem setup. We model robotics tasks as goal-conditioned partially observed Markov decision processes []: $\mathcal{M}(\phi):=(O,A,R,P,T\mu,G)$ with observation space $O$, action space $A$, reward function $R$, transition function $P$ , task horizon $T$, initial state distribution $\mu(o)$, and goal space $G$ that specifies the task semantically. Conditioned on a task $g$ an agent $\pi:O\rightarrow A$ aims to maximizes its value function, or the expected cumulative reward over the task horizon, $V^{\pi}(o_{1};g)=\mathbb{E}_{\mu,\pi,P}[r(o_{1};g)+\dots+r(o_{T};g)]$. However, reward and value functions can be difficult to define for robotics applications given their heterogeneity. Given this, a popular universal notion of value is task progress [, , , , ]. This kind of temporal value function maps an observation and goal specification to a real number between 0 0 and $1$: $V:\mathcal{O}\times\mathcal{G}\rightarrow[0,1]$, where initial observations of the environment have value 0 0 and goal-satisfying observations have value $1$. Under this definition, an expert trajectory $\tau=(o_{1},\dots,o_{T})\sim\pi_{E}$, has value function $V^{\pi_{E}}(o_{t};g)=\frac{t}{T}$. In this work, our goal is to obtain such a temporal value function $V$ that can predict such task progress $v_{1},\dots v_{T}$ for each frame of video $o_{1},\dots,o_{T}$.

Though we seek to leverage priors imbued in large foundation models, as shown in Section 4 simply prompting a VLM with video frames fails to produce meaningful estimates. To make VLMs amenable to value prediction, we propose three key components that comprise the GVL method: 1) autoregressive value prediction, 2)input observation shuffling, and 3) in-context value learning.

1. Autoregressive value prediction.
Traditionally, value functions $V(\cdot):\mathcal{O}\rightarrow\mathbb{R}$ are trained to be self-consistent by enforcing the bellman equation

$$ $V^{\pi}(o_{t})=R(o_{t})+\mathbb{E}_{\pi,P}\left[V(o_{t+1})\right].$ (1) $$

When parameterizing a value function as a feed-forward neural network, this is typically done by minimizing the mean-squared error of the equality above. As values for different observations within the same trajectory are related via the bellman equation, the resulting value function remains consistent even if we query it with only a single observation. VLMs on the other hand are not inherently trained with any consistency objective. Thus, if we independently query a VLM with different observations from the same trajectory it is likely to produce inconsistent values.
Our insight is that providing the entire trajectory as input instead of just a single observation offers VLMs greater opportunity to generate self-consistent value estimates. Concretely, given a language description of the task $l_{\text{task}}$ we ask the VLM to auto-regressively generate values given the entire video as context:

$$ $v_{t}=\textrm{VLM}(o_{1},\dots,o_{T};v_{1},\dots,v_{t-1};l_{\text{task}}), \forall t\in[2,T].$ (2) $$

We abbreviate this auto-regressive prediction process as $v_{1},\dots,v_{T}=\textrm{VLM}(o_{1},\dots,o_{T};l_{\text{task}})$. This simple mechanism allows the VLM to attend to all previous predictions and frames when making the next value prediction, enabling it to produce globally consistent estimates over long-horizon sequences without needing to be trained like classical feed-forward value functions. Though this design choice enables VLMs to produce consistent values, it doesn’t necessitate that the values are meaningfully. Naiv̈ely prompting a VLM in this manner tends to produce linear, monotonic value functions for every single video, regardless of optimality.

2. Input observation shuffling. Empirically we find that when presented a choronological sequence of frames VLMs discover the short-cut solution of outputting monotonically increasing values, often ignoring the task description or the actual quality of the trajectory. One hypothesis is that as VLMs are trained on ordered video frames for captioning and question answering, the chronology itself is a cue for downstream tasks unrelated to value prediction. As a consequence, model naiv̈e prompting results in unfaithful low-quality value predictions.
To break this temporal bias, we propose randomly shuffling the input frames. In this manner, GVL forces the VLM to pay attention to each individual frame and output faithful value predictions using all information provided in context. Concretely, GVL prompts a VLM as:

$$ $v_{\tilde{1}},\dots,v_{\tilde{T}}=\textrm{VLM}(o_{\tilde{1}},\dots,o_{\tilde{T }};l_{\text{task}},o_{1}),\quad\text{where}\quad({\tilde{1}},\dots,{\tilde{T}} )=\texttt{permute}(1,\dots,T).$ (3) $$

where the permute operator randomly shuffles the temporal indicies. Note however, that we cannot shuffle every frame. If we do so, then the arrow of time in the original video can be ambiguous – i.e., in many cases, the reverse video is also physically plausible, making is the ground-truth order impossible to predict. Thus, as in the above equation we condition the VLM on the first input frame allow it to use the first observation as an anchor point for all other shuffled frames.

3. In-context value learning.
While auto-regressive prediction and shuffling are enough to obtain good performance, GVL can perform even better by leveraging the appealing properties of VLMs. Notably, large models often exhibit in-context learning, where tasks can be learned by simply providing examples []. This enables flexible and versatile in context value learning, by which GVL’s predictions can steadily improve by providing examples at test time without any model fine-tuning. In particular, we can simply prepend shuffled videos and their ground-truth task progress as in-context examples to boost the value prediction quality via few-shot learning:

$$ $v_{\tilde{1}},\dots,v_{\tilde{T}}=\textrm{VLM}\left(o_{\tilde{1}},\dots,o_{ \tilde{T}},l_{\text{task}}\mid\texttt{permute}\left((o_{1},v_{1}),(o_{2},v_{2} ),\dots,(o_{M},v_{M})\right)\right)$ (4) $$

As we show in Section 4, GVL benefits from flexible forms of in-context examples, including videos from unrelated tasks and even humans. Though GVL zero-shot is already effective across a broad range of tasks and robots, in-context learning can still realize substantial improvement on the most difficult bimanual dexterous tasks.

Practical implementation.
To predict temporal value functions in practice, GVL asks the VLM to output integer-valued percentage numbers between 0 and 100. Given that real-world robot video datasets are of different lengths and taken at different frequencies, we subsample all videos so that there are 30 frames in the input sequence to ensure comparable findings across datasets. See the Appendix for the full prompt and implementation.

## 4 Experiments

We conduct large scale experiments assessing GVL’s value prediction generalization and in-context learning capabilities. Specifically, we study the following questions:

- 1.
Can GVL produce zero-shot value predictions for a broad range of tasks and embodiments?
- 2.
Can GVL improve from in-context learning?
- 3.
Can GVL be used for other downstream robot learning applications?

In all our experiments, we use Gemini-1.5-Pro [] as the backbone VLM for GVL; we ablate this model choice and find GVL effective with other VLMs as well. After thorough study of GVL’s value prediction capabilities, we study several downstream applications in visuomotor policy learning, aiming to improve data quality at dataset, trajectory, and individual transition levels.

Evaluation metric.
Our goal is to evaluate GVL value estimation at scale on as many robot datasets as possible, holistically testing its generalization capabilities and understanding its limitations. This makes it difficult to use traditional evaluation metrics for value functions, such as observing downstream learned policy performance, as they require value functions that are specifically trained or finetuned for individual tasks and embodiments. This quickly becomes very expensive for universal value functions that are intended for use across a large set of diverse real-world tasks and robots, many of which the practitioner may not have access to. Prior works on large-scale value learning have resorted to visually observing the smoothness of the value curve on expert trajectories as a qualitative “eye-test” for model generalization [, , ], but such evaluation is conducted on only few selected videos. We formalize and scale up this intuitive approach and introduce
a lightweight, yet predictive method for evaluating value models: Value-Order Correlation (VOC). This metric computes the rank correlation between the predicted values and the chronological order of the input expert video:

$$ $\texttt{VOC}=\texttt{rank-correlation}\left(\texttt{argsort}(v_{\tilde{1}}, \dots,v_{\tilde{T}});\texttt{arange}(T)\right);$ (5) $$

VOC ranges from $-1$ to $1$, where $1$ indicates that the two orderings are perfectly aligned. Expert quality demonstrations, by construction, have values that monotonically increase with time, and thus a good value model should have high VOC scores when evaluated on expert videos. On the other hand, fixing a good value model, low-quality trajectories should have low VOC scores. This is because sub-optimal trajectories often contain high repetition of visually similar frames due to the presence of redundant, re-attempt actions or poorly-placed cameras. As such, the values along the trajectories should not be monotonic, resulting in low correlation with the ground-truth timestep order. As we will show in our experiments, this value rank correlation metric has strong predictive power for the quality of the values as well as downstream policy learning performance, validating its usefulness as a standalone evaluation metric for value predictions.

### 4.1 Large-scale real-world evaluation

To study GVL’s zero-shot value prediction capability, we evaluate its VOC on two large expert robotics datasets.

Open X-Embodiment dataset. First, we consider the Open X-Embodiment (OXE) dataset []. an aggregation of trajectory data from 50 standalone academic robot datasets that consists of diverse tasks, robots, and camera viewpoints. For each of the 50 datasets, we randomly sample 20 trajectories and evaluate GVL zero-shot on each of the sampled trajectories. Note that not all OXE datasets have language task annotations, so we use the last frame of the trajectory as the goal specification when text annotation is not provided. To better contextualize GVL’s value prediction quality, we compare to a state-of-the-art multi-modal value model LIV [], a contrastive vision-language model [] fine-tuned with value learning objective on human videos for in-the-wild value estimation. LIV predicts the temporal value of an input observation by computing its embedding distance to the embedding of the goal image or task description.

For evaluation, we plot the histogram of all 1000 (50$\times$20) Value Order Correlation (VOC) scores in Fig. 3, split by goal modalities. Given that most OXE datasets contain human-collected expert demonstrations, good value models should have high VOC scores; however, we acknowledge that there are sub-optimal trajectories within OXE that can introduce noise in our results. After we first establish GVL as an effective universal value model. we will present how GVL can be used to detect low-quality data in Section 4.3. As shown in Fig. 3, on both goal modalities, GVL consistently generates VOC scores that heavily skew to the right, indicating that it is able to zero-shot recover the temporal structure hidden in the shuffled demonstration videos, i.e., coherent value predictions. GVL’s performance is also markedly better than LIV on language goals (Fig. 3 left). Here, LIV’s predictions are random, suggesting that its embedding space does not contain sufficient knowledge for predicting dense values for arbitrary unseen robot videos. On image goals, LIV’s prediction problem is arguably simpler because an embedding space that simply captures image similarity can result in ascending values that correlate with timesteps. Even then, GVL generates better quality value predictions as judged by slightly higher VOCs (Fig. 3 right).
In summary, GVL can indeed effectively utilize the world knowledge afforded by the backbone VLM to achieve effective value predictions zero-shot for the breadth of real-world robotic tasks and datasets.

Figure: Figure 3: Zero-shot value predictions on OXE datasets. Left: GVL significantly outperforms LIV on datasets with language goals. Right: GVL still outperforms LIV on datasets with image goals despite solving the more difficult task of frame re-shuffling.
Refer to caption: extracted/5984057/figures/correlation_histogram_text_goal.png

Challenging bimanual datasets.
OXE datasets primarily focus on simpler, short horizon single-arm tasks.
To further stress test GVL, we evaluate on a new diverse dataset of 250 distinct household tabletop tasks on the bi-manual ALOHA systems [, ]. This dataset includes highly challenging, long-horizon skills, such as removing three gears sequentially from a NIST board, folding a dress in eighth-fold, hanging a t-shirt on a cloth rack. See the bottom right of Fig. 2 for representative ALOHA tasks.
For each task, we evaluate on 2 human teleoperated demonstrations to evaluate GVL zero-shot. The aggregate histogram over all 500 (250 $\times$ 2) VOC scores is illustrated in Fig. 4. As shown, GVL is capable of generating value predictions that are positively correlated on more than 60% of them with median VOCs of 0.12. This is promising, but worse than the performance on the OXE datasets. While our main results use the top-down camera shown in Fig. 6, we show the effect of using GVL across each of the four cameras used in our ALOHA setup in Fig. 15. In Fig. 5, we present several qualitative examples of GVL predictions; see our [project website](https://generative-value-learning.github.io/) for additional examples. In the next section, we extensively explore how to improve GVL on this dataset using in-context learning techniques.

Figure: Figure 4: GVL scales up to 250 ALOHA bi-manual tasks and can improve with in-context examples.
Refer to caption: x3.png

### 4.2 Multi-Modal In-Context Value Learning

As the diverse ALOHA dataset is significantly more challenging, we explore whether GVL can benefit from in-context learning, where additional shuffled observation-value pairs are presented in the VLM context window (Eq. 4).

Few-shot in-context learning. First, we collect an additional demonstration for each of the 250 tasks and use its shuffled value-observation pairs as context for one-shot GVL value prediction for the same set of 500 evaluations. As seen in Fig. 4, with one in-context trajectory, GVL’s performance substantially improves with 90% positive VOCs and a median VOC of 0.37. We further investigate whether performance can improve with more in-context examples on a represented subset of 13 tasks for which have more than $500$ demonstrations. For these tasks, we evaluate few-shot GVL on 500 distinct trajectories per task with up to 5 in-context examples. The average VOCs over tasks and trajectories is shown in Fig. 4 (Right). We see that GVL demonstrates appealing in-context scaling as the average score steadily improves as we increase the number of in-context examples. Even with 5 in-context trajectories, meaning 150 total shuffled images, GVL is able to utilize its full context and exhibit strong generalization. This result demonstrates how state-of-art long-context-window VLMs, such as Gemini-1.5-Pro, can be re-purposed to make for general-purpose value functions with impressive test-time improvement capability, quickly mastering value predictions with minimal supervision.

Figure: (a) fold_dress from the top-down view.
Refer to caption: extracted/5984057/figures/raw_output/fold_dress.png

Cross-embodiment in-context learning.
Examples in-context are not limited to robot demonstrations. One advantage of GVL is that it can still benefit from in-context learning even when the demonstrations come from a different embodiment. Specifically, we record humans performing the same tasks as the ALOHA robot demonstrations and then use these human demonstrations as in-context examples for value prediction. As shown in Fig. 6, GVL with one cross-embodiment in-context example can effectively improve over its zero-shot counterpart. In the Appendix, we also show that GVL can similarly benefit from cross-task in-context learning. In conclusion, GVL presents a versatile framework for in-context value learning that can scale up to even the most challenging manipulation tasks.

Figure: (a) In-context examples from human videos
Refer to caption: extracted/5984057/figures/cross_embodiment/human_frame_001_cropped.jpg

### 4.3 GVL Applications

As GVL can generate high-quality value estimates, it can be applied to a number of downstream tasks including dataset quality estimation, success detection, and weighted imitation learning.

**Table 1: Average Value-Order Correlation (VOC) scores on selected OXE datasets. As shown, demonstration datasets with un-occluded camera views generally have high scores. In contrast, exploration datasets have low scores.**
| Dataset | Avg. VOC |
| --- | --- |
| RT-1 [] | 0.74 |
| Dobb-E [] | 0.53 |
| Bridge [] | 0.51 |
| QT-OPT [] | 0.19 |
| DROID [] | -0.01 |
| RoboNet [] | -0.85 |

Dataset Quality Estimation. Robotic action models are increasingly trained on large mixtures of datasets [, , ] and selecting the right mixture is critical for policy performance []. However, dataset mixing is often done in an ad-hoc fashion by visual inspection []. Having validated that GVL is an effective zero-shot value model, we investigate whether we can in turn use GVL’s VOC scores to determine dataset quality within OXE.
To this end, for each OXE dataset in Fig. 3, we compute the average correlation scores for its sampled trajectories and present the ranking of the average score in Appendix B. In Table 1, we present a subset of selected representative large-scale datasets in OXE. We see that datasets have large spread in their VOC scores, but these scores are interpretable and match human intuitions. Specifically, datasets collected from human teleoperators with relative fixed camera placements, such as RT-1 [], Dobb-E [], and Bridge [, ], have high VOC scores, despite their diversity in scenes and tasks. In contrast, datasets with autonomous data collection via scripted motions or motor babbling, such as QT-OPT [] and RoboNet [], contain high number of suboptimal trajectories that do not exhibit smooth temporal structure to be re-shuffled.

Interestingly, DROID [], a recent large household manipulation ataset is ranked very low, consistent with prior works  [] that found that removing DROID from large action model training improved final performance. After inspecting trajectories from DROID with a low VOC score from GVL we found that many have poor camera angles that do not capture robot motion or have the arm or manipulated objects heavily occluded. These observations indicate that GVL VOC can be indicative of dataset quality.

**Table 2: Comparison of VLM success detectors.**
| Method | Accuracy | Precision | Recall |
| --- | --- | --- | --- |
| GVL-SD (Zero-Shot) | 0.71 | 0.71 | 0.71 |
| GVL-SD (One-Shot) | 0.75 | 0.85 | 0.70 |
| SuccessVQA [] | 0.62 | 0.33 | 0.73 |
| SuccessVQA-CoT | 0.63 | 0.44 | 0.68 |

Success detection and filtered imitation learning. Next we consider more granular intra-dataset quality control by investigating how GVL can be used as a success detector for trajectory filtering, enabling filtered imitation learning on mixed quality datasets. As discussed, good value models should return low VOC scores on unsuccessful trajectories; in particular, it is difficult for GVL to re-shuffle frames within sub-optimal trajectories which often contain irregular or repetitive behavior.
Thus, we can use GVL for success detection by filtering trajectories that have VOC scores below certain numerical threshold; we refer to this procedure as GVL-SD. We evaluate GVL-SD on six simulated bimanual dexterous manipulation tasks on the ALOHA system (see Fig. 10). Simulation is well-suited for this experiment because we can naturally control for data quality and reproducibility. More specifically, for each task, we construct a mixed quality dataset by rolling out a pre-trained policy of roughly 50% success rate for 1000 episodes, mirroring real-world autonomous data collection settings with high failure rate []. We compare to SuccessVQA [], which poses success detection as a Visual-Question Answering problem. To ensure that the same amount of information is provided, we feed the full video sequence to the VLM; therefore, this baseline tests whether the VLM is equipped with video understanding capability good enough for out-of-the-box success detection. In addition, we consider SuccessVQA-CoT, which uses chain-of-thought prompting [] to encourage the VLM to output intermediate textual reasoning outputs before providing the final success answer. Unless otherwise stated we use a VOC threshold of $0.5$ for GVL-SD.

Figure: Figure 7: GVL for Success Detection. Left: GVL behaves qualitatively differently on successful and failed trajectories. Right: GVL (No-Shuffling) loses discriminability on failure trajectories.
Refer to caption: x6.png

**Table 3: Real-World ALOHA Policy Learning Results. AWR with GVL (One-Shot) outperforms IL baselines when the predicted values have high VOCs.**
| Real-World ALOHA Tasks | GVL + DP | DP | Avg. VOC |
| --- | --- | --- | --- |
| bowl-in-rack | 7/10 | 6/10 | 0.57 |
| banana-handover | 7/10 | 5/10 | 0.73 |
| close-laptop | 9/10 | 6.5/10 | 0.59 |
| open-drawer | 4/10 | 6/10 | 0.09 |
| remove-gears | 4.67/10 | 7/10 | 0.19 |
| pen-handover | 1.5/10 | 0/10 | 0.43 |
| fold-dress | 7/10 | 7/10 | 0.66 |

For all methods, we report the accuracy, precision, and recall in  Table 2. GVL-SD consistently outperforms or matches SuccessVQA on all classification metrics. In particular, SuccessVQA has low precision, indicating that the base VLM systematically biases towards outputting failure. Adding one in-context demonstration further improves GVL’s performance across all metrics. In Fig. 7 (Left), we also visualize the histogram of the VOC scores GVL produces on success and failure trajectories. As expected, GVL on failure trajectories renders a uniform distribution when the task is unsuccessful, indicating the model’s inability to uncover the original temporal order – success and failure trajectories have distinct distributions over the correlation values indicating that GVL can adequately separate them. Fig. 7 (Right) shows that the histograms without shuffling are largely the same independent of success or failure. This shows that by forcing the VLM to perform the more difficult prediction task over shuffled frames, GVL can elicit better zero-shot values.

Figure: Figure 8: Success-Filtered Imitation Learning on ALOHA Simulation Tasks. Left: Using GVL-SD for success-filtered BC substantially outperforms SuccessVQA. Right: GVL-SD is not sensitive to the VOC threshold for improving imitation learning.
Refer to caption: x8.png

Now, we use the above success detection methods for filtered imitation learning; for all methods, we use Action Chunking Transformer (ACT) as the imitation learning algorithm []; ACT hyperparameters are tuned for ACT on the success-only subset and are fixed for all methods. Given the noisiness in model checkpoints performance, we report the average success rate of the last 10 model training checkpoints. Results for the six simulation tasks are shown to the right of Fig. 8, where GVL-SD’s improved success detection leads to better performance over SuccessVQA. In fact, SuccessVQA often hurts performance, likely because of its low precision which causes the policy to train on a high number of false positive (i.e. failure) trajectories. In Fig. 8 (Right) we show the effect of varying the VOC threshold in $\{-1.0,0,0.25,0.5,0.75\}$ in comparison to training on all the data with ACT; note that this is the same using the lowest threshold, $-1.0$ as it is a lower bound on the VOC metric. As seen, GVL consistently outperforms ACT regardless of threshold values; when the threshold value is too high, i.e., $0.75$, we see a slight dip in performance when the overall dataset size becomes too small.

Figure: (a) banana-handover
Refer to caption: extracted/5984057/figures/real_tasks/banana_handover_cropped.png

Advantage-weighted regression for real-world visuomotor control.
Finally, we illustrate how GVL can assign importance weights to individual transitions within trajectories at a fine-grained level akin to offline reinforcement learning. For these experiments we use real-world demonstration data collected by human teleoperation on bi-manual ALOHA robot setups. Unlike simulation, our datasets only contain successful task executions but can be sub-optimal and multi-modal. Thus, we directly utilize GVL’s values with advantage weighted regression (AWR) [, ], in which we weight each individual transition by the estimated advantange, or GVL value difference for that step:

$$ $\mathcal{L}(\theta):=-\mathbb{E}\left[\mathrm{exp}\left(\tau\cdot(v_{k+1}-v_{k })\right)\cdot\log\pi_{\theta}(a_{k}\mid o_{k})\right]$ (6) $$

If an action were deemed to make progress towards the task goal, then its future value $v_{k+1}$ should be significantly higher than the present value $v_{k}$, leading to a large positive weight. By upweighting the most promising actions in diverse human-collected datasets, we posit that AWR can outperform imitation learning approaches.

We use diffusion policy (DP) as the policy backbone [] for each task, and compare training diffusion policies with GVL (One-Shot) advantage weighting or lack thereof. We evaluate on 7 tasks with 10 trials per task and report success rate in Table 3. As can be seen, on a majority tasks, GVL-DP outperforms DP and we see a clear correlation between improvement over DP and the VOC score. That is, when the value predictions are of high quality as judged by VOC, policy learning can benefit from GVL value weighting. On open-drawer and remove-gears, the top-down view does not provide sufficient resolution to distinguish task progress (see Fig. 9), as a consequence, the value predictions can be noisy, which can hurt policy learning. However, given the in-context learning results, we believe that it is possible to improve policy learning even on difficult tasks with non-ideal camera viewpoints.

### 4.4 Ablations

Finally, we ablate key algorithmic design choices of GVL to validate their necessity. In the Appendix, we additionally demonstrate that GVL’s performance is robust to the choice of backbone VLMs as well as input
camera viewpoint.

Is autoregressive value prediction necessary?
We consider an ablation that simply asks the VLM to predict values of input observations one by one without GVL’s autoregressive batch prediction mechanism. This ablation, which we refer to as VLM (Single Frame), essentially poses value estimation as a VQA problem. We compare this ablation to GVL on a subset of RT-1 dataset as in Section 4.1; the average VOC for VLM (Single Frame) is a mere $-0.08$, a significant drop from GVL’s 0.74 on RT-1 dataset. As seen, pre-trained VLMs by themselves are poor value estimators, generating inconsistent values that are too noisy to be used in practice.

Is input observation shuffling necessary?
As discussed, we find that removing shuffling collapses ICV’s predictions into generating degenerate values; that is, regardless of the quality of the provided trajectory, GVL tends to predict monotonically increasing values, resulting in inflated VOC scores that cannot be used to discriminate successful and failure trajectories.; see Fig. 7 (Right). To further qualitatively illustrate this phenomenon, in Fig. 11 in the Appendix, we overlay raw GVL value predictions with frame shuffling and lackthereof to understand the spread of the value curves. We see that the overlay for original GVL looks “messy”, suggesting that GVL outputs varied value curves that better capture the heterogeneity of the queried video qualities. In contrast, without frame shuffling, GVL predictions indeed collapses onto a few linear ascending patterns.

## 5 Conclusion

We have introduced Generative Value Learning (GVL), a universal value function via VLM autoregressive value prediction on shuffled video frames. GVL can zero-shot output dense and high-quality value predictions for diverse and challenging real-world robotic tasks, spanning various robot embodiments and task categories. With few-shot learning from the same task, different task, or different embodiment, GVL performance steadily improves. We have demonstrated several use cases of using GVL to perform dataset, trajectory, and transition selection to improve downstream policy learning performance and generalization. We believe that GVL takes an important step in using foundation models supervision for robot learning.

Limitations and future work. We have not investigated whether pre-trained VLMs can be fine-tuned to perform better value predictions. In addition, though we test on diverse camera viewpoints, we have not yet investigated whether multi-view observations can improve value prediction quality. In addition, our evaluation metric Value-Order Correlation is most suitable for a-periodic tasks for which there exists a unique ordering of frames from an expert demonstration. Tasks such as wiping or stirring may be hard to discern. Though these limitations present avenues for future work, we believe GVL is a step towards improved in-the-wild value estimation.

## Acknowledment

We thank Jie Tan, Pannag Sanketi, Oliver Groth, and the rest the Google DeepMind Robotics team for helpful discussions and providing feedback on the paper.

## Appendix A Prompt

In this section, we provide the full prompt provided to the VLM for GVL predictions. The same prompt is used for all OXE datasets.

[
linecolor=gray,
linewidth=0.5pt,
roundcorner=10pt,
backgroundcolor=lightgray,
innerleftmargin=5pt,
innerrightmargin=5pt,
innertopmargin=5pt,
innerbottommargin=5pt
]

## Appendix B GVL OXE Dataset VOC Breakdown

In this section, we provide the full list of average VOC score for each OXE dataset. In Table 4, we provide the VOC scores for GVL with Gemini-1.5-Pro as the backbone VLM. In Table 5, we provide the VOC scores for GVL with GPT-4o as the backbone VLM.

**Table 4: GVL (Gemini-1.5-Pro) OXE Dataset VOC Scores**
| Dataset | VOC Score |
| --- | --- |
| utokyo_pr2_opening_fridge_converted_externally_to_rlds | 0.8095 |
| utokyo_xarm_bimanual_converted_externally_to_rlds | 0.7955 |
| utokyo_xarm_pick_and_place_converted_externally_to_rlds | 0.7880 |
| fractal20220817_data | 0.7385 |
| maniskill_dataset_converted_externally_to_rlds | 0.7260 |
| berkeley_autolab_ur5 | 0.7185 |
| nyu_door_opening_surprising_effectiveness | 0.6685 |
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 0.5875 |
| utaustin_mutex | 0.5810 |
| iamlab_cmu_pickup_insert_converted_externally_to_rlds | 0.5585 |
| fmb | 0.5555 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 0.5295 |
| dobbe | 0.5295 |
| toto | 0.5270 |
| bridge | 0.5145 |
| austin_sirius_dataset_converted_externally_to_rlds | 0.5100 |
| asu_table_top_converted_externally_to_rlds | 0.5055 |
| berkeley_rpt_converted_externally_to_rlds | 0.4835 |
| berkeley_cable_routing | 0.4470 |
| usc_cloth_sim_converted_externally_to_rlds | 0.4410 |
| jaco_play | 0.4205 |
| bc_z | 0.4065 |
| viola | 0.4035 |
| berkeley_mvp_converted_externally_to_rlds | 0.3900 |
| roboturk | 0.3545 |
| austin_buds_dataset_converted_externally_to_rlds | 0.3415 |
| stanford_hydra_dataset_converted_externally_to_rlds | 0.3325 |
| tokyo_u_lsmo_converted_externally_to_rlds | 0.3140 |
| berkeley_fanuc_manipulation | 0.2685 |
| cmu_stretch | 0.2625 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 0.2410 |
| kuka | 0.1915 |
| dlr_sara_pour_converted_externally_to_rlds | 0.1600 |
| taco_play | 0.0945 |
| dlr_edan_shared_control_converted_externally_to_rlds | 0.0855 |
| droid | -0.0060 |
| stanford_robocook_converted_externally_to_rlds | -0.0690 |
| imperialcollege_sawyer_wrist_cam | -0.1225 |
| kaist_nonprehensile_converted_externally_to_rlds | -0.1310 |
| austin_sailor_dataset_converted_externally_to_rlds | -0.1715 |
| cmu_play_fusion | -0.3445 |
| stanford_kuka_multimodal_dataset_converted_externally_to_rlds | -0.3770 |
| stanford_mask_vit_converted_externally_to_rlds | -0.4505 |
| nyu_franka_play_dataset_converted_externally_to_rlds | -0.4555 |
| uiuc_d3field | -0.7025 |
| cmu_franka_exploration_dataset_converted_externally_to_rlds | -0.7395 |
| columbia_cairlab_pusht_real | -0.7625 |
| robo_net | -0.8485 |
| dlr_sara_grid_clamp_converted_externally_to_rlds | -1.0000 |

**Table 5: GVL (GPT-4o) OXE Dataset VOC Scores**
| Dataset | VOC Score |
| --- | --- |
| nyu_door_opening_surprising_effectiveness | 0.883 |
| utokyo_pr2_opening_fridge_converted_externally_to_rlds | 0.864 |
| berkeley_mvp_converted_externally_to_rlds | 0.8285 |
| utaustin_mutex | 0.813 |
| fractal20220817_data | 0.803 |
| utokyo_xarm_pick_and_place_converted_externally_to_rlds | 0.7665 |
| berkeley_autolab_ur5 | 0.755 |
| utokyo_xarm_bimanual_converted_externally_to_rlds | 0.749 |
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 0.734 |
| austin_sirius_dataset_converted_externally_to_rlds | 0.7235 |
| toto | 0.713 |
| dlr_edan_shared_control_converted_externally_to_rlds | 0.6595 |
| bridge | 0.6445 |
| berkeley_fanuc_manipulation | 0.6295 |
| berkeley_rpt_converted_externally_to_rlds | 0.6235 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 0.603 |
| roboturk | 0.57 |
| jaco_play | 0.5615 |
| iamlab_cmu_pickup_insert_converted_externally_to_rlds | 0.557 |
| uiuc_d3field | 0.5395 |
| usc_cloth_sim_converted_externally_to_rlds | 0.5355 |
| asu_table_top_converted_externally_to_rlds | 0.5025 |
| maniskill_dataset_converted_externally_to_rlds | 0.499 |
| kaist_nonprehensile_converted_externally_to_rlds | 0.492 |
| viola | 0.4605 |
| austin_buds_dataset_converted_externally_to_rlds | 0.454 |
| cmu_play_fusion | 0.4235 |
| tokyo_u_lsmo_converted_externally_to_rlds | 0.3875 |
| austin_sailor_dataset_converted_externally_to_rlds | 0.3015 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 0.2675 |
| berkeley_cable_routing | 0.255 |
| dlr_sara_pour_converted_externally_to_rlds | 0.252 |
| imperialcollege_sawyer_wrist_cam | 0.239 |
| robo_net | 0.237 |
| stanford_hydra_dataset_converted_externally_to_rlds | 0.205 |
| cmu_stretch | 0.1895 |
| bc_z | 0.176 |
| nyu_franka_play_dataset_converted_externally_to_rlds | 0.1735 |
| stanford_robocook_converted_externally_to_rlds | 0.16 |
| kuka | 0.132 |
| stanford_mask_vit_converted_externally_to_rlds | -0.173 |
| stanford_kuka_multimodal_dataset_converted_externally_to_rlds | -0.1785 |
| columbia_cairlab_pusht_real | -0.1815 |
| cmu_franka_exploration_dataset_converted_externally_to_rlds | -0.2075 |
| taco_play | -0.2705 |
| eth_agent_affordances | -0.279 |
| dlr_sara_grid_clamp_converted_externally_to_rlds | -1 |

## Appendix C Simulation Tasks

In Figure 10, we illustrate the six simulation tasks used for the success detection and filtered imitation learning experiment. For each task, we use VR teleoperation to collect 500 trajectories for initial policy training. After the policy converges, we rollout the last checkpoint for 1000 imtes, resulting in naturally balanced mix-quality datasets of about half success and half failure trajectories.

Figure: (a) single_insertion
Refer to caption: extracted/5984057/figures/sim_tasks/single_insertion_init.png

## Appendix D Additional Results

In this section, we present additional results and analysis.

GVL and No-Shuffling ablation qualitative comparison. As shown in Fig. 11, GVL generates value predictions that are varied over time; in contrast, without frame shuffling, the predictions all collapses onto a few monotonic patterns.

Figure: Figure 11: GVL without shuffling produces uninformative monotonic values regardless of trajectory quality.
Refer to caption: x10.png

Figure: Figure 12: Success-Filtered Imitation Learning on ALOHA Tasks. Left: Using GVL-SD for success-filtered BC substantially outperforms SuccessVQA. Right: GVL-SD is not sensitive to the VOC threshold for improving imitation learning.
Refer to caption: x12.png

Zero-Shot Aloha Sim Results. In Fig. 12 we include results for GVL-SD zero-shot instead of one-shot. The results are qualitatively similar, where GVL-SD consistently outperforms SuccessVQA, and different VOC threshold values all provide performance gain.

Different VLM backbone. We additionally consider GPT-4o as the backbone VLM to better understand GVL’s performance in relation to the backbone VLM model. For evaluation, we plot the histogram of all 1000 (50$\times$20) Value Order Correlation (VOC) scores across all trajectories in Figure 13. As shown, GVL, independent of the backbone models, consistently generates VOC scores that heavily skews to the right, indicating that it is able to zero-shot recover the temporal structure hidden in the shuffled demonstration videos, i.e., coherent value predictions.

Figure: Figure 13: GVL has comparable performances with different backbone VLMs; the main difference is in the backbone model’s refusal rate and conforming to the response template, which is reflected in the tall bar at $-1.0$.
Refer to caption: extracted/5984057/figures/correlation_histogram_gemini_gpt.png

Cross-task in-context learning. We investigate whether examples from other tasks can also unlock GVL’s in-context learning capability. On the previous ALOHA-13 tasks, we randomly pair up tasks, where we draw one demonstration from one task as the one-shot in-context example for another. Then, we compare VOCs with the original same-task one-shot setup. The results are shown in Figure 14. We see that providing examples from a different task is still beneficial, though the improvement is not as much as same-task examples. This is to be expected as intra-task examples still provide clue on the output format as well as a generic notion of task progress, but such information is not specific to the target task. That said, cross-task ICL enables the flexibility of enabling foundation model guidance on a task without any task-specific prior.

Figure: Figure 14: GVL demonstrates cross-task in-context learning capability: its value predictions can be improved by value examples from different tasks.
Refer to caption: x14.png

Figure: Figure 15: GVL works better on more in-distribution table view, but one-shot improvement benefits all camera views.
Refer to caption: extracted/5984057/figures/pdf/aloha4.png

Does GVL work on different camera viewpoints?
On our ALOHA setup, we collected all demonstrations using four camera viewpoints. Besides the top-down view reported in the main experiment above, we test whether GVL remains performant when using alternative viewpoints, especially gripper views that are likely more out-of-distribution with respect to the natural images used for VLM training. The aggregate zero-shot and one-shot results are shown in Figure 15. As seen, on average, GVL zero-shot works best on the Table viewpoint. This is not surprising, as images taken with the front facing table camera are arguably visually closer to naturally captured images used for VLM training. Yet, with in-context examples, GVL consistently improves on all camera viewpoints. In practice, this means that GVL is robust to camera viewpoints – even when a camera viewpoint is determined to be sub-optimal post-hoc, practitioners can make up for that by simply providing few in-context examples.

Figure: Figure 16: GVL VOC for video and language description pairs. Shuffling enables GVL to pay attention to the language task description in order to faithfully predict observation values.
Refer to caption: extracted/5984057/figures/shuffle_true_heatmap.png

Figure: Figure 17: No-shuffling ablation VOC for video and language description pairs. Removing shuffling makes VLM output high VOCs independent of task descriptions.
Refer to caption: extracted/5984057/figures/shuffle_false_heatmap.png

Does GVL pay attention to the task specification?
To validate that GVL is not merely recovering the temporal coherence in the shuffled input video but actively tracking visual progress according to the task language command, we compute the VOC scores for every combination of task input video and language description in the ALOHA-13 split. The heatmap visualization of the average VOC for every pairing is illustrated in Fig. 16 and Fig. 17 for GVL and the no shuffling ablation. On 9 out of 13 tasks, GVL achieves the highest VOC when the input video and the task description matches; in many unmatched cases, the model simply refuses to output value predictions, stating that the frames and the language description are not related. In contrast, when we do not shuffle the input frames, the quality dramatically drops.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - It would be great to see LIV’s performance on the ALOHA dataset. I assume that, since GVL outperformed LIV on OXE, it would do so again on ALOHA, but it would be nice to see these results formalized.
- What is ALOHA-13? I see the descriptions of ALOHA-250, but I don’t see the same for ALOHA-13 for some reason.
soundness: 3
strengths: Although the proposed technique is simple, it seems (based on myriad experiments) to be effective. Figure 2 demonstrates a clear superiority of GVL over LIV, the baseline that the authors chose to test, for text-based goals. Figure 4 demonstrates that GVL can be decently effective at predicting values correctly over more complex videos. Finally, GVL was effective at training agents via both (1) data filtering for imitation learning and (2) advantage estimation for advantage weighted regression.

Breaking things down by category:
- Originality: Due to the simplicity of this technique, I think the originality is somewhat low, however the method is effective, especially in comparison with other simple alternatives.
- Quality: With the exception of the potential circularity I mention below, the experiments performed seemed to be of high quality, and they seem to demonstrate the quality of this technique.
- Clarity: There were some minor questions that I have detailed below, but for the most part, the paper was well-written and easy to understand.
- Significance: The authors provide solid motivation for the significance of this technique by demonstrating how it could be used for model training and dataset quality checks.
summary: This paper presents Generative Value Learning (GVL), a novel technique that utilizes Vision-Language Models (VLMs) to serve as a value function. In particular, by shuffling the order of frames within the video and utilizing few-shot prompting, GVL exceeds the performance achievable using VLMs without shuffling and without temporal dependence. Furthermore, the authors demonstrate that, by comparing the GVL value estimates with the true frame order, it is possible to identify and filter out low quality video demonstration data. The authors demonstrate that GVL can be used to train more effective policies via both data filtering and more accurate value estimation.
weaknesses: The use of Value Order Correlation (VOC)—the correlation between GVL and frame order—seemed somewhat circular. It is used on the OXE dataset to judge GVL’s performance, a choice that assumes that the contents of OXE are of sufficient quality. (If, for instance, OXE contained many sub-optimal trajectories, an ideal value estimator would show low VOC.) Later, there is a discussion about the use of GVL+VOC to identify sub-optimal datapoints/subsets within OXE. This seemed confusing to me since it seemed to contradict the earlier assertion that OXE was of sufficient quality for judging GVL. 

It would be good to see more analysis regarding GVL’s ability to accurately estimate values in videos where repetition is present. Currently, the existence of such an ability has to be inferred from downstream effects (e.g., the ability to use GVL for data filtering helps during imitation learning).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Overall, I enjoyed reading the paper, but I'm just concerned with the strength of the experiment section. I'm willing to consider changing my score if the experiments section is made stronger.
soundness: 3
strengths: Pros:
1. The high-level of the paper is easy to understand because of the effort that the authors took to portray it on the figures.
2. The experiments that were performed to evaluate the method are easy to see how the proposed method is better than the baseline/(baselines in some cases).
summary: The paper presents a method that enables existing VLMs to understand the task progress/completion, that could help in the use cases like dataset filtering, success detection and visuomotor control/policy learning. The authors also propose an autoregressive value prediction instead of the traditional bellman returns.
weaknesses: Weakness
1. Currently, the block diagram only provides a high-level idea of what the paper is about, which is good. But it would to nice to have the language component of the model. For example, I get that you are trying to predict the value estimate of a frame, but without the task description, it’d be impossible to predict whats the value estimate.
2. The experiments are very limited, although the use cases are diverse enough. In each of these use cases, however, the comparisons are quite restricted. My understanding is that you are using a policy that is learnt on the downstream task. There are so many representation learning works that do that and I urge the authors to find more recent baselines for the use cases, especially, for the visuomotor control
3. Why are the authors not comparing with LIV on the visuomotor tasks?
4. Is there a plot/table that shows how much incorporating observation shuffling helps or using autoregressive value prediction helps than the traditional bellman returns? I think this is important to understand the efficacies of the 2 proposed incorporations

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - Can GVL be applied for online RL training?
soundness: 3
strengths: - Well motivated and set desideratas for a “universal value estimator”: accurately estimating state and consistency, highlighting the limitations of prior works
- Interesting finding; shuffling the video frames help mitigate the temporal bias found in videos ⇒ better value estimations
- Introduces a new evaluation metric: Value-Order Correlation which measures how well predicted values correlate with the ground-truth timestep in expert videos.
- Extensive evaluation of GVL’s value estimates across a large suite of real-world robot datasets with varying tasks and embodiments
- GVL can be applied to a wide variety of tasks such as dataset quality estimation, success detection, and weighted imitation learning.
- Good analysis and ablation studies to verify the importance of each component in the GVL approach and robustness to different environment factors.
summary: This paper presents Generative Value Learning (GVL) a value function estimator for predicting task progress from visual trajectories. They show that without any task-specific training, their approach can in-context zero-shot and few-shot predict values for more than 300 real-world tasks across different embodiments. The approach is simple: autoregressive prediction, shuffle inputs, and provide in-context examples of task progress.
weaknesses: - The real-world results are not too impressive. Applying GVL to compute values only leads to a small improvement over the based diffusion policy baseline. It would be interesting to see more results showing how the value estimations of GVL help with downstream policy learning.
- Method itself is technically simple, but still well-motivated and supported with empirical results.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - In Figure 1, some arrows are bidirectional. Replacing these with unidirectional arrows could improve consistency.
- In the Dataset Quality Estimation paragraph of Section 4.3, the reasoning feels incomplete. It may strengthen the experiment to directly train models with and without specific low-quality datasets and compare the resulting performance outcomes.
soundness: 4
strengths: - Framing value estimation as a temporal ordering problem over shuffled videos is a compelling approach.
- The proposed rank correlation metric is thoughtfully designed and well-suited to this problem setting.
- The experiments are extensive, covering a wide range of real-world tasks.
summary: This paper casts universal value estimation of video frames as a temporal ordering problem over shuffled video frames. To address this, the authors introduce Generative Value Learning (GVL), which leverages the world knowledge embedded in vision-language models (VLMs) to estimate task progress. Here, the value for each frame is represented as the VLM's prediction of task completion rate. Experimental results demonstrate that GVL can be effectively applied to various downstream tasks, using a Value-Order Correlation (VOC) metric that measures the rank correlation between the rank of estimated values and the original frame order.

Overall, I believe this paper is suitable for publication at ICLR, though there is room for further development, especially in the clarity of the writing.
weaknesses: - My primary concern is with the method's applicability. As I understand, calculating the value for each frame requires access to a complete video, which could limit its use in online policy learning without an offline dataset or expert demonstrations. This restricts the downstream tasks to scenarios where offline datasets are available, while the compared method [1] remains applicable in online policy learning.
- The manuscript contains several typos and minor errors that should be addressed. I recommend a careful proofreading. A non-exhaustive list includes:
    - Line 163: missing comma.
    - Line 178: missing space after “2).”
    - Line 197: "meaningfully" should be "meaningful."
    - Line 393: "ataset" should be "dataset."

[1] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, pp. 23301–23320. PMLR, 2023.

Meta-review: The paper presents Generative Value Learning (GVL), a novel approach leveraging Vision-Language Models (VLMs) to predict task progress from visual trajectories. The reviewers' comments and the authors' responses provide a comprehensive view of the paper's strengths and areas for improvement.
The paper has several notable strengths. The proposed GVL method is conceptually interesting and supported by extensive experiments across diverse real-world tasks. The authors effectively addressed the reviewers' concerns, improving the paper's quality and clarity. While the method's simplicity was initially noted as a potential weakness, the authors demonstrated its effectiveness through comprehensive evaluations and comparisons. Overall, the paper meets the standards for acceptance, as it presents a novel and valuable contribution to the field of vision-language models and robotics, with the potential to impact downstream applications in policy learning and dataset management.

Decision: Accept (Spotlight)