Video Prediction Models as Rewards for Reinforcement Learning
Authors: Alejandro Escontrela
Ademi Adeniji
Wilson Yan
Ajay Jain
Xue Bin Peng
Ken Goldberg
Youngwoon Lee
Danijar Hafner
Pieter Abbeel
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement learning
generative modeling
learning from demonstrations
video prediction
unsupervised reinforcement learning
TL;DR: VIPER uses the conditional likelihoods from pre-trained video prediction models as rewards to learn policies from action and reward-free video.

Abstract: Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning.
A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for a reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across a wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. We see our work as starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available on the project website: https://ViperRL.com

Video Prediction Models as Rewards for Reinforcement Learning
Alejandro Escontrela"
Ademi Adeniji
Wilson Yan
Ajay Jain
Xue Bin Peng
Ken Goldberg
Youngwoon Lee
Danijar Hafner
Pieter Abbeel
University of California, Berkeley
Equal contribution
Abstract
Specifying reward signals that allow agents to learn complex behaviors is a long- standing challenge in reinforcement learning. A promising approach is to extract preferences for behaviors from unlabeled videos. which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available the project website: https: / http///eteeeelaleeeeeeeeeeeeeele me /viper
see our work as
Introduction
Manually designing reward function is laborious and often leads to undesirable outcomes [32]. This
r== In pol X2 x1a)
a major bottleneck for developing general decision making agents with reinforcement learning (RL), A more scalable approach is to learn complex behaviors from videos, which be easily acquired at scale for many applications (e.g.. Youtube videos).
Frozen Video Prediction Model
Previous approaches that learn behaviors from videos reward the similarity between the agent's current ob- servation and the expert data distribution [37, 38, 30, 43]. Since their rewards only condition on the current observation, they cannot capture temporally mean- ingful behaviors. Moreover, the approaches with adversarial training schemes [30, 43] often result in mode collapse, which hinders generalization.
X2
X
Figure 1: VIPER uses the next-token likeli- hoods a frozen video prediction model as general reward function for various tasks.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).
"Corresponing author: escontrela beerkkeeeeeerkeeuuu

---

(a) DeepMind Control Suite
(b) Robot Learning Benchmark
(c) Atari
(15 tasks)
(6 tasks)
tasks)
Figure 2: VIPER achieves expert-level control directly from pixels without access to ground truth rewards expert actions on 28 reinforcement learning benchmark tasks.
Other works fill in actions for the (action-free) videos using an inverse dynamics model [42, 8]. Dai et al. [8] leverage recent advances in generative modeling to capture multi-modal and temporally- coherent behaviors from large-scale video data. However, this multi-stage approach requires perform- ing expensive video model rollouts to then label actions with learned inverse dynamics model.
In this paper, propose using Video Prediction Rewards (VIPER) for reinforcement learning. VIPER first learns a video prediction model from expert videos. We then train an agent using reinforcement learning maximize the ligelikel of agent trajectories estimated by the video prediction model, as illustrated in Figure 1. Directly leveraging the video model's likelihoods as reward signal encourages the agent match the video model's trajectory distribution. Additionally, rewards specified by video models inherently measure the temporal consistency of behavior, unlike observation-level rewards. Further, evaluating likelihoods is significantly faster than performing video model rollouts, enabling faster training times and more interactions with the environment.
We summarize the three key contributions of this paper follows:
We present VIPER: a novel, scalable reward specification algorithm which leverages rapid im- provements in generative modeling to provide RL agents with rewards from unlabeled videos.
We perform an extensive evaluation, and show that VIPER can achieve expert-level control without task rewards on DMC tasks [44], RL.Bench tasks [20], and 7 Atari tasks [3] (see examples in Figure 2 and Appendix A.5).
We demonstrate that VIPER generalizes to different environments for which no training data was provided, enabling cross embodiment generalization for tabletop manipulation.
Along the way, we discuss important implementation details that improve the robustness of VIPER.
2 Related Work
Learning from observations is an active research area which has led many advances in imitation learning and inverse reinforcement learning [27, 1, 49]. This line of research is motivated by the fact that learning policies from expert videos is scalable way of learning wide variety of complex tasks, which does not require access to ground truth rewards, expert actions, or interaction with the expert. Scalable solutions to this problem would enable learning policies using the vast quantity of videos on the internet. Enabling policy learning from expert videos can be largely categorized into two approaches: (1) Behavioral cloning [31] expert videos labelled with predicted actions and (2) reinforcement learning with a reward function learned from expert videos.
Labelling videos with predicted actions An intuitive way of leveraging action-free expert videos guess which action leads to each transition in expert videos, then to mimic the predicted actions. Labelling videos with actions can be done using an inverse dynamics model, p(als, s'), which models an action distribution given the current and next observations. An inverse dynamies model can be learned from environment interactions [28, 42, 29] or a curated
dataset [2, 34]. Offline reinforcement learning [24] can be also used instead of behavioral cloning for more efficient use of video data with predicted actions [34]. However, the performance of this approach heavily depends on the quality of action labels predicted by an inverse dynamics model and the quantity and diversity of training data.

---

Reinforcement learning with videos Leveraging data from online interaction can further improve policies trained using unlabelled video data. To guide policy learning, many approaches have learned reward function from videos by estimating the progress of tasks [37, 38, 25, 23] or the divergence between expert and agent trajectories [43, 30]. Adversarial imitation learning approaches [43, 30] learn a discriminator that discriminates transitions from the expert data and the rollouts of the current policy. Training a policy to maximize the discriminator error leads to similar expert and policy behaviors. However, the discriminator is prone to mode collapse, it often finds spurious associations between task-irrelevant features and expert/agent labels [50, 19], which requires variety of techniques stabilize the adversarial training process [7]. In contrast, VIPER directly models the expert video distribution using recent generative modeling techniques [10, 47], which offers stable training and strong generalization.
Using video models as policies Recently, UniPi [8] uses advances in text-conditioned video generation models [33] to plan a trajectory. Once the future video trajectory is generated, UniPi executes the plan by inferring low-level controls using a learned inverse dynamics model. Instead of using slow video generations for planning, VIPER uses video prediction likelihoods to guide online learning of a policy.
Video Prediction Rewards
In this section, we propose Video Pre- diction Rewards (VIPER), which learns complex behaviors by leveraging the log- likelihoods of pre-trained video predic- tion models for reward specification. Our method does not require any ground truth rewards or action annotations, and only re- quires videos of desired agent behaviors. VIPER implements rewards as part of the environment, and can be paired with any RL algorithm. We overview the key components of our method below.
Algorithm 1: VIPER
Train video prediction model pe on expert videos. while not converged do
Choose action:
ag Tx;;)
Step environment: Jz+1 f- env(a:)
Fill in reward: Ti f In po z(-k=t) + Br
Add transition (Iz, Qe. F(41) to replay buffer.
Train from replay buffer using any RL algorithm.
3.1 Video Modeling
selihhodd-baed video models are a popular paradigm of generative models trained to model the data distribution by maximizing an exact lower bound on the log-likelihood of the data. These models have demonstrated their ability to fit highly multi-modal distributions and produce samples depicting complex dynamics, motion, and behaviors [40, 18, 17, 45].
Our method can integrate any video model that supports computing likelihoods over the joint distribution factorized in the following form:
T
logp(Eee)) = Σ log logr(zz a1a-1);
(I)
where Fir is the full video consisting of T frames, FI,..., IT. When using video models with limited context length k; it is common to approximate likelihoods of an entire video sequence with its subsequences of length & as follows:
T
logp(z1:T) Rd
log p(a; (maax(11-AA:a--1))
In this paper, we use an autoregressive transformer model based on VideoGPT [47, 36] as our video generation model. We first train a VQ-GAN [10] encode individual frames H, into discrete codes 2, Next, we learn an autoregressive transformer to model the distribution of codes through the following maximum likelihood objective:
T
2
max 8
log po( 21 , 21:-1).
(3)
3

---

DMC (DrQ)
DMC (Dv3)
Atari (Dv3)
RLBench (Dv3)
600
1000
12
0.60
450
750
9
0.45
300
500
6
0.30
0.15
150
250
3
0
0
0.00
250K
500K
5M
10M
5M
10M
0
500K
1M
Task Oracle
VIPER
AMP
AMP Multi-Task
GAIfO
BCO
Figure 3: Aggregate results across 15 DMC tasks, 7 Atari games, and 6 RL tasks. DMC results are provided for DrQ and DreamerV3 (Dv3) RL agents. Atari and RLBench results are reported for DreamerV3, Atari scores are computed using Human-Normalized Mean.
where z is the i-th code of the t-th frame, and Z is the total number of codes per frame. Computing requires marginalizing over all
the exact conditional likelihoods p0 -1;t-1) 1)) is intractable, possible combinations of codes. Instead, we use the conditional likelihoods over the latent codes (2 21:2-1 as an approximation. In our experiments, we show that these likelihoods are sufficient to capture the underlying dynamics of the videos.
Note that our choice of video model does not preclude the use of other video generation models, such as MaskGiT-based models [48, 45, 13] or diffusion models [18, 40]. However, we opted for an autoregressive model due to the favorable properties of being able to model complex distributions while retaining fast likelihood computation. Video model comparisons are performed in Section 4.5.
3.2 Reward Formulation
Given pretrained video model, VIPER proposes an intuitive reward that maximizes the conditional log-likelinee ligliikli for each transition (It, az. 2141) observed by the agent:
r/ VIPER = In p0 (1:.).
This reward incentivizes the agent to find the most likely trajectory under the expert video distribution as modeled by the video model. However, the most probable sequence does not necessarily capture the distribution of behaviors we want the agent to learn.
For example, when flipping a weighted coin with p(heads 0.6) 1000 times, typical sequences will count roughly 600 heads and 400 tails, in contrast to the most probable sequence of 1000 heads that will basically never be seen in practice [39]. Similarly, the most likely image under density model trained on MNIST images is often the image of only background without a digit, despite this never occurring in the dataset [26]. In the reinforcement learning setting, an additional issue is that solely optimizing a dense reward such as r VIPERR can lead to early convergence to local optima.
To overcome these challenges, we take the more principled approach of matching the agent's 's trajectory distribution q(x1.)) to the sequence distribution po (x1:r) of the video model by minimizing the KL-divergence between the two distributions [41, 15]:
KT. [q((1:)) || p(*1r))] F.g In ps (= H [q(
cross-cntropy
entropy
(5)
Σ
Eq
In In x1:e) ]
+ H [g(2141 x1,e)]
r VIPER
exploration
The KL objective shows that for the agent to match the video distribution under the video prediction model, it has to not only maximize the VIPER reward but also balance this reward while maintaining high entropy over its input sequences [21]. In the reinforcement learning literature, the entropy bonus over input trajectories corresponds to an exploration term that encourages the agent to explore and inhabit diverse distribution of sequences within the regions of high probability under the video prediction model. This results in the final reward function that the agent maximizes:
rt
KL - r
VIPER + expll
(6)
where 3 determines the amount of exploration. To efficiently compute the likelihood reward, we approximate the context frames with a sliding window discussed in Equation 2:
VIPER ~ In ( #=max(I.t-)it))
(7)

---

Figure 1 shows the process of computing rewards using log probabilities under the video predic- tion model. VIPER is agnostic the choice of exploration reward and in this paper we opt for Plan2Explore [35] and RND [5].
3.3 Data Curation
In this work, we explore whether VIPER provides adequate reward signal for learning low-level control. We utilize the video model likelihoods provided by an autoregressive video prediction model pre-trained on data from a wide variety of environments. We curate data by collecting expert video trajectories from task oracles and motion planning algorithms with access state information. Fine-tuning large (ext-conditioned video models [33, 45, 40] on expert videos would likely lead to improved generalization performance beyond the curated dataset, and would make for interesting future research direction. We explore the favorable generalization capabilities of video models trained on small datasets, and explore how this leads more general reward functions in Section 4.3.
Experiments
We evaluate VIPER on 28 different tasks across the three domains shown in Figure 2. We utilize 15 tasks from the DeepMind Control (DMC) suite [44]. 7 tasks from the Atari Gym suite [4], and tasks from the Robot Learning Benchmark (RLBench) [20]. We compare VIPER agents to variants of Adversarial Motion Priors (AMP) [30], which uses adversarial training to learn behaviors from reward-fie free and action-free expert data. All agents are trained using raw pixel observations from the environment, with no access to state information or task rewards. In our experiments, we aim to answer the following questions:
1. Does VIPER provide an adequate learning signal for solving a variety of tasks? (Section 4.2)
2. Do video models trained on many different tasks still provide useful rewards? (Section 4.2)
3. Can the rewards generalize novel scenarios where expert data is available? (Section 4.3)
4. How does the video model data quality and quantity affect the learned rewards? (Section 4.4)
5. What implementation details matter when using video model likelihoods as rewards? (Section 4.5)
4.1 Video Model Training Details
Training data To collect expert videos in DMC and Atari tasks, Task Oracle RI. agents are trained until convergence with access to ground truth state information and task rewards. After training, videos of the top episodes out of 1000 episode rollouts for each task are sampled as expert videos, where & = 50 and 100 for DMC and Atari, respectively. For RLBench, a sampling-basee based motion planning algorithm with access to full state information is used to gather 100 demos for each task, where selected tasks range from "easy" to "medium".
Video model training We train a single autoregres- sive video model for each suite of tasks. Example images for each suite are shown in Appendix A.5. For DMC, we train a single VQ-GAN across all tasks that encodes 64 x 64 images to 8 x 8 discrete codes. Similarly for RLBench and Atari, we train VQ-GANs across all tasks within each domain, but encode 64 x 64 images to 16 x 16 discrete codes. In practice, the level of VQ compression depends on the visual complexity of the environment more texturally simple environments (e.g., DMC) allow for higher levels of spatial compression. We follow the original VQ-GAN architecture [10], consisting of CNN encoder and decoder. We train VQ-GAN with a batch size of 128 and learning rate 10 for 200k iterations.
T
T= 16
T= 32
48
T= 64
Figure 4: Video model rollouts for 3 different evaluation environments.
To train our video models, we encode each frame of the trajectory and stack the codes temporally to form an encoded 3D video tensor. Similar to VideoGPT [47], the encoded VQ codes for all video
5

---

frames are jointly modeled using an autoregressive transformer in raster scan order. For DMC and Atari, we train on 16 frames time with one-hot label for task conditioning. For RLBench, we train both single-task and multi-task video models on 4 frames with frame skip 4. We add the one-hot task label for the multi-task model used in our cross-embodiment generalization experiments in Section 4.3. We perform task conditioning identical to the class conditioning mechanism in VideoGPT, with learned gains and biases specific to each task ID for each normalization layer.
Figure 4 shows example video model rollouts for each domain. In general, our video models are able to accurately capture the dynamics of each environment to produce probable futures. Further details on model architecture and hyperparameters can be found in Appendix A.2. All models are trained on TPUv3-8 instances which are approximately similar Nvidia V100 GPUs.
4.2
Video Model Likelihoods as Rewards for Reinforcement Learning
To learn behaviors from expert videos, we provide
T=0
T=50
T=100
T=200
reward signals using Equation 6 in VIPER and the discriminator error in AMP. Both VIPER and AMP
OOD Reference
are agnostic to the choice of RL algorithm; but, in this paper, we evaluate our approach and baselines with
two popular RL algorithms: DrQ [22] and Dream- erV3 [16]. We use Random Network Distillation [5] (model-free)) as the exploration objective for DrQ, and Plan2Explore [35] imodel-based for DreamerV3. Hyperparameters for each choice of RL algorithm are shown in Appendix A.3. We compare VIPER to two
1e5
-50
VIPER Return
00.5
VIPER
variants of the AMP algorithm: the single-task vari-
-100
-1.0
ant where only expert videos for the specific task are
provided to the agent, and the multi-task case where expert videos for all tasks are provided to the agent.
150
-1.5
0
250
500
500 1K
Sequence Step
Task Return
As outlined in Algorithm 1, we compute Equation 6 for every environment step to label the transition with reward before adding it the replay buffer for policy optimization. In practice, batching Equation 6 across multiple environments and leveraging parallel likelihood evaluation leads to only a small decrease in training speed. DMC agents were trained using I Nvidia V V100 GPU, while Atari and RLBench agents were trained using 1 Nvidia A100 GPU.
Reference
OOD (Random)
Figure 5: VIPER incentivizes the agent to maximize trajectory likelihood under the video model. As such, it provides high re- wards for reference (expert) sequences, and low rewards for unlikely behaviors. Mean rewards y VIPER and returns are computed across 40 trajectories.
We first verify whether VIPER provides meaningful rewards aligned with the ground truth task rewards. Figure 5 visualizes the ground truth task rewards and our log- likeehooooo rewards (Equation 7) for a reference (expert) trajectory and a random dittr---iitributionn trajectory. In the reward curves on the left, VIPER starts predict high rewards for the expert transitions once the transitions becomes distinguishable from a random trajectory, while it consistently outputs low rewards for out-of-distribun transitions. The return plot the right clearly shows positive correlation between returns of the ground truth reward and our proposed reward.
Then, we measure the task rewards when training RL agents with predicted rewards from VIPER and basclines and report the aggregated results for each suite and algorithm in Figure 3. The full results can be found in Appendix A.4.
In DMC, VIPER achieves near expert-level performance from pixels with our video prediction rewards alone. Although VIPER slightly underperforms Task Oracle, this is surprising as the Task Oracle uses full state information along with dense task rewards. VIPER outperforms both variants of AMP. Worth noting is the drastic performance difference between the single-tast and multi-task AMP algorithms. This performance gap can possibly be attributed to mode collapse, whereby the discriminator classifies all frames for the current task as fake samples, leading to an uninformative reward signal for the agent. Likelihood-based video models, such as VIPER, are less susceptible to mode collapse than adversarial methods.

---

In Atari, VIPER approaches the performance of the Task Oracle trained with the original sparse task reward, and outperforms the AMP baseline. Shown in Figure 6, we found that masking the scoreboard in each Atari environment when training the video model improved downstream RI. performance.
Since the video model learns to predict all aspects of the data distribution, including the scoreboard, it tends provide noisier reward signal during policy training when the agent encounters out-of-distribution seen by the video model. For example, expert demos for Pong only contain videos where the player scores and the opponent's score is always zero. During policy training. we observe that the Pong agent tends to exhibit more erratic behaviors as soon as the opponent scores least one point, whereas when masking the scoreboard, learned policies are generally more stable. These results suggest the potential benefits of finetuning large video models on expert demos to learn more generalizable priors, as opposed to training from scratch.
No Mask
1.0
0.5
scores not
Mask
0.0
5M
10M
Task Oracle
Mask
No Mask
Figure 6: RL training curves on Atari Pong when using VIPER trained with without masking the scoreboard.
For RLBench, VIPER outperforms the Task Oracle because RLBench tasks provide very sparse rewards after long sequences of actions, which pose challenging objective for RL agents. VIPER instead provides dense reward extracted from the expert videos, which helps learn these challenging tasks. When training the video model, we found it beneficial to train at reduced frame rate, accomplished by subsampling video sequences by a factor of 4. Otherwise, we observed the video model would assign high likelihoods to stationary trajectories, resulting in learned policies rarely moving and interacting with the scene. We hypothesize that this may be partially due to the high control frequency of the environment, along with the initial slow acceleration of the robot arm in demonstrations, resulting in very little movement between adjacent frames. When calculating likelihoods for reward computation, we similarly input observations strided by time, e.g..
p(a; T1-1.-
4.3 Generalization of Video Prediction Rewards
Prior works in the space of large, pretrained generative models have shown a powerful ability to generalize beyond the original training data, demonstrated by their ability to produce novel generations (e.g., unseen text captions for image or video generation [33, 45, 40]) well as learn more generalizable models from limited finetuning data [8, 46). Both capabilities provide promising directions for extending large video generation models to VIPER, where we can leverage text-video models to capture priors specified by text commands, or finetune on a small set demonstrations to learn task-specific prior that better generalizes to novel states.
Train
Train
OOD
Figure Sampled video predictions for in distribution reference videos (Train) and an OOD arm/task combination (OOD). The video model displays cross-embodiment generalization to arm/task combination not observed in the training data. Video model generalization can enable specifying new tasks where no reference data is available.
7

---

Success
In this section, we seek to understand how this generalization can be used to learn more gen- eral reward functions. We train a model on two datasets of different robot arms, and evaluate generalization capabili-
0.5
VIPER
Ref Rand
0.0
10 0
0.5M 1M
25 50
the cross-embodiment ties of the model. Specifically, we gather demon- strations for 23 tasks on the Rethink Robotics Sawyer Arm, and demonstrations for 30 tasks the Franka Panda robotic arm, where only 20 tasks are overlapping between arms. We then train condiconnitioned autoregressive video model on these demonstration videos and evaluate the video model by querying unseen arm/task combinations, where single initial frame is used for open loop predictions.
Training Step
Sequence Step
Figure 8: (Left) Training curve for RL agent trained with VIPER on OOD task. (Right) Task- conditional likelihood for reference and random trajectory for an OOD task.
Sample video model rollouts for distribution training tasks and an OOD arm/task combination
shown in Figure 7. Even though the video model was not directly trained on demonstrations of the Franka Panda arm to solve the saucepan task in RLBench, trajectories for the arm and task combination. Figure 8 further validates this observation by assigning higher likelihood to expert videos (Ref) compared to random robot trajectories (Rand). We observe that these generalization capabilities also extend to downstream RL, where model with VIPER to learn a policy for the Franka Robot arm to solve an OOD task without requiring demos for that specific task and arm combination. Figure 21 further extends this analysis to include more in distribution and out of distribution tasks. These results demonstrate a promising direction for future work in applying VIPER larger scale video models that will be able to better generalize and learn desired behaviors only through a few demonstrations.
able to generate reasonable
trained video
4.4 Impact of Data Quality and Quantity
Learning from sub-ope uub-optimal data is an important feature of learned reward models, as large-scale video data may often contain suboptimal solutions given task, As such, we evaluate VIPER's ability to learn rewards from sub-optimal data.
We train video models with suboptimal (good) data, which has 50-75% returns of the expert data, In Figure 9, VIPER learns the suboptimal behaviors provided in the suboptimal video data. This suboptimal VIPER can be still useful if combined with a sparse task reward, comparable an agent leamedd from dense task rewards.
Additionally. evaluate how VIPER performs under different video dataset sizes. As shown in Figure 10, VIPER can learn a meaningful reward function only with one expert trajectory. although adding more videos quickly improves the performance of VIPER, with diminishing returns after 50 trajectories.
DMC (Dv3)
Atari (Dv3)
1000
750
100
9
500
3
250
0
0
5M
10M
0
5M
10M
Extr Only VIPER (Expert)
Extr + VIPER (Good) VIPER (Good)
Figure 10: DMC performance with VIPER trained on different dataset sizes, where N is the number of expert trajectories. *Original dataset size.
Figure 9: Atari performance with VIPER models trained suboptimal data.
4.5
Ablation Studies
In this section, we study the contributions of various design decisions when using video prediction models as rewards: (1) how weight the exploration objective, (2) which video model use, and (3) what context length to use. Ablation studies are performed across DMC tasks.

---

Exploration
Video Model
Context Length
1000
1000
1000
500
500
500
0
0
4M
8M
4M
8M
4M
8M
= 1.0
VideoGPT BYOL MaskGIT
16
8 8.0.5
0.0
Figure 11: Effect of exploration reward term, video model choice, and context length on downstream RL performance. An equally weighted exploration reward term and longer video model context leads to improved performance. MaskGIT substantially underperforms VideoGPT as choice of video model. The BYLL model performs moderately due to the deterministic architecture not properly handling multi-modale multi-modality.
Exploration objective As discussed in subsection 3.2, exploration objective may help the RL agent learn a distribution of behaviors that closely matches the distribution of the original data and generalizes, while preventing locally optimal solutions. In Figure 11, ablate the 3 parameter introduced in Equation using a VideoGPT-like model as the VIPER backbone. 3 0 corresponds exploration objective, whereas 3 signifies equal weighting between y VIPER and expl, We ablate the exploration objective using the Plan2Explore [35] reward, which provides the agent with a reward proportional to the disagreement between ensemble of one-step dynamics models. Using no exploration objective causes the policy's 's behavior to collapse, while increasing the weight of the exploration objective leads to improved performance.
Video model Although our experiments focus on using an autoregressive VideoGPT-like model compute likelihoods, VIPER generally allows for any video model that supports computing conditional likelihoods or implicit densities.
Figure 11 shows additional ablations replacing VideoGPT model with similarly sized MaskGIT [6] model, where frames are modeled with MaskGIT over space, and autoregressive time. MaskGIT performs substantially worse than the VideoGPT model, which is possibly due to noisier likelihoods from parallel likelihood computation. addition, while VideoGPT only requires I forward pass to compute likelihood, MaskGIT requires many forward passes the sequence of tokens for a frame, resulting in an approximately 8x slowdown in reward computation, and 2.5x in overall RL training.
Finally, we evaluate the performance of video model that computes implicit densities using a negative distance metric between online predictions and target encodings with recurrent Bootstrap Your Own Latent (BYOL) architecture [12, 11]. We refer the reader to Appendix A. for more details about the implementation. While BYOL outperforms MaskGIT in Figure 11, its deterministic recurrent architecture is unable to predict accurate embeddings more than a few steps into the future. This limits the ability of the learned reward function capture temporal dynamics. We observe that agents trained with B BYL often end up in a stationary pose, which achieves high reward from the BYO model.
Context length The context length l of the video model is an important choice in determining how much history incorporate into the conditional likelihood. At the limit where O, the reward is the unconditional likelihood over each frame. Such a choice would lead to stationary behavior on many tasks. Figure 11 shows that increasing the context length can help improve performance when leveraging a VideoGPTTbased model for downstream RL tasks, subject to diminishing returns. We hypothesize that a longer context length may help for long-horizon tasks.

---

Conclusion
This paper presents Video Prediction Rewards (VIPER), a general algorithm that enables agents learn behaviors from videos of experts. To achieve this, we leverage simple pre-trained au- toregressive video model to provide rewards for reinforcement learning agent. These rewards parameterized the conditional probabilities of particular frame given context past of frames. In addition, we include an entropy maximization objectíve to ensure that the agent learns diverse behaviors which match the video model's trajectory distribution. VIPER succeeds across 3 benchmarks and 28 tasks, including the DeepMind Control Suite, Atari, and the Reinforcement Learning Benchmark. We find that simple data augmentation techniques can dramatically improve the effectiveness of VIPER. We also show that VIPER generalizes to oitt---distribution tasks for which no demonstrations were provided. Moreover, VIPER can learn reward functions for wide variety of tasks using single video model, outperforming AMP [30], which suffers from mode collapse as the diversity of the expert videos increases.
Limitations of our work include that VIPER is trained using in-domain data of expert agents, which reaiill-vvilaable data source in the real world. Additionally, video prediction rewards trained on stochastic data may lead the agent to prefer states that minimize the video model's uncertainty, which may lead to sub-optimal behaviors. This challenge may be present for cases where either the environment is stochastic or the demonstrator provides noisy demonstrations. Moreover, selecting the right trade-off between VQCode size and context length has significant impact on the quality of the learned rewards, with small VQCodes failing to model important components of the environment (e.g.. the ball in Atari Breakout) and short context lengths leading to myopic behaviors which results in poor performance. Exploring alternative video model architectures for VIPER would likely be fruitful research direction.
To improve the generalization capabilities of VIPER, larger pre-trained video models are necessary. Future work will explore how fine-tuning or human preferences can be used improve video prediction rewards. Using text-to-video models remain another interesting direction for future work in extracting text-conditioned task-specific priors from pretrained video model. In addition. extending this line of work to leverage Video Diffusion Models [18] as video prediction rewards may lead to interesting outcomes.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by an NSF Fellowship. NSF NRI #2024675, ONR MURI N00014- 22-1-2773, Komatsu, and the Vanier Canada Graduate Scholarship. We also thank Google TPU Research Cloud and Cirrascale (https//cirrasaal.comm for providing compute resources.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, 2004.
[2] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning act by watching unlabeled online videos. In Advances in Neural Information Processing Systems, 2022.
[3] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 160.::6666..15444 2016.
[5] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
[6] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325 2022.
[7] Soumith Chintala, Emily Denton, Martin Arjovsky, and Michael Mathieu. How to train a gan? tips and tricks make gans work. https://githu:t www/soumttttteeee..e t thgganhacks, 2020.
[8] Yilun Dai, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuur- mans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXivv2332.0011 2023.
[9] Andreas Doerr, Christian Daniel, Martin Schiegg, Nguyen-Tuong Duy, Stefan Schaal, Marc Toussaint, and Trimpe Sebastian. Probabilistic recurrent state-space models. In International conference machine learning, pages 1280-1289. PMLR, 2018.
[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for nigh-resoluttin image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12873-12883, 2021.
[11] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in neural information processing systems, volume 33, pages 21271-21222 2020.
[12] Zhaohan Guo, Shantanu Thakoor, Miruna Pistar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastier Grill, Yunhao Tang, et al. Byol-explore: Exploration by bootstrapped prediction. In Advances in neural information processing systems, volume 35, pages 31855-31870, 2022.
[13] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martin and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.1 11864,11444 2022.
[14] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2563 PMLR, 2019.

---

[15] Danijar Hafner, Pedro A Ortega, Jimmy Ba. Thomas Part, Karl Friston, and Nicolas Heess. Action and perception as divergence minimization. arXiv preprint arXiv:2009.0199) 2020.
[16] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 2Et1::300.004444, 2023.
[17] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
[18] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, 2022.
[19] Andrew Jaegle, Yury Sulsky, Arun Ahuja, Jake Bruce, Rob Fergus, and Greg Wayne. Imitation by predicting observations. International Conference on Machine Learning, pages 4665 PMLR, 2021.
[20] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.
[21] Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
[22] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021.
[23] Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph Lim. Generalizable imitation learning from observation via inferring goal proximity. In Neural Information Processing Systems, 2021.
[24] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01665 2020.
[25] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In IEEE International Conference on Robotics and Automation, 2018
[26] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't tt know? In International Conference Learning Representations, 2019.
[27] Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
[28] Scott Nickum, Sarah Osentoski, George Konidaris, Sachin Chitta, Bhaskara Marthi, and An- drew G Barto. Learning grounded finite-state representations from unstructured demonstrations. The International Journal of Robotics Research, 3412):131-157, 2015.
[29] Deepak Pathak, Parsa Mahmoudieh, Michael Luo, Pulkit Agrawal, Dian Chen, Fred Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation. In International Conference on Learning Representations, 2018
[30] Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP: adversarial motion priors for stylized physics-based character control. ACM Transactions Graphics (TOG), 40(4):1-20, 2021.
[31] Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in Neural Information Processing Systems, pages 305-313, 1989.
[32] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning, pages PMLR, 2018
12

---

[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding. In Advances Neural Information Processing Systems, 2022.
[34] Karl Sehmeckpeper, Oleh Rybkin, Kostas Danilidiss Sergey Levine, and Chelsea Finn. Rein- forcement learning with videos: Combining offline observations with interaction. In Conference on Robot Learning. 2021.
[35] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning explore via self-supervised world models. In International Conference on Machine Learning, 2020.
[36] Younggyo Seo, Kimin Lec, Fangchen Liu, Stephen James, and Pieter Abbeel. Harp: Autore- gressive latent video prediction with high-fidelity image generator. In 2022 IEEE International Conference on Image Processing (ICIP), pages 3944-3947. IEEE, 2022.
[37] Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. Robotics: Science and Systems, 2017.
[38] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation, pages 1134-1 1134-114,, 2018.
[39] Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILEE mobile computing and communications review, 5(1):3-55, 2001.
[40] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In International Conference Learning Representations, 2023.
[41] Emanuel Todorov. Efficient computation of optimal actions. Proceedings of the national academy of sciences, 106(28):11478-11488 2009.
[42] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In International Joint Conferences on Artificial Intelligence, 2018.
[43] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprini /807.06 2018
[44] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. iin_contoll: Software and tasks for continuous control. Software Impacts, 100022, 2020. ISSN 2665-9638.
[45] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description, arXiv preprint ar.0v:2200.02.99, 2022.
[46] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[47] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprini arXiv:2104. 10157. 2021.
[48] Lijun Yu, Yong Cheng. Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. arXiv preprint arXiv:22/2.05/11
2022.
[49] Brian D Ziebart, Andrew I. Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. Association for the Advancement of Artificial Intelligence, 2008.
13

---

[50] Konrad ZoIna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarej, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang. Task-relevant adversarial imitation learning. In Conference Robot Learning, 2020.
A Appendix
A.1 Bootstrap Your Own Latent Details
Model Architecture. The BYOL model used in this work derives from the recurrent B BYOL-Explore architecture proposed in [12], with few modifications. Namely Guo et al. propose B BYOL-Explore, multi-step predictive latent world model. The BYOL-Explore architecture trains an online network using targets generated by an exponential moving average target network on future observations. Observations o, are first encoded into an observation representation fo Online predictions are computed by then encoding the observation representation using a closed-loop RNN hg (fa (or). Guo et al. also pass actions into the closed loop RNN, but we wish learn an action-free reward mechanism that can be trained solely from videos. At each timestep, the carry by from the closed loop RNN is then fed into an open-loop RNN hg which predicts open-loop representations K steps into the future: (br,s E RM [==]==1 where b = hg(bt,k-1) The original B BYOL-Explore architecture feeds actions a++k++1 inputs to the open loop cell hg as well, but opt omit these inputs for the same reason outlined above.
k=1
Given the deterministic RNN architecture used to predict online targets, omitting the action condi- tioning may lead to poor performance multi-modal or stochastie environments. more principled approach would utilize a probabilistic recurrent state space model [9, 14] to account for the multi- modality of futue states. We leave this approach to future work.
Finally, a predictor head ge (b,,k) outputs online predictions. We refer the reader to [12] for a figure of the outlined architecture.
The target network is an observation encoder fo whose parameters are an exponential moving average of fe. The loss function for the online network is then defined as:
go(
folortt) }
LBYOL-Explore t,
sg
(be,
K(O)
1
Esplore (0)
Lsyoo.Esplore (0, k),
B(T
K(t
where (tt) min(K, - is the valid open- open-loop horizon for trajectory of length and sg is the stop-gradient operator.
Computing Rewards. The uncertainty associated with the transition (ot, aty 0c+1) is the of the corresponding prediction losses:
4 = Σ EByOL-Explore (0, j. p).,
p+g=/+1
where 0 - 9 K and S 2.
This accumulates all the losses corresponding the latent dynamics model uncertainties relative to the observation 0++1- For our purposes, we calculate rewards as = - f, which is equivalent to negating the reward used in the original BYOL-Explore formulation, and can be interpreted as negating exploration objective to ensure that the agent stays close to the trajectory distribution found in the original dataset.
14

---

A.2
Video Model Hyperparameters
Table I: Hyperparameters and training details for all VQ-GAN models
RLBench
DMC
Atari
(single multi task)
Input size
64
64 3
64 x 64 x 3
64 x 64 x 3
Latent size
x 8
16 x 16
16 16
3 (commitment loss coefficient)
0.25
0.25
0.25
Batch size
128
128
128
Learning rate
10
10
10
Learning schedule
constant
constant
constant
Training steps
200k
200k
200k
Base channels
128
128
128
Ch mult
[1, 1, 1, 2] 1
[1, 2, 21 1
[1, 2, 2]
Num res blocks
2
Codebook size
1024
1024
1024
Codebook dimension
64
64
64
Perceptual loss weight
0.1
0.1
0.1
Disc base features
32
32
32
Disc gradient penalty weight
108
10
10a
Disc max hidden feature size
512
512
512
Disc mbstd group size
4
4
4
Disc mbstd num features
1
1
Disc loss weight
0.1
0.1
0.1
Table 2: Hyperparameters and training details for all VideoGPT and MaskGit models
RLBench (single task)
RLBench (multi-task)
DMC
Atari
Sequence length
16
16
Frame skip
Latent size
x 8
16 x 16
16 x 16
16 x 16
Number of classes
16
8
n/a
34
Batch size
32
32
32
32
Learning rate
10-4
10-4
10-4
10-4
Learning rate schedule
constant
constant
constant
constant
Training steps
500k
500k
500k
500k
Adam (81,
(0.9, 0.999)
(0.9, 0.999)
(0.9, 0.999)
(0.9, 0.999)
Hidden dim
256
512
256
512
Feedforward dim
1024
2048
1024
2048
Number of heads
8
8
8
8
Number of layers
8
8
8
8
Dropout
0
0
0
15

---

A.3 Reinforcement Learning Hyperparameters
Table 3: Hyperparameters and training details for DreamerV3
RLBench (single task)
RLBench
DMC
Atari
(multi-task)
General
Replay Capacity (FIFO)
106
106
106
106
Start learning (prefill)
0
Batch Size
16
16
16
16
Batch length
64
64
64
64
MLP Size
2 x 512
4
1024
4
1024
4 x 1024
Activation
LayerNorm swish
World Model
RSSM Size
512
4096
4096
4096
Number of Latents
32
32
32
32
Classes per Latent
32
32
32
32
KL Balancing
0.667
0.667
0.667
0.667
Actor Critic
Imagination Horizon
15
15
Discount
0.995
0.995
0.995
0.995
Return Lambda
0.95
0.95
0.95
0.95
Target Update Interval
50
50
50
50
All Optimizers
Gradient Clipping
100
100
100
100
Learning Rate
10-4
10-4
10-4
10-4
Adam epsilon
10-6
10-6
10-6
10-6
Plan2Explore
Ensemble size
10
10
10
10
16

---

Table Hyperparameters and training details for DrQ
DMC
Actor Critic
MLP Size
256 x 256
CNN Features
32 X 64 x 128 x 256
CNN Filters
3 x 3 x 3
CNN Strides
2x2x2XX
Learning Rate
3e 4
Discount
0.99
Number of Critics
0.005
Init Temperature
0.1
Augmentations
Random crop
Random Network Distillation
RND CNN Features
32 x 64 x 64
RND MLP Size
512 x 512
RND learning rate
3e 4
RND Exploration Weight
17

---

A.4
Environment Training Curves
Lamp
Push button
Reach target
0.6
0.75
0.6
0.4
0.50
0.4
0.2
0.25
0.2
0.0
0.00
0.0
0.25
-0.2
0
1M
2M
1M
2M
0
1M
2M
Take umbrella out umbrella stand
Take off saucepan
Toilet seat down
1.2
1.2
0.9
0.8
0.8
0.6
0.4
0.4
0.3
0.0
0.0
0.0
1M
2M
1M
2M
Task Oracle
VIPER
AMP
Figure 12: Results across Reinforcement Learning Benchmark tasks, with mean and standard
deviation computed across seeds.
Assault
Atlantis
Boxing
Freeway
4.5
60
1.2
3.0
45
0.9
0
1.5
15
-4
0
-8
0.0
SM
10M
10M
5M
10M
5M
10M
Kangaroo
Pong
Zaxxon
6.0
1.2
4.5
0.9
1.2
3.0
0.6
1.5
0.3
0.4
0.0
0.0
0
SM
10M
5M
10M
5M
10M
Task Oracle
VIPER
AMP
Figure 13: Results across 7 Atari tasks. Scores computed using Kluman-Normailon Mean across seeds.
18

---

cartpole_balance
Cartpole_swinguss
cheetah_ree senetah_run
cup_ catch
1000
1000
750
900
750
750
500
500
600
500
300
250
250
250
0
0
5M
10M 15M
0
5M
10M
15M
5M
10M
15M
0
5M
10M
15M
finger_spie
finger turn hard
hopper @opper_stand
pendulum _andulum_swingup
1200
1000
900
M
900
900
750
600
600
600
500
300
300
300
250
0
5M
10M 15M
0
5M
10M
15M
5M
10M
15M
0
5M
10M
15M
pointmass_ed _aasy
pointmass hard
quadruped_ree runn
quadruped_ walk
1000
1000
1000
900
750
750
750
600
100M
500
500
500
300
250
250
250
5M
10M
15M
5M
10M
15M
5M
10M 15M
5M
10M
15M
reacher e
reacher hard
walker walk
1200
1000
900
900
750
MW
600
300
300
250
0
0
5M
10M 15M
0
5M
10M
15M
5M
10M 15M
Task Oracle
VIPER
AMP
AMP Multi-Task
Results across 15 DeepMind Control Suite tasks, with mean and standard deviation computed across seeds. AMP runs were stopped early due to poor performance.
Figure 14:
19

---

A.5 Training Environments
Figure 15: A single autoregressive video model is trained on 30 tasks for the Franka Panda and 23 tasks for the ReThink Robotics Sawyer, using 16 x 16 VQCodes and a context length of 4, with frame skip 4.
U
Figure 16: single task conditiodd autoregressive video model is trained on 15 DeepMind Control
Tasks, using 8 x 8 VQCodes and context length of 16.
20

---

6700
Figure 17: A single autoregressive video model is trained 7 Atari tasks, using 16 16 VQCodes and context length of 16.
21

---

A.6 Visualizing Video Prediction Uncertainty
Video model uncertainty can be visualized by upsampling the conditional likelihoods over VQCodes. Since VQCodes tend to model local features, this provides useful tool for analyzing which regions of the image the video model is uncertain about. We visualize video prediction uncertainty for three environments below:
RGB
MIPER
20
40
60
80
100
120
(a) Reference Trajectory
RGB
VIPER
20
40
60
80
100
120
(b) Random Trajectory
Figure 18: Uncertainty visualized for a reference trajectory (top) and a random trajectory (bottom). Brighter values correspond to higher likelihoods. For the random trajectory, the video model assigns lower likelihoods to regions of the image containing the ball. This is especially true for random trajectories, where the video model, trained solely on expert trajectories, cannot accurately predict what happens when the agent plays sub-optimally.
RGB
MIPER
20
40
60
80
100
120
Figure 19: Uncertainty visualized for random trajectory from the dme cartpole balance task. Brighter values correspond to higher likelihoods.
22

---

RGB
VIPER
20
40
60
80
100
(a) Reference Trajectory
RGB
PVPER
20
40
60
80
100
(b) Random Trajectory
Figure 20: Uncertainty visualized for a reference trajectory (top) and a random trajectory (bottom). Brighter values correspond to higher likelihoods. Notice the high uncertainty over the position of objects on the table for the first frame. This corresponds to the unconditional likelihood (with no context). Since the position of the saucepan randomized for every episode, the video model assigns some uncertainty to this initial configuration. For the random trajectory, there is high uncertainty assigned the position of the arm since the video model has not seen trajectories that display poor behavior.
23

---

A.7
VIPER Out-of-distrlbution Analysis
1e3
-0.5
VIPER
-1.0
-1.5
-2.0
Expert*
Expert
Suboptimal
Suboptimal
Random Pixels
24 In-Dist Tasks
7 OOD Tasks
24 In-Dist Tasks
7 OOD Tasks
Figure 21: Predicted y VIPER various RLBench tasks. We evalute 10 trajectories for each task.
*Training data for the VIPER model.
24


Reviews:
[{"review_id": "PaaNTAlENf", "reviewer": "Reviewer_Zre9", "summary": "This paper proposes a simple method that uses a pre-trained video prediction model to provide rewards for online RL. The design includes using VQ-GAN to encode discrete embeddings and incorporating an exploration bonus (opt for Plan2Explore and RND). In experiments, the authors also show the learned rewards provide a useful learning signal for online RL. ", "questions": "1. Can the author compare the proposed method to more imitation learning papers?\n2. Doesn't the method suffer from the problem of OOD issues when there is very little expert data? Even when a dozen or so pieces of expert data exist, it seems that the OOD problem exists, i.e., the pre-trained video prediction model may falsely overestimate the probability of some behaviors that are not expert behaviors.\n3. After I thought deeply about it, I always thought that there is an OOD problem with the method, which is consistent with standard offline RL, as the policy network will make an effort to explore and discover behaviors with a high probability/likelihood, however, these behaviors may be falsely overestimated by the video prediction network. \n4. In the main paper, I did not see the results that \"VIPER can achieve expert-level control without task rewards on 15 DMC tasks, 6 RLBench tasks, and 7 Atari tasks\". \n5. In addition, the authors only emphasize achieving expert-level performance and do not compare it to a large number of imitation learning baselines. This tends to raise doubts about the performance of the method, since with enough expert data, simple behavioral cloning can also achieve expert-level performance. ", "limitations": "The authors briefly discuss the limitations of the paper. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well-written and the idea is clear.\n2. The authors make comparisons on multiple tasks. ", "weaknesses": "1. The effectiveness of the proposed method may be limited if the expert data is scarce. \n2. In many imitation learning papers almost only one expert trajectory is needed, however, this paper undoubtedly requires a lot of expert data (to train the video prediction model).\n3. Although the authors make comparisons on multiple tasks, there are few baselines. There are many papers on imitation learning that do not make experimental comparisons, e.g. [1, 2, 3, 4, 5]. \n\n[1] Optimal Transport for Offline Imitation Learning\n[2] Demodice: Offline imitation learning with supplementary imperfect demonstrations\n[3] Behavioral cloning from observation\n[4] Generative adversarial imitation from observation \n[5] CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We provide detailed answers to the questions below.\n\n**[Weakness 1 & 2] The effectiveness of the proposed method may be limited if the expert data is scarce. … this paper undoubtedly requires a lot of expert data (to train the video prediction model).**\n\nFigure D in the global response PDF shows an additional ablation study on the dataset size. We find that VIPER achieves high performance even with only 10 expert videos (10% of the original dataset size), and learns a less efficient but still meaningful behavior even with only 1 expert video.\n\nThe primary motivation for our work is to derive rewards from video models trained on diverse data rather than a few task-specific demonstrations, as general video models will likely become available in the future. In our paper, we train a single video model per suite (e.g. one video model for 50 RLBench tasks).\n\nWhile prior works, such as GAIfO [A] and AMP [B], struggle with mode collapse when the data is diverse, VIPER scales well as more data is used. Moreover, by learning from diverse videos, VIPER can exhibit generalization to an unseen tabletop manipulation setting (Section 4.3).\n\n**[Weakness 3 & Q1] Can the author compare the proposed method to more imitation learning papers?**\n\nWe have included additional baselines, GAIfO and BCO [C] in Figure A of the PDF in the global response. The results show that BCO fails to learn from pixels in all three task suites and GAIfO performs worse than VIPER but similarly to the AMP baseline.\n\n- GAIfO is similar to the AMP baseline in the original submission but optimizes KL Divergence instead of Pearson Divergence. Both perform similarly in our experiments and are outperformed by VIPER. Both GAIfO and AMP are prone to mode collapse in the multi-task setting, whereas VIPER scales favorably with the number of tasks.\n\n- We had originally compared VIPER to BCO but found that it performed poorly from pixels even with extensive tuning, and thus we did not include it originally (Figure A of global response PDF). Further investigation indicates that its inverse dynamics model struggles with the hard generalization challenge of labeling the expert videos with actions after being only trained on the suboptimal trajectories encountered by the agent so far.\n\nAs suggested by Reviewer Zre9, we will include the BCO and GAIfO results in the revised paper.\n\n**[Q2 & Q3] Doesn't the method suffer from the problem of OOD issues when there is very little expert data?**\n\nFor the ability of VIPER to learn from very little expert data, such as a single expert video trajectory, we refer to Figure D in the global response PDF (as discussed above).\n\nTo show the OOD generalization of VIPER more generally, the original submission includes Figure 8 that shows generalization of the video model to unseen combinations of robot arms and tasks, and successful policy learning based on this reward.\n\nTo further address this point, we conducted an additional analysis of the rewards assigned by VIPER to various trajectory distributions after training on only expert videos of in-distribution tasks (Figure B in the global response PDF). Notably, we find that VIPER assigns higher rewards to expert behavior on OOD tasks than to suboptimal and random behaviors on in-distribution tasks. The reward histograms show that VIPER reflects different levels of task performance well, even out of distribution.\n\n**[Q4] In the main paper, I did not see the results that \"VIPER can achieve expert-level control without task rewards on 15 DMC tasks, 6 RLBench tasks, and 7 Atari tasks\".**\n\nThank you for pointing this out. As shown in Figure 3 of the original paper, VIPER performs similarly to the agents trained with the ground-truth task rewards on the DMC, RLBench, and Atari tasks **on average**. Specifically, VIPER achieves expert-level performances on 6/6 RLBench tasks (Figure 10), 5/7 Atari tasks (Figure 11), and 8/15 DMC tasks (Figure 12). We will clarify and state this explicitly in the revised paper.\n\n**[Q5] In addition, the authors only emphasize achieving expert-level performance and do not compare it to a large number of imitation learning baselines. This tends to raise doubts about the performance of the method, since with enough expert data, simple behavioral cloning can also achieve expert-level performance.**\n\nWe refer to our answer above to **Weakness 3 & Q1**, where we discuss two additional baselines that we added to the paper: GAIfO and BCO (Figure A of the PDF in the global rebuttal). The adversarial methods do not perform as well as VIPER even in the single task setting, and perform poorly in the multi-task setting due to mode collapse. \n\nWe would also like to clarify that VIPER _does not_ assume access to ground truth actions, whereas simple behavioral cloning does.\n\n---\n\nWe again thank the reviewer for their insightful comments, and look forward to engaging with them in the reviewer-author discussion period. Please let us know if there are any concerns preventing you from raising your score.\n\n[A] AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. Peng et al. SIGGRAPH 2021\n\n[B] Generative adversarial imitation from observation. Torabi et al. arXiv 2018\n\n[C] Behavioral Cloning from Observation. Torabi et al. IJCAI 2018\n"}, {"review_id": "9HFjyssDit", "reviewer": "Reviewer_BsvD", "summary": "The paper proposes to use a large video-prediction model for learning a reward model for RL. The agent's performance is evaluated on a total of 8 envs from 3 benchmarks: DMC, Atari, and RLBench. The paper argues that the proposed model also generalizes to different environments for which no training data was provided, enabling cross-embodiment generalization for tabletop manipulation.", "questions": "1. In line 128, the paper states \"For example, when flipping a weighted coin with p(heads = 0.6) 1000 times, typical sequences will count roughly 600 heads and 400 tails, in contrast to the most probable sequence of 1000 heads that will basically never be seen in practice\". Could the authors explain why is the sequence of 1000 heads the most probable one ?\n2. Does the algorithm work with good trajectories as well or does it need access to expert trajectories? e.g. in line 172, what if they were using the top 650 to top 550 episodes, in place of the top 100 episodes. This would make for an useful ablation experiment.\n3. Arent the video datasets \"too small\"? Given that the video models are trained for hundreds of thousands of updates, I wonder if the video models are drastically overfitting, leading to (i) the learned policies not showing any diverse behaviours and (ii) the learned policies failing with stochastic envs. This would make for another useful ablation experiment.\n4. Line 201 states that TPUs were used for training while 226 states that GPUs were used for training. Which is it :)", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and easy to follow.\n* Section 4 is very well organized. It starts by asking base questions like \"can VIPER provide an adequate learning signal for solving a variety of tasks?\" before jumping on to the evaluation of the performance of the RL algorithm.", "weaknesses": "Limited baselines: The paper compares with just 1 baseline (the second \"baseline\" is more of an ablation of the first baseline). e.g. there is https://sites.google.com/view/vip-rl that claims to provide \"dense visual reward\". The paper itself lists a bunch of baselines (in the related work) but does not compare to them.\n\nLimited ablations: See questions.\n\nOverall, I think the paper is interesting but I want to see performance improvement over a bunch of baselines and some ablations. I would encourage the authors to engage during the rebuttal period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their helpful comments. We provide detailed answers to their questions below.\n\n**[Weaknesses] Overall, I think the paper is interesting but I want to see performance improvement over a bunch of baselines and some ablations.**\n\nWe have included additional baselines, GAIfO [B] and BCO [C] in Figure A of the PDF in the global rebuttal. The results show that BCO fails to learn from pixels in all three task suites and GAIfO performs worse than VIPER but similarly to the AMP baseline [A].\n\n- GAIfO is similar to the AMP baseline in the original submission but optimizes KL Divergence instead of Pearson Divergence. Both perform similarly in our experiments and are outperformed by VIPER. Both GAIfO and AMP are prone to mode collapse in the multi-task setting, whereas VIPER scales favorably with the number of tasks.\n\n- We had originally compared VIPER to BCO but found that it performed poorly from pixels even with extensive tuning, and thus we did not include it originally (Figure A of global response PDF). Further investigation indicates that its inverse dynamics model struggles with the hard generalization challenge of labeling the expert videos with actions after being only trained on the suboptimal trajectories encountered by the agent so far, especially when trained from pixels.\n\n**[Q1] Why is the sequence of 1000 heads the most probable one?**\n\nWe clarified this thought experiment in the paper as discussed here. Flipping a biased coin with $p(\\text{heads}=0.6)$ 1000 times usually yields sequences with roughly 600 heads and 400 tails. Those sequences roughly have probability $0.6^{600} \\times 0.4^{400} = 5e^{-293}$. In contrast, the sequence of 1000 heads has the probability of $0.6^{1000} = 1e^{-222}$, which is  more likely (and \"degenerate\") than the sequence with 600 heads, $5e^{-293}$. \n\nSimilarly, if the policy aims to replicate the most probable trajectory under the video model, its behavior may be very different from the reference videos the video model was trained on and possibly result in degenerate behavior. We will make this clearer in the revised version.\n\n**[Q2] Does the algorithm work with good trajectories as well or does it need access to expert trajectories?**\n\nGiven that the RL objective is to maximize the likelihood of agent trajectories under the video model, when trained with good trajectories (we interpreted it as suboptimal trajectories), the policy would learn behaviors similar to those in the distribution of good trajectories. \n\nWe have conducted additional experiments to verify this (Figure C of the PDF in the global rebuttal). We found that VIPER can extract reward signals from good, but suboptimal, trajectories, which achieve 50-75% the return of the expert trajectories (this varies between tasks, but these trajectories achieve between 500 and 800 undiscounted return). Additionally, we observed faster convergence when training from sparse task rewards in conjunction with rewards extracted from a VIPER model trained on “good” data.\n\n**[Q3] Aren’t the video datasets “too small”? … I wonder if the video models are drastically overfitting… leading to (i) the learned policies not showing any diverse behaviours and (ii) the learned policies failing with stochastic envs**\n\nWe did not observe much overfitting as our video models were generally small (~10M parameters). Depending on the environment, training curves for test set loss generally converged to a point at or slightly higher than the train set loss. We point the reviewer to Figures 7 & 8 in the original submission, where we probe the video model’s ability to generalize to an out-of-distribution task. Figure 8 shows that this generalization enables VIPER to provide a good reward signal for policy learning in an OOD environment. \n\nTo better understand performance of VIPER on OOD data, we have additionally evaluated VIPER rewards on both in-distribution and out-of-distribution data ranging from suboptimal to expert (Figure B of the PDF in the global rebuttal). Our analysis suggests that VIPER learns a reward model that is not adversely affected by OOD training samples, and video prediction rewards are consistently higher for expert trajectories, even for tasks which were not seen during video model training. In summary, VIPER exhibits generalization to OOD states and tasks which were not encountered during video model training.\n\nRegarding diversity, we note that RLBench demonstrations cover multiple modes because they are gathered by a custom random shooting trajectory optimizer. Additionally, each RLBench environment is randomized for each demo. Additionally, even though the DMC and Atari environments are deterministic, the policies used to gather the demonstrations are stochastic, leading to diverse behaviors.\n\n**[Q4] Were TPUs or GPUs used?**\n\nSorry for the confusion. All video models were trained using TPUs (v3), and all RL models were trained on GPUs (V100, A100). In general, both video models and RL models can be trained on either TPUs or GPUs. We will clarify this in the revised version.\n\n---\n\nWe would be delighted if you could confirm which of your concerns we resolved and whether there are any issues remaining that might prevent acceptance of the paper.\n\n\n\n\n[A] AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. Peng et al. SIGGRAPH 2021\n\n[B] Generative adversarial imitation from observation. Torabi et al. arXiv 2018\n\n[C] Behavioral Cloning from Observation. Torabi et al. IJCAI 2018"}, {"review_id": "pXcniyF68f", "reviewer": "Reviewer_SQjv", "summary": "This paper proposes to use prediction likelihoods from autoregressive video models as reward functions to train reinforcement learning agents. Specifically, the conditional likelihood $\\log p(x_{t+1}|x_{1:t})$ is augmented with an exploration reward to avoid suboptimal behavior. The authors conduct extensive experiments and show that likelihoods of autoregressive video models can be effective for reward specification. They also show that in certain cases the video models can generalize to unseen task domains, encouraging satisfying behaviors. Their ablation study compares different video models, different exploration objectives, and different context lengths. ", "questions": "Apart from the question I raised in Weakness, I hope the authors would also like to resolve my concerns in Section 4.3. \n\nWhile the paper claimed that specifying reward functions with video models can generalize to OOD tasks, Section 4.3 only demonstrate a particular case where there is a recombination of robot and objects to be manipulated. Is it possible to make the evaluation of generalization more systematic? I guess readers may be more interested in a discussion of what \"types\" of generalization are possible to eliminate the influence of particularity. ", "limitations": "As stated in Weakness, there is a technical limitation of the proposed method that the authors do not seem to notice. Other limitations are well documented in Section 5. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality: Though the idea of using likelihood of states/observations as a reward is not novel, taking the temporal coherence into consideration with an autoregressive factorization is novel at my end. \n\nQuality: This work is strong in its efforts in extensive experiments. \n\nClarity: This paper is straightforward to follow. The narrative is very intuitive. Experimental details are very well documented. \n\nSignificance: Learning memory-based reward function with sequence modeling is an interesting direction to explore given the current advances of generative models. ", "weaknesses": "In spite of the impressive amount of experiments presented in this work, one fundamental problem unresolved in this work is why the autoregressive likelihood, which is inherently non-Markov, can work with general RL algorithms, in which TD learning strongly depends on Markovian rewards. Latent-space model-based RL methods such as Dreamer used in this work are too particular because the latent-space modeling may resolve the limitation of TD learning as a byproduct. This means the empirical result from this work cannot be trivially generalized to other RL methods, rendering the thesis statement an overclaim. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments. We provide detailed answers to your questions below:\n\n**[Weaknesses]  one fundamental problem unresolved in this work is why the autoregressive likelihood, which is inherently non-Markov, can work with general RL algorithms, in which TD learning strongly depends on Markovian rewards.**\n\nThank you for pointing out the non-Markovian nature of VIPER. We agree that non-Markovian VIPER rewards could be an issue in TD learning. However, in practice, state-of-the-art RL algorithms based on TD learning can handle many non-Markovian rewards (in the simplest case by leveraging frame stacking), such as the rewards provided by adversarial imitation learning [A, B], Minecraft [C], Crafter [D], and many more POMDP environments. We will make this point clear in the revised paper. Despite the non-Markovian property of  VIPER, we show that VIPER works well with a model-free off-policy RL algorithm, DrQ, which does not use a recurrent latent state model, in Figure 3 of the original paper. \n\n**[Q1]: Is it possible to make the evaluation of generalization more systematic? I guess readers may be more interested in a discussion of what \"types\" of generalization are possible to eliminate the influence of particularity.**\n\nIn this work, we focus mainly on the application of video prediction models to reward specification and show one specific type of generalization: cross-embodiment generalization, whereby the video prediction model is trained on videos of tasks for the sawyer and panda arm, with some tasks held out for the panda arm, requiring that the model generalize and provide accurate rollouts for the held out tasks.  We have extended this analysis to include more tasks (Figure B of the PDF in the global rebuttal). Our analysis suggests that VIPER learns a reward model that is not adversely affected by OOD training samples, and exhibits generalization to OOD tasks which were not encountered during video model training. Video prediction rewards are consistently higher for expert trajectories, even for tasks which were not seen during video model training.\n\nRelated works such as UniPi [E] conduct more extensive combinatorial generalization experiments that evidence the current capabilities of large video prediction models. We believe scaling up the training dataset and video model size should lead to a more general reward model. \n\n---\n\nWe thank the reviewer again for their insightful comments, and look forward to engaging with them in the reviewer-author discussion period. Please let us know if there are any concerns preventing you from raising your score.\n\n\n[A] AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. Peng et al. SIGGRAPH 2021\n\n[B] Generative adversarial imitation from observation. Torabi et al. arXiv 2018\n\n[C] MineRL: A Large-Scale Dataset of Minecraft Demonstrations. Guss et al. IJCAI 2019\n\n[D] Benchmarking the Spectrum of Agent Capabilities, Hafner. ICLR 2022\n\n[E] Learning Universal Policies via Text-Guided Video Generation. Du et al. arXiv 2023\n"}, {"review_id": "OO2NDPm3Kv", "reviewer": "Reviewer_d6Ah", "summary": "This paper proposes a learning-from-observation algorithm that builds a reward based on a video predictor trained from action-free expert videos. Experimental results show that online reinforcement learning algorithms can learn a working policy from their reward only effectively. This paper contains rich and informative ablation studies and analyses to verify their design choices.", "questions": "1.\tHow good is the generalization ability of VIPER? We definitely need evaluations on more OOD combinations to support the statement in the third contribution. \n2.\tFor Fig.8, which OOD combination the curve shows? In addition, this curve doesn’t include an error bar like other experiments in the paper.", "limitations": "The authors listed and discussed the limitations including the lack of in-domain expert data in the real world, the sub-optimal performance with stochastic data, and the sensitive performance to the VQCode size and context length.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper writing is clear and easy to understand\n2.\tDistilling knowledge from action-free videos to policies is a promising future direction for robotics.\n3.\tExperiments are rich. The authors test their method with two different online RL methods, two different exploration losses, three task domains, and three different video prediction models.", "weaknesses": "The experiments on the generalization ability (Sec. 4.3) are not convincing enough. The video prediction model in Sec.4.3 is trained with 23 Rethink-robot-arm tasks and 30 Franka-robot-arm tasks. There should be dozens of OOD arm/task combinations that can be evaluated. However, according to L300, we only see the performance on only ONE OOD combination. How is the performance on other OOD combinations? Besides, the learning curve in Fig.8 doesn’t include a task oracle like other experiments in the paper. So we also don’t know how good the OOD performance is. Therefore, I think the third contribution of this paper, “VIPER generalizes to different environments”, is not well-supported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer d6Ah for their thoughtful comments, and provide detailed answers to their questions below.\n\n**[Q1] How good is the generalization ability of VIPER? We definitely need evaluations on more OOD combinations to support the statement in the third contribution.**\n\nThank you for pointing this out. We agree that a more thorough understanding of out-of-distribution robustness is helpful to better understand the performance of VIPER on OOD data. Thus, we have additionally evaluated VIPER on both in-distribution (24 training tasks) and out-of-distribution (7 held-out tasks) data ranging from suboptimal to expert (Figure B of the PDF in the global rebuttal).\n\nOur analysis suggests that VIPER learns a reward model that is not adversely affected by OOD training samples, and video prediction rewards are consistently higher for expert trajectories, even for tasks which were not seen during video model training. On the other hand, adversarial imitation learning baselines, AMP [A] and GAIfO [B], exhibit poor generalization properties [C] due to its instability in training and mode collapse, as explained in [D - Section 4.2]. In summary, VIPER exhibits generalization to OOD states and tasks which were not encountered during video model training. Given the relatively small size of the training dataset, we were excited to see the video model generalize to novel arm/task combinations. Scaling up the video training dataset and model size may yield stronger generalization results.\n\nWe are also in the process of running RL experiments for more OOD arm/task combinations. We will include these results in the revised version.\n\n\n**[Q2] For Fig.8, which OOD combination the curve shows? In addition, this curve doesn’t include an error bar like other experiments in the paper.**\n\nThe left plot in Figure 8 shows a training run for the OOD arm/task combination visualized in Figure 7: the panda performing take_lid_off_saucepan. We will update the main text of the paper to better reflect this.We only included one trajectory for this visualization / training curve, but will update the revised text to include more RL training runs. Figure B the revised PDF demonstrates that VIPER provides meaningful rewards on 7 OOD tasks on the panda: push_button, push_button(s), toilet_seat_down, reach_target, take_lid_off_saucepan, take_umbrella_out_of_stand, reach_target.\n\n---\n\nWe would like to thank the reviewer once again for their insightful comments, and look forward to engaging with them in the reviewer-author discussion period.\n\n[A] AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. Peng et al. SIGGRAPH 2021\n\n[B] Generative adversarial imitation from observation. Torabi et al. arXiv 2018\n\n[C] Learning Robust Rewards with Adversarial Inverse Reinforcement Learning. Fu et al. arXiv 2018\n\n[D] Large Scale GAN Training for High Fidelity Natural Image Synthesis. Brock et al. arXiv 2019\n"}, {"review_id": "1r9TtJXs0r", "reviewer": "Reviewer_qQw6", "summary": "The authors present Video Prediction Rewards, an algorithm that leverages transformer-based video prediction models as action-free reward signals for reinforcement learning. The reward induced by this video prediction model incentivizes the agent to find the most likely trajectory under the expert video distribution. By further incorporating some exploration rewards, such as RND, the proposed method obtains good performance across a wide range of DMC, Atari, and RLBench tasks.", "questions": "See the weakness", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to read.\nThe authors aim to address a crucial problem in reinforcement learning, i.e., the reward function design. The authors propose a concise method, and experimental results also validate the effectiveness of the approach.\n", "weaknesses": "I'm concerned about the problem of out-of-distribution. Can the pre-trained video prediction models accurately evaluate unseen behaviours?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that the reviewer recognized the importance of learning reward functions for RL and found our method concise and effective across a wide range of tasks. The PDF in the global rebuttal contains:\n\n- Additional experiments with more baselines [A, B] (Figure A)\n- Further analysis of OOD generalization (Figure B), and an analysis of VIPER models trained on sub-optimal data (Figure C)\n- An analysis of VIPER performance as a function of dataset size (Figure D).\n\n---\n\nPlease let us know if there are any concerns preventing you from raising your score. If there are any suggestions or concerns that we could address to strengthen the paper further, we would be glad to engage during the discussion period.\n\n[A] Generative adversarial imitation from observation. Torabi et al. arXiv 2018\n\n[B] Behavioral Cloning from Observation. Torabi et al. IJCAI 2018\n"}, {"review_id": "UkYGcAhihS", "reviewer": "Reviewer_1FB4", "summary": "This paper proposes Video Prediction Rewards (VIPER), a general architecture that extracts reward functions from action-free expert demonstrations. To match the agent's trajectory distribution with the source distribution, VIPER optimizes the agent to maximize a log-likelihood estimated by an auto-regressive video model and an entropy term to encourage exploration.\n\nExperiments on DMC, Atari, and RLBench demonstrate the soundness and efficiency of the reward function extracted by VIPER.", "questions": "- To my understanding, VIPER's setting is similar to Generative Adversarial Imitation Learning (GAIL), while GAIL uses the critic as the surrogate of the distance between the expert trajectory distribution and generated trajectory distribution, VIPER directly estimates the distance (KL divergence) by modeling the log-likelihood with a generative model. I have some reservations regarding the benefits of doing so.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "VIPER addresses the practical challenge of how to extract reward functions from action-free expert demonstrations in order to optimize our agents, which is useful in settings like self-driving.\nVIPER has the following strengths:\n- VIPER can extract effective reward functions and thus promote policy optimization in a range of visual control tasks.\n- Experiments show that reward functions learned by VIPER can generalize to a variety of tasks and even OOD tasks.", "weaknesses": "Although VIPER shows good experiments results, some weaknesses still exist:\n\n- The data efficiency of VIPER seems to be low as it requires nearly 10M data to converge in DMC. Also, it seems that VIPER can not leverage sub-optimal demonstrations, which could be important for improving data efficiency.\n- It could be difficult and expensive to acquire a generative video model for real-world tasks, especially with visual distractors. \n- Also, I think current tasks are a little bit less challenging, and thus it might be easier to define a reward function than acquire expert demonstrations. Therefore, it could be interesting if we could test VIPER's performance with tasks that are hard to define rewards, e.g. embodied tasks like [Habitat](https://github.com/facebookresearch/habitat-sim) or self-driving platforms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments. We provide detailed answers to your questions below:\n\n**[Weakness 1-1] The data efficiency of VIPER seems to be low as it requires nearly 10M data to converge in DMC.**\n\nWhile agents trained with VIPER may require millions of training steps to converge in DMC, agents trained with relevant baselines [A] [B] [C] either take longer to converge, or do not converge at all (additional baselines can be found in Figure A of the PDF in the global rebuttal). Additionally, VIPER outperforms learning from sparse task rewards for all evaluated RLBench tasks (Figure 10, supplementary material) and some Atari tasks (Figure 11, supplementary material). This is quite a promising result, as we can leverage videos to successfully complete tasks which cannot be learned from the sparse task reward. Additionally, VIPER exhibits generalization to a specific cross-embodiment manipulation setting (Figure 7 & 8 of the original submission).\n\n**[Weakness 1-2] Also, it seems that VIPER can not leverage sub-optimal demonstrations, which could be important for improving data efficiency.**\n\nWe have conducted additional experiments to verify this (Figure C in the global response PDF). We found that VIPER can still extract reward signal from suboptimal trajectories, which achieve 50-75% return of the expert trajectories. Additionally, we observed faster convergence when training from sparse task rewards in conjunction with rewards extracted from a VIPER model trained on suboptimal data.\n\n**[Weakness 2] It could be difficult and expensive to acquire a generative video model for real-world tasks, especially with visual distractors.**\n\nFigure D of the PDF in the global rebuttal shows further ablations of running VIPER trained on progressively smaller fractions of data, where performance starts degrading at around 10% of the original training data, which amounts to 10 videos in DMC and 10 videos in RLBench. The policy learns less efficient but still meaningful behavior even when VIPER is trained on only 1 expert video.\n\nThank you for pointing this out. We believe that a promising future direction for VIPER is to leverage pretrained large-scale video generation models, as mentioned in the conclusion section. Doing so relaxes the data requirements for training from scratch, and allows the video model to better generalize to potentially unseen tasks and visual distractors. UniPi [D] shows that this is possible by fine-tuning a large video model on a set of task-specific robotics data, where the resulting model can generate accurate video predictions starting from novel task settings. Therefore, in the VIPER setting, such a model can potentially be used as a generalizable reward function for VIPER training on a diverse range of tasks.\n\n**[Weakness 3] Also, I think current tasks are a little bit less challenging, and thus it might be easier to define a reward function than acquire expert demonstrations. Therefore, it could be interesting if we could test VIPER's performance with tasks that are hard to define rewards, e.g. embodied tasks like Habitat or self-driving platforms.**\n\nThank you for suggesting interesting and challenging environments to test VIPER. As much as we are excited about the application of VIPER to more challenging environments, we would like to highlight that it is not trivial to define rewards for the robotic manipulation tasks in RLBench. As can be seen in Figure 3 (right) and Figure 10, DreamerV3 failed to learn all 6 tasks with the oracle task rewards whereas VIPER were able to provide more informative rewards. Additionally, The authors consider DMC and Atari to be an important set of benchmarks to consider for this work, as they are a popular choice for evaluating RL algorithms, and make it easy to compare methods. We will leave examining VIPER on more challenging domains, such as Habitat and self-driving, to future work.\n\n**[Q1] To my understanding, VIPER's setting is similar to Generative Adversarial Imitation Learning (GAIL)... I have some reservations regarding the benefits of doing so.**\n\nAs the reviewer mentioned, adversarial imitation learning approaches (GAIL, GAIfO, and AMP) have a similar setting with VIPER. However, adversarial training is notoriously prone to mode collapse, exhibits poor generalization properties, and is often challenging to stabilize. Please refer to [E, F] for more thorough discussion about challenges in adversarial training.\n\nVIPER does not formulate KL Divergence minimization as a two-player game. Instead, VIPER treats the video prediction model as a pre-trained and fixed reward model which the RL agent is trained to maximize. As such, VIPER does not exhibit mode collapse. On the other hand, Figure 3 of the original paper and Figure A in the global rebuttal show that VIPER outperforms the adversarial imitation learning baselines (AMP and GAIfO) both in the single and multi-task setting. For example, the VIPER model trained on 16 DMC tasks provides a good reward signal for downstream policy learning, while the corresponding adversarial imitation learning baselines trained on 16 DMC tasks exhibit mode collapse and do not enable policy learning.\n\n---\n\nWe would like to thank the reviewers for their thoughtful comments. Please let us know if there are any concerns preventing you from raising your score.\n\n[A] AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control. Peng et al. SIGGRAPH 2021\n\n[B] Generative adversarial imitation from observation. Torabi et al. arXiv 2018\n\n[C] Behavioral Cloning from Observation. Torabi et al. IJCAI 2018\n\n[D] Learning Universal Policies via Text-Guided Video Generation. Du et al. arXiv 2023\n\n[E] Learning Robust Rewards with Adversarial Inverse Reinforcement Learning. Fu et al. ICLR 2018\n\n[F] Large Scale GAN Training for High Fidelity Natural Image Synthesis. Brock et al. ICLR 2019"}]

Author rebuttal:
We thank all reviewers for their constructive feedback and for helping us make our paper a stronger submission! 

- First of all, we highlight that we have included **two new imitation learning from observation baselines (Figure A)**, BCO and GAIfO. In short, VIPER consistently shows significant improvement over these baselines on all three task suites. We also hypothesize reasons as to why VIPER outperforms the adversarial and behavioral cloning approaches.

- We have also conducted additional experiments to show the generalization capability of **VIPER on out-of-distribution tasks (Figure B)**. We find that VIPER consistently predicts higher rewards for expert trajectories on both in-distribution and out-of-distribution tasks.

- Additionally, we have added new ablation studies on the use of **suboptimal data (Figure C)**. We found that VIPER can still extract reward signals from suboptimal trajectories. We also observed faster convergence when training from sparse task rewards in conjunction with VIPER rewards trained on suboptimal data.

- Finally, we have added an ablation studying the effect of **dataset size (Figure D)** on the learned VIPER model. We find that VIPER learns a less efficient but still meaningful behavior even with only 1 expert video.

We included these new experimental results in the attached PDF. We hope we have addressed all your concerns and questions. Please let us know if there are any concerns preventing you from raising your score.

Meta-review: The paper proposes novel technique for learning prediction models from unlabeled video and leveraging these models to better enable vision-based behavior learning. How this type of behavior learning should be done is an important question within the NeurIPS community, and the paper provides a compelling and interesting entry in the discussion; it is clear, well-written, and provides compelling empirical evidence that the proposed technique is effective. 

The authors successfully engaged in a discussion with the reviewers regarding the ability of the method to generalize in out-of-distribution scenarios, and they’re encouraged to strengthen the original submission with the additional results and discussion on this point and others that arose during the discussion period.
Decision: Accept (poster)