VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training
Authors: Yecheng Jason Ma
Shagun Sodhani
Dinesh Jayaraman
Osbert Bastani
Vikash Kumar
Amy Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Pre-Training for Control
Offline RL
Goal-Conditioned RL
Deep RL
Robot Learning
Self-Supervised Learning
Visuomotor Control

Abstract: Reward and representation learning are two long-standing challenges for learning an expanding set of robot manipulation skills from sensory observations. Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purpose reward learning remains an open question. We introduce $\textbf{V}$alue-$\textbf{I}$mplicit $\textbf{P}$re-training (VIP), a self-supervised pre-trained visual representation capable of generating dense and smooth reward functions for unseen robotic tasks. VIP casts representation learning from human videos as an offline goal-conditioned reinforcement learning problem and derives a self-supervised dual goal-conditioned value-function objective that does not depend on actions, enabling pre-training on unlabeled human videos. Theoretically, VIP can be understood as a novel implicit time contrastive objective that generates a temporally smooth embedding, enabling the value function to be implicitly defined via the embedding distance, which can then be used to construct the reward for any goal-image specified downstream task. Trained on large-scale Ego4D human videos and without any fine-tuning on in-domain, task-specific data, VIP can provide dense visual reward for an extensive set of simulated and $\textbf{real-robot}$ tasks, enabling diverse reward-based visual control methods and significantly outperforming all prior pre-trained representations. Notably, VIP can enable simple, few-shot offline RL on a suite of real-world robot tasks with as few as 20 trajectories.

Published as a conference paper at ICLR 2023
TOWARDS UNIVERSAL VISUAL REWARD AND REPRE- SENTATION VIA VALUE-IMPLICIT PRE- TRE-TRAINING
Yecheng Jason Ma Shagun Sodhani', Dinesh Jayaraman', Osbert Bastani", (Vikash Kumart Amy Zhangi")
FAIR, Meta Al University of Pennsylvania"
https://sites: http:///te...ooole...co///ee////
ABSTRACT
Reward and representation learning are two long-standing challenges for learning expanding of robot manipulation skills from sensory observations. Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purge purposs reward learning remains an open question. We introduce Value-Implicii Pre-training (VIP), self-supervise pre-trained visual representation capable of generating dense and smooth reward functions for unseen robotic tasks. VIP casts representation learning from human videos as an offline goal-conditioned reinforcement learning problem and derives a self- supervised goal-conditioned value-function objective that does not depend on actions, enabling pre-training on unlabeled human videos. Theoretically, VIP be understood novel implicit time contrastive objective that generates temporally smooth embedding, enabling the value function to be implicitly defined via the embedding distance, which can then be used construct the reward function for any -male specified downstream task. Trained on large-scale Ego4D human videos and without any fine-tuning on in-domain, -secifcci data, VIP can provide dense visual reward for an extensive set of simulated and real-robot tasks, enabling diverse reward-based visual control methods and outperforming all prior pre-trained representations. Notably, VIP can enable simple, femple.fwwsthe few-shot offline RL on a suite real-worle robot tasks with as few as 20 trajectories.
INTRODUCTION
A long-standing challenge in robot learning is to develop robots that can learn a diverse and expanding set of manipulation skills from sensory observations (e.g., vision). This hope of developing general- purpose robots demands scalable and generalizable representation learning and reward learning to provide effective task representation and specification for downstream policy learning. Inspired
by pre-training successes in computer vision (CV) (He et al.] 2020 2022 and natural language
processing (NLP) (Devlin et al., 2018 Radford et al. [201 (2021) pre-training visual representations out-of-domain natural and human data (Deng et al.] [2009] Grauman et al.) (022)) has emerged as an effective solution for acquiring a general visual representation for robotic manipulation (Shah & Kumar 2021 Parisi al. 20222] Nair al. 2022 Xiao al. 2022) This paradigm is favorable the traditional approach of in in-domain representation learning because it does not require any intensive specispe data collection or representation fine-tuning, and a single fixed representation can be used for variety of unseen robotic domains and tasks (Parisi et al. 2022).
key unsolved problem pre-training for robotic control the challenge of reward specification. Unlike simulated environments, real-world robotics tasks do not come with privileged environment state information or a well-shaped reward function defined over this state space. Prior pre-trained rep- resentations for control demonstrate results in only visual reinforcement learning (RL) in simulation. assuming access to a well-shaped dense reward function (Shah & Kumar 2021 Xiao et al. 2022). or
"Corresponding author; jasonyma@seas www.n.een......m........ edu. f equal advising, randomized order.

---

Published as a conference paper at ICLR 2023
Value-Implicit Pre-training (VIP)
Diverse Visuomotor Control:
Imitation, Trajectory Optimization, Online RL, Few-Shot Real- Offline RL
Attract
VIP Embedding
Positive
"Negatives"
Anchor
Reward
VIP Network
Representation
Initial
Middle
Goal
Figure I: Value-Implicit Pre-training (VIP). Pre-trained on large-scale, in-the-will human videos, frozen VIP network provide visual reward and representation for downstream unseen robotics tasks and enable diverse visuomotor control strategies without any task-specific fine-tuning.
visual imitation learning (IL) from demonstrations (Parisi et al. 2022 Nair et al. 2022). In either case, substantial engineering effort required for learning each new task. Instead, a simple and general way of specifying real-world manipulation tasks by providing goal image (Andrychowicz et al.) 2017] Pathak et al. 2018) that captures the desired visual changes to the environment. However, as we demonstrate in our experiments, existing pre-trained visual representations do not produce effective reward functions in the form of embedding distance to the goal image, despite their effectiveness as pure visual encoders. Given that these models are already some of the most powerful models derived from computer vision, it begs the pertinent question of whether universal visual reward function learned entirely from out-of of-momain data is even possible.
In this paper, show that such general reward model can indeed be derived from pre-trained visual representation, and we acquire this representation by treating representation learning from diverse comman-vided data big offline goal-conditioned reinforcement learning problem. Our idea philosophically diverges from all prior works: instead of taking what worked the best for CV tasks and "hope for the best" in visual control, we propose more principled approach of using reinforcement learning itself as a pre-training mechanism for reinforcement learning. Now, this formulation certainly seems impractical first because human videos do not contain any action information for policy learning. Our key insight is that instead of solving the impossible primal problem of direct policy learning from out-of-domain, action-free videos, we can instead solve the Fenchel dual problem of goal-conditioned value function learning. This dual value function, we will show, can be trained without actions in an entirely self-supprvised manner, making suitable for pre-training on (out-of-domail videos without robot action labels.
Theoretically, we show that this dual objective amounts to a novel form of implicit time contrastive learning, which attracts the representations of the initial and goal frame in the same trajectory, while implicitly repelling the representations of intermediate frames via recursive one-step temporal- difference minimization. These properties enable the representation to capture long- Long-range temporal dependencies over distant task frames and inject local temporal smoothness over neighboring frames, making for smooth embedding distances that show the key ingredient of effective reward function. This contrastive lens, importantly, enables the value function to be implicitly defined as similarity metric in the embedding space, resulting in our simple final algorithm, Value- Implicit Pre-training (VIP); see Fig. [ for an overview.
Trained on the large-scale, in-the-wild Ego4D human video dataset (Grauman et al. 2022) using simple sparse reward, VIP is able to capture a general notion of goal-directed task progress that makes for effective wwward-peccfficatee set of simulated and real-robot tasks, VIP's visual reward significantly outperforms those of prior pre-trained representations on a diverse set of reward-based policy learning paradigms. Coupled with standard trajectory optimizer (Williams et al., 2017). VIP can solve = 30% of the tasks without any task-specific hyperparameter or representation fine-tuning and is the only representation that enables non-trivial progress on the set of more difficult tasks. Given more optimization budget, VIP's performance can further improve up to = 45%, whereas other representations do worse due to reward hacking. When serving both the visual representation and reward function for visual online RL, VIP again significantly outperforms prior methods by a mide-nargin and achieves 40% aggregate success rate. To the best of our knowledge, these are the first demonstrations of a successful
for unseen robot tasks specified via goal images. On an extensive
2

---

for all Published as a conference paper at ICLR 2023
perceptual reward function learned using entirely out-of-domain data. Finally, we demonstrate that few-sfhh few-shot offline RL with as few as 20 trajectories on a diverse suite of real-robot manipulation tasks, demonstrating for the first time that offline RL is possible in this low-data regime and paving the way towards truly scalable and autonomous robot learning.
VIP can enable wealdworld
2 RELATED WORK
We review relevant literature on (1) Out-of-Domain Representation Pre- Fre-Training for Control, (2) Perceptual Reward Learning from Human Videos, and (3) CondiCondt RL as Representation Learning. Due to space constraint, the latter two are included in App.
Out-of-Domaie Representation Tre-Training for Control. Bootstrapping visual control using frozen representations trained on out- of-oomain non-robot data is a nascent field that has seen fast progress over the past year. (Shah & Kumar (202T) demonstrates that pre-tras rre-traind ResNet (He) et al. 2016) representation on ImageNet (Deng et al. (209)) serves as effective visual backbone for simulated dexterous manipulation RL tasks. Parrisi et al. (2022) finds ResNet models trained with
unsupervised objectives, such as momentum contrastive learning (MOCO) (He et al. 2020), to surpass supervised objectives (e.g, image classification) for both visual navigation and control tasks. Xiao et al. (2022) demonstrates that -atkedd-autonncder (He et al. (022)) trained on diverse video data (Goyal al., 2017, Shan et al.] 2020) can be an effective visual embedding for online RL. The closest work to ours is R3M et al. (0022). which is also pre-trained on the Ego4D dataset and attempts to capture temporal information the videos by using time-contrastive learning (Sermanet et al.. 2018); whereas VIP is fully self-supervised R3M additionally requires video textual descriptions to align its representation. These prior works primarily re- re-purpose existing objectives and models for visual control and do not address the reward specification challenge. In contrast, VIP is the first to propose a novel RL-based objective for out-of-domain pre-training and is capable of producing generalizable dense reward signals that enable several new visuomotor control strategies that have not been demonstrated in this setting before.
3 PROBLEM SETTING AND BACKGROUND
In this section, we describe our problem setting of out-of-domain pre-training and provide formalism for downstream representation evaluation. Additional background on condicood reinforcement learning and contrastive learning is included in App.
Out-of-Domain Pre- Fre-Training Visual Representation. We assume access training set of video data D = (o. == (oi...,, %%2) ) where each o E O :=IR x Wx is a raw RGB image; note that this formalism also captures standard image datasets (e.g., ImageNet), if we take h; us Like prior works, we assume D to be out-of-domain and does not include any robot task or domain-specific data. A learning algorithm A ingests this training data and outputs visual encoder = A(D): IR #x x3 + K, where K is the embedding dimension.
Representation Evaluation. Given a choice of representation Φ. every evaluation task can be instantiated as a Markov decision process M := (Ø(O), A, R(or, 0++1: Ø, TT the state space the induced space of observation embeddings, and the task is specified via a (set of) goal image(s) g. Specifically, for a given transition tuple (ot, 0++1) ). we define the reward to be the goal-embedding distance difference (Lee et al. 2021 et al. 2022
7, g), in which
R(o,, 00+1: Φ, (gt)= == Sp(0:+1:9) - Sp(on:)) = (1 (0(+11))) + (780(0441:9) Sa(or:
where S. is a distance function in the direppresenaatinn space; in this work, we set So(orgg) gg) -I]o(o;) - وا))1 This reward function can be interpreted as a raw embedding distance reward with reward shaping (Ng et al. (199)) term that encourages making progress towards the goal. This preserves the optimal policy but enables more efficient and robust policy learning.
Under this formalism, parameters of & are frozen during policy learning (it is considered a part of the MDP), and we want to learn policy π R + A that outputs an action based on the embedded observation ~ (()()). Note that while policies do not explicit condition on a goal as in multi-goal RL (Andrychowicz et al. 2017) setting because there is only one goal to be attempted per task.
evaluation tasks are defined via visual goals. the
3

---

Published as a conference paper at ICLR 2023
4 VALUE-IMPLICIT -RE-TRAINING
In this section, we demonstrate how self-supervised value-function objective can be derived from computing the dual of an offline RL objective on passive human videos (Section 4.Ip. Then, we show how this objective amounts to a novel implicit formulation of temporal contrastive learning (Section 4.2). which helps inducing a temporally smooth embedding favorable for downstream visual reward specification. Finally, we leverage this contrastive interpretation to instantiate simple implementation (<10 lines of PyTorch code) of our dual value objective that does not explicitly learn value network (Section 4.3 culminating in our final algorithm, Value-Inar Implicit Pre- Pre-training (VIP).
4.1 FOUNDATION: SELF-SUPERVISER VALUE LEARNING FROM HUMAN VIDEOS
While human videos are out-of-domain data for robots, they are in-domain for learning goal- conditioned policy TH over human actions, a _ (o(o) ((g)), for some human action space A", Therefore. given that human videos naturally contain goal ddireciee behavior, one reasonable idea of utilizing offline human videos for representation learning is to solve an offline goal-conditioned RL problem over the space of human policies and then extract the learned visual representation. To this end, we consider the following KL-regularized offline RL objective (Nachum et al. 2019) for some to-be-specified reward r)o.g):
Σ A)ooos)) DKL(d (o, (g) (o,a
max EE5h Ti
where d= (o, ; is the distribution over observations and actions тн visits conditioned on Observe that a "dummy" action is added to every transition (o 0j %f+1) in the dataset D so that the KL regularization is well-defined, and at can be thought as the unobserved true human action taken transition from observation of, o While this objective is mathematically sound and encourages learning a conservative =" is seemingly implausible because the offline dataset D does not come with any action labels nor can A be concretely defined in practice. However, what this objective does provide is an elegant dual objective over a value function that does not depend on any action label in the offline dataset. In particular, leveraging the idea of Fenchel duality (Rockafellar 1970) from convex optimization, we have the following result:
Proposition 4.1. Under assumption of deterministic transition dynamics, the dual optimization problem of qquationnis
max, miny Epio) [1] Foo (d(o)) - log Roo (exp(rlo.g) + V(á(o), +(a))))) .3)
where pa (o:g) g) is the noal-conditioned initial observation distribution, and D(o, d': is the goal- conditioned distribution of two consecutive observations in dataset D)
As shown, actions do not appear in the objective. Furthermore, since all expectations in equation 3 can be sampled using the offline dataset, this dual value-function objective can be self-supervised with an appropriate choice of reward function. In particular, since our goal is to acquire value function that extracts a general notion of goal-directed task progress from passive offline human videos, we set r(a,g) = T(o = g) 1, which we refer to as (o) shorthand. This reward provides a constant negative reward when is not the provided goal g. and does not require any task-specific engineering. The resulting value function (d(o); ø(g)) captures the discounted total number of steps required to reach goal g from observation o. Consequently, the overall objective will encourage learning visual features é that are amenable to predicting the discounted temporal distance between two frames in a human video sequence. With enough size and diversity in the training dataset, hypothesize that this value function can generalize to completely unseen (robot) domains and tasks.
4.2 ANALYSIS: IMPLICIT TIME CONTRASTIVE LEARNING
While equation 3 will learn some useful visual representation via temporal value function optimization, in this section, show that it can be understood as novel implicit temporal contrastive learning objective that acquires temporally smooth embedding distance over video sequences, underpinning VIP's efficacy jointly as visual representation and reward for downstream control.
4

---

Published as a conference paper at ICLR 2023
We begin by simplifying the expression in equation 3 by first assuming that the optimal V is found:
ming Bo(a) (1 Battool ("(o(o); o(g))]
log (o exp (5,()) 7V (V((o)))o)) V(o(o). ((6))
.(4
where we have also re-written the maximization problem as a minimization problem. Now, after few algebraic manipulation steps (see App. for derivation), if we think of (V(())): similarity metric in the embedding space, then we can massage equation resembles the InfoNCE (Oord al., (018) time contrastive learning (Sermanet al. 2018 (see App. A.2 for a definition and additional background) objective:
an expression that
(o(o)a(o))
min& - P))
log E Ep(o.ol (o) (40
In particular, p(g) can be thought of the distribution of "anchor" observations, (o(s;)) the distribution of "positive" samples, and D(o, g'g) the distribution of "negative" samples. Counter-intuitivell and in contrast to standard single-view time contrastive learning (TCN), in which the positive observations are temporally closer to the anchor observation than the negatives, equation S has the positives to be as temporally far away as possible, namely the initial frame in the the same video sequence, and the negatives to be middle frames sampled in between. This departure is accompanied by the equally intriguing deviation of the lack of explicit repulsion of the negatives from the anchor; instead, they are simply encouraged to minimize the (exponentiated) one-step temporal-differene error in the representation space (the denominator in equation 5); see Fig. [ Now, since the value function encodes negative discounted temporal distance, due the recursive nature of value temporal- difference (TD), in order for the one-step TD error to be globally minimized along a video sequence, observations that are temporally farther away from the goal will naturally be repelled farther away in the representation space compared to observations that are nearby in time; in App. C.3] we formalize this intuition and show that this repulsion always holds for optimal paths. Therefore, the repulsion of the negative observations is an implicit, emergent property from the optimization of equation 5 instead of an explicit constraint as in standard (time) contrastive learning.
Now, we dive into why this implicil time con- trastive learning is desirable. First, the explicit attraction of the initial and goal frames enables capturing long-rang semantic temporal depen- dency as two frames that meaningfully indicate the beginning and end of a task are made close in the embedding space. This closeness is also well-defined due to the one-step TD backup that makes every embedding distance recursively de- fined to be the discounted number of timesteps to the goal frame. Combined with the implicit yet structured repulsion of intermediate frames, this push-and-pe and- mechanism helps inducing temporally smooth and consistent representation.
Task
VIP (In-Domain)
TCN (In-Domain)
Figure 2: Learned 2D representation of a held-out task demonstration by VIP and TCN trained on task-specific in-domain data. The color gradient indicates trajectory time progression (purple for beginning, red for end). The inset plots are embedding distances to the last frame.
In particular, as we pass a video sequence in the training set through the trained representation, the embedding should be structured such that two trends emerge: (1) neighboring frames are close-by in the embedding space, (2) their distances to the last (goal) frame smoothly decrease due to the recursively defined embedding distances. To validate this intuition, in Fig, we provide simple toy example comparing implicit vs. standard time contrastive learning when trained on in-domain, task-specific demonstrations; details are included in App. F.2 As shown, standard time contrastive learning only enforces a coarse notion of temporal consistency and learns a non-locally smooth representation that exhibits many local minima. In contrast, VIP learns a much better structured embedding that is indeed temporally consistent and locally smooth.
4.3 ALGORITHM: VALUE-IMPLICIT PRE- TRE-TRAINING (VIP)
Recall that V is assumed to be known for the derivation in Section 4.2 but in practice, its analytical form is rarely known. Now, given that V plays the role distance measure in our implicit time contrastive learning framework, simple and practical way to approximate V is to set it to be
5

---

Published as a conference paper at ICLR 2023
choice of similarity metric, bypassing having to explicitly parameterize it neural network. In this work, we choose the common choice of the negative L2 distance used in prior work Sermanet et al. (2018); Nair et al. ) V (Ø(o), ((g)) == - l 2(ا)و12- Given this choice, our final
representation learning objective is as follows:
146) Fpto) (1 - putte)) [l - d(g\لaal + log Beoa exp (ll66o) - +[g)] [,) - (lo(dd) - )is) (6)
in which we also absorb the exponent of the log-sum-exp term in 4 into the inner exp(.) term via Jensen's inequality; found this upper bound to be numerically more stable. To sample video trajectories from D, because any sub-trajectory of a video is also a valid video sequence, VIP samples these sub-trajectories and treats their initial and last frames samples from the goal and initial-state distributions (Step 3 in Alg. I]. Altogether, VIP training is illustrated in Alg. [T] it is simple and its core training loop can be implemented in fewer than 10 lines of PyTorch code (Alg. 2 in App. D.3)
Algorithm Value-Implicit Pre- Traininnn (VIP)
1: Require: Offline (human) videos D) = ((o) ., oi ). visual architecture
2: for number of training iterations do
3: Sample sub-trajectorrie (o',..., o 0 +1 of h
D2
4: C(Ø) = 1- E [l(o(e)) =(ob))ll]] + log 1/2 Enn
exp (I(00E) -o(0))ll - (oi) ( -1) - (lob)
5: Update ø using SGD: á + as VL(
5 EXPERIMENTS
In this section, demonstrate VIP's effectiveness both a pre-trained visual reward and representa- tion on three distinct reward-basee based policy learning settings. We begin by detailing VIP's training and introducing baselines. Then, we present the full evaluation results for each setting, and we conclude with qualititave analysis, delving into VIP's unique effectivvness.
VIP Training. We use standard ResNet50 (He et al., 2016) architecture VIP's visual backbone and train on a subset of the Ego4D dataset (Grauman et al. (022)) a large-scale egocentric video dataset consisting of humans accomplishing diverse tasks around the world. These choices are identical to a prior work (Nar et al. 2022);; additionally, we use the exact same hyperparameters (e.g..
batch size, optimizer, learning rate) as Nair et al. (2022). See App. Dooo details.
Baselines. The closest comparison is R3M (Nair al., 2022), which pre-trains on the same Ego4D dataset using combination of time contrastive learning, L, weight regularization, and language embedding consistency losses. We also consider a self-supervised ResNet50 net- work trained ImageNet using Momentum Contrastive (MoCo). a supervised ResNet50 network trained on ImageNet, and CLIP Rad- ford et al. (202T), covering a wide range of pre- existing visual representations that have been
Figure 3: Frankakitchen example goal images.
Cui et al. 2022), though none has been tested in our three reward-based settings in which the reward also has be produced by the representation. Note that the visual backbone for all methods is ResNet50, enabling fair comparison of the pre-training objectives. Besides VIP, all models are taken from their publicly released checkpoints. In Appendix G.T we additionally compare to MoCo and Masked Auto-Encoder (MAF) models trained on Ego4D)
used for robotics control (Shah & Kumar 2021 Parisi et al. 2022:
Evaluation Environments For our simulation experiments, we consider the FrankaKitchen (Gupta ct al. 2019) environment, in which 7-DoF Franka robot is tasked with manipulating common household kitchen objects to pre-specified configurations. We use all 12 subtasks supported in the environment and 3 camera views (left, center, right) for each task, in total of 36 visual manipulation tasks. Furthermore, consider two initial robot state for every task, one Easy setting in which the end-effector is initialized close to the object of interest, and one Hard setting in which the end-effector is uniformly initialized above the stove edge regardless of the task. The task horizon is 50 (resp. 100)

---

Published as a conference paper at ICLR 2023
Trajectory Optimization (MPPI)
Online RL (NPG)
30-
Easy
10-
0-
100000
503020
12-
10-
Harc
10
5--
100
0.0
C4
0.6
0.8
1.0
MPPI Steps
Env Steps
led
Figure 4: Visual trajectory optimization and online RL aggregate results (cumulative success rate %).
for the Easy (resp. Hard) setting. Each task is specified via goal image; see Fig. 3 for example goal images for all three views, and see Fig. 9 App. E. for initial and goal frames for all tasks).
5.1 TRAJECTORY OPTIMIZATION & ONLINE REINFORCEMENT LEARNING
We evaluate pre-trained representations' capability pure visual reward functions by using them to directly synthesize a sequence of actions using trajectory optimization particular, we use model- predictive path-integral (MPPI) (Williams et al., 2017). To evaluate each proposed sequence of actions, we directly roll out in the simulator for simplicity. Vanilla trajectory optimization, though sample efficient, is prone to local minima in the reward landscape due a lack of trial-and-erron exploration. We hypothesize that online RL may be able to overcome bad local minima, but comes with the added challenge of demanding the pre-trained representation to provide both the visual reward and representation for learning a closed-loop policy. To further study the importance of a learned dense reward in online RL, compare using ground-trui task sparse reward coupled with VIP's visual representation, VIP (Sparse). The RL algorithm we use is natural policy gradient (NPG) (Kakade 200T). We leave all experiment details are in App. E.4 In Fig. we report each representation 's cumulative success rate averaged over all configurations (3 seeds cameras 12 tasks 108 runs); the success rate for online RL is computed over a separate of test rollouts.
Examining the MPPI results, we see that VIP is substan- tially better than all baselines in both Easy and Hard set- tings, and is the only representation that makes non-trivial progress on the Hard setting. In Fig. we couple VIP and the strongest baselines (R3M. Resnet) with increasingly more powerful MPPI optimizers (i.e., more trajectories per optimization step; default 32 are used in Fig. As shown,
MPPI Optimizer Scaling
Success Rate
Ranviet
0.45
0.45
0.28
A12
while VIP stcadily benefits from stronger optimizers and reach average success rate of 44%, baselines often do worse when MPPI is given more compute budget, sug- gesting that their reward landscapes are filled with local minima that do not correlate with task progress and are easily exploited by stronger optimizers. To further validate these observations, in App. E.3 we report the L2 error to the ground-trutt goal-image robot and object poses after each environment step taken by MPPI. We find that VIP is able to minimize both the robot and object pose errors quite robustly over diverse tasks and views, whereas several baselines in fact increase the robot pose error on average. Given these findings, we hypothesize that VIP's s reward functions are able capture task-salient information in the visual observations. In App. G.4 we validate this hypothesis and find that on at least one camera view for 8 out of the 12 tasks, VIP's rewards are highly correlated with the human-engineered based dense rewards, with correlation coefficients as high as R² = 0.95, highlighting its potential of replacing manual reward engineering without any prior knowledge about the robot domain or tasks.
615
D.00
000
20
254
Trajector es per MPPI Step
Figure VIP benefits from compute scaling downstream control.
Switching gears to online RL, VIP again achieves consistently superior performance. VIP (Sparse)'s inability solve any task, despite strong visual representation provided by VIP itself, indicates
7

---

Published as a conference paper at ICLR 2023
Table I: Real-robot offline RL results (success rate % averaged over 10 rollouts with standard deviation reported).
Pre-Trained VIP-BC
In-Domain VIP-RWR
Environment
VIP-RWR
R3M-RWR
R3M-BC
Scratch-BC
VIP-BC
CloseDrawer
100 0
50 - 50
80 + 40
10 + 30
30 = 46
0
0" =
PushBottle
90 f 30
50 - 50
70 i 46
50 + 50
40d 48
0" =
0" i
PlaceMelon
60 + 18
10 - 30
0 -
0 0:: 0
0 0
0" =
0" =
FoldTowel
90 + 30
20 + 10
0 -
0 0
0
0* =
0" -
the necessity of dense reward in solving these challenging visual manipulation tasks and further accentuates VIP's versatility doubling as both visual reward and representation; furthermore, we find that sparse reward coupled with even the true state representation unable to make any progress. Finally, comment that whereas sparse reward still requires human engineering via installing additional sensors (Rajeswar et al.] [022] Singh et al. (2019) and faces exploration challenges (Nair al. 2018), with VIP, the end-user has provide only goal image.
5.2
REAL WORLD FEW-SHOT OFFLINE REINFORCEMENT LEARNING
In this section, we demonstrate how VIP's reward and representation can power a simple and practical system for real-world robot learning in the form of few-shof offline reinforcement learning, making offline RL simple, sample-efficient, and more effective than BC with almost no added complexity.
To this end. we consider simple swwar-weighted regression (RWR) (Peters & Schaal 2007 Peng ct al. 2019) approach, in which the reward and the encoder are provided by the pre-trained model ø
[(x)= -E Li [exp(r R(o, o'; o, g)) log T(a @(o))],
(7)
where R is defined via equation D annd= T is the temperature scale. Compared to BC, which would be equation 7 with uniform weights all transitions, RWR can focus policy learning on transitions that have high rewards (i.c., high task progress) under the deployed representation. Consequently, if
the chosen representation is predictive of task progress (i.c., assigning high rewards to key transitions for the task), then RWR should be able to outperform BC, especially on tasks that naturally admit key transitions (e.g.. picking up the towel edge our Fold PoldTowel task). This intuition is theoretically proven (Kumar al. 2022) and holds even if the offline data consists of solely expert demonstrations, though not validated on real-world tasks.
We introduce 4 tabletop manipulation tasks (see Fig. D and Fig. 13) requiring a real 7-DOF Franka robot to manipulate objects drawn from distinct categories of objects. For each task, we collect in-domain, task-specific offline data Dtask of ~ 20 demonstrations with randomized object initial placements for policy learning; we provide detailed task and experiment descriptions in App. FF) We compare VIP to R3M, the only other representation that has been deployed on real robots in the visual imitation setting, and instantiate necessary for low-data regime offline RL, we also train in-domain VIP- VIP-[RWR, (RWRRBCR from scratch using only Dtask, where the VIP representation is learned using Deask first and then frozen during the policy training via RWR/BC. In addition, we include Scratch-BC. for which the regression loss is used to learn the policy and the representation in completely end-to-end manner.
To assess whether pre-training is
We train all policies using the same set of hyperparam- eters used for real-world BC training in Nair et al. (2022). and evaluate each method 10 rollouts, cov- ering the distribution of ob- ject's initial placement in the dataset. The average success rate (%) and stan- dard deviation across roll- outs are reported in T As shown, VIP-RWR im- proves upon VIP-BC on all tasks and provides substantial benefit on the harder tasks that are multi-stage in nature. In contrast, R3M-RWR, while able to improve R3M-BC on the simpler two tasks involving pushing an object, fails to make any progress on the harder tasks. In-domain VIP. based methods fail completely and their actions are almost always pre-empted by the hardware safety
Rewerd Curves Compatison
Reward Ourves Comparison
K.91
Normalteed Dictunce
Normelized Distance
McCo
3.09
am
azc
Frome
Frame
Figure 6: Emhedding distance curves on Ego4D (T.) and real-robot (R) videos.
8

---

Published as a conference paper at ICLR 2023
check to prevent robot damage (indicated by +), suggesting significant overfitting in training the VIP representation using just the scarce task-specific data Diack. The low performance of BC-based methods on the harder PickP MacceeaaaMMeaa lon and FoldTowel tasks indicates that in this low-data regime, regardless of what visual representation the policy uses, good reward information is necessary for task success. Altogether, these results corroborate the necessity of pre-training in achieving few-shot offline RL in the real-worid and highlight the unique effectiveness of VIP in realizing this goal. Qualitatively, we find VIP-RWR policies acquire intelligent behavior such as robust key action execution and task re-attempt; see App. F.4 and our supplementary video for analysis.
5.3 QUALITATIVE ANALYSIS
We hypothesize that VIP learns the most temporally smooth embedding that enables effective zero-shot reward-specifice specifiicpciiiiti and present several qualitative experiments investigating this claim. First, in Fig. 6 we visually overlay and compare the embedding distance-to to-ooe- curves for each representation on representative videos from both Ego4D and our real-robot dataset; the curve for every representation is normalized have initial distance to enable comparison. As shown, VIP has the most visually smooth curves, whereas all other methods exhibit "bumps" (i.e., positive slope at a step) that signal more prevalent presence of local minima in their reward landscapes; in App. G.5 we provide additional embedding curves. Finally, we quantify the total number of "bumps" each representation encounters over both datasets in App. G.6 and find VIP indeed has much fewer bumps.
In addition to the number of bumps, we posit that the magnitude of bumps also a key distinguish- ing factor among represen- tations. Because the reward (equation I is the nega- tive embedding distance-to- goal difference, positive reward at a step is equiva- lent to a negative slope
Embedding
Embedding Difference
1750
1500-
(arge) negative rewardd
small positive reward -smooth reward
Differente rifference Ratio
Count
bummp reward curves
1060
curves
positive reware
Amakiing negative rewcards ch
230-
-44
-03
-02
-b1 Embedding Reward
0.0
62
83
-0.4
-03
-02
-0.1
00
611
02
0.3
Embeddlig Reward
Figure 7: Embedding reward histogram our real-robot dataset.
that step in the corresponding embedding distance curve. The ideal representation should have a reward histogram that has tall peak in the first positive bin, indicating that its embedding distance curves consist of mostly small, negative slopes that make for smooth curves, In Fig. 7 we compare VIP and R3M representations (comparisons to other baselines are in App. G.7) by overlaying their respective normalized embedding reward histogram computed over the entirety of the real-robot dataset (we include the histograms computed over Ego4D in App. G.8). In addition, we create a bar-plot over the count-difference ratio for each bin, VIP|-|RaM] We see that VIP has much higher [R3M]
count in the first positive-reward bin 25%+ more than R3M), fewer negative rewards overall, and much fewer extreme rewards (s-100% to R3M) in either direction, indicating that on aggregate VIP's reward landscape is much smoother than that of R3M, and this trend holds against all other baselines. These findings confirm our hypothesis that VIP learns the most temporally smooth representation. Surprisingly, despite trained on Ego4D, VIP's smoothness property transfers the robot domain, suggesting that VIP representation indeed has learned generalizable features that are predictive of goal-directed task progress, and it is this generalization that lies at the heart of VIP's ability to perform zero-shot reward-specification.
6 CONCLUSION
We have proposed Value-Implicit Pre-training (VIP), self-supervised value-based pre-training objective that is highly effective in providing both the visual reward and representation for downstream unseen robotics tasks. VIP derived from first principles of dual reinforcement learning and admits an appealing connection to an implicit and more powerful formulation of time contrastive learning, which captures long-range temporal dependency and injects local temporal smoothness in the representation to make for effective zero-shot reward specification. Trained entirely on diverse, in-the-wild human videos, VIP demonstrates significant gains over prior statc-of-art pre-trained visual representations on an extensive set of policy learning settings. Notably, VIP can enable sample-efficien real-world offline RL with just handful of trajectories. Altogether, we believe that VIP makes an important contribution in both the algorithmic frontier of visual pre-training for RL and practical real-world robot learning.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work was done while YJM was an intern at Meta AL We thank Aravind Rajeswaran and members of Meta AI for helpful discussions, Jay Vakil for assistance the real-robot experiments, and Vincent Moens for assistance on releasing the model.
REPRODUCIBILITY STATEMENT
We have open-sourced code for using our pre-trained VIP model and training a new VIP model using any custom video dataset https::/gitthi: com/ www.//obbbeeeerrcc//a.p.... the instruction for model training and inference is included in the README, md file in the supplementary file, and the hyperparameters are already configured. A PyTorch-based pseudocode for VIP is also included in Alg. 2
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Alckh Agarwal, Nan Jiang, and Sham M Kakade. Reinforcement learning: Theory and algorithms. 2019.
Anish Agarwal, Abdullah Alomar, Varkey Alumootil, Devavrat Shah, Dennis Shen, Zhi Xu, and Cindy Yang. Persim: Data-efficient offline reinforcement learning with heterogeneous agents via personalized simulators. Advances in Neural Information Processing Systems, 34:1 18644-8566 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild, arXiv preprint arXiv:2207.0944 2022.
Steven J Bradtke and Andrew Barto. Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1):33-57, 1996.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07499 2021.
Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2 16817, 2021.
Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, pp. 893-905. PMLR, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 1 13396, 2021.
Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. arXiv preprint
2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyúska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller- Freitar, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long- morizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: /910. 1956, 2019.
11

---

Published as a conference paper at ICLR 2023
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 16000-16009, 2022.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. arXiv preprini 1rXiv:2109. 10813, 2021.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618, 2022.
Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph J Lim. Generalizable imitation learning from observation via inferring goal proximity, A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL hhttps://openreviee net forum? 1pd.ddd--
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.0/643, 2020.
Yunfei Li, Tian Gao, Jiaqi Yang. Huazhe Xu, and Yi Wu. Phasic self-imitative reduction for sparse- reward goal-conditioned reinforcement learning. In International Conference on Machine Learning, pp. 12765-12781. PMLR, 2022.
Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint urXiv:2202.02433, 2022a.
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via f-advantage regression. arXiv preprint arXiv:2206.03022 2022b.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung. Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pp. 879 879- 893. PMLR, 2018.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108. 2021.
Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellaar duality, 2020.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint /912. 020044 2019.
12

---

Published as a conference paper at ICLR 2023
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6292-6299. IEEE, 2018.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprini arXiv:2203. /2601, 2022.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application to reward shaping. In Icml, volume 99, pp. 278-287, 1999.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 18X.:01807.03748, 2018.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness pre-trained vision models for control. arXiv preprint 2335:22003033580, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-pens sesh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 2050-2053, 2018
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 4rXiv:1910.00117, 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by rewird-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745-750, 2007.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Sai Rajeswar, Cyril Ibrahim, Nitin Surya, Florian Golemo, David Vazquez, Aaron Courville, and Pedro O. Pinhciro. Haptics-based curiosity for sparse-reward tasks. In 5th Annual Conference on Robot Learning, 2021. URL https: http:ht://ppeeeeiieeeeee/ / forum? i 1 VVGGOOELLLLLL
net
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint /r0iiv/709,/// 10087, 2017.
R. Tyrrell Rockafellar. Convex analysis. Princeton Mathematical Series. Princeton University Press, Princeton, N. J., 1970.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis. Sergey Levine, and Chelsea Finn. Reinforce- ment learning with videos: Combining offline observations with interaction. arXiv preprint arXiv:20/ 1.06507, 2020.
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint arXiv: /61i. 1612,0669, 2016.
13

---

Published as a conference paper at ICLR 2023
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. cone-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning, arXiv preprint arXiv:2107.03380, 2021.
Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2020.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07854 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Tongzhou Wang and Phillip Isola. On the learning and learnablity of quasimetrics. arXiv preprini arXiv:2206. 15478, 2022.
Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2): 344-357, 2017.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.061 06211033..63... 2022.
Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj. Samarth Sinha, and Animesh Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 7827-7834.
IEEE, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta worldw A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data offline reinforcement learning. arXiv preprint arXiv:220.001741,
2022.
Kevin Zakka, Andy Zeng. Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pp. 537-546. PMLR, 2022.
Siyuan Zhang and Nan Jiang. Towards hyperparameter--fre policy selection for offline reinforcement learning. Advances in Neural Information Processing Systems, 14:12864-12875 2021.
14

---

Published as a conference paper at ICLR 2023
Part I
Appendix
Table of Contents
Additional Background
15
A.1 Goal-Conditioned Reinforcement Learning
15
A.2 InfoNCE & Time Contrastive Learning,
16
B Extended Related Work
17
Technical Derivations and Proofs
17
Proof Proposition 4.1
VIP Implicit Time Contrast Learning Derivation
................................
18
VIP Implicit Repulston
18
D
VIP Training Details
19
D.1
Dataset Processing and Sampling
19
D.2
VIP Hyperparameters
19
D.3
VIP Pytorch Pscudocode
20
E
Simulation Experiment Details.
20
F.1
FrankaKitchen Task Descriptions
20
E.2
In Domain Representation Probing
20
E.3
Trajectory Optimization)
23
E.4
Reinforcement Learning
23
F
Real- World Robot Experiment Details
23
F
Task Descriptions
23
F2
Training and Evaluation Details
................................
24
F3
Additional Analysis & Context
................................
25
F4
Qualitative Analysis
25
G
Additional Results
26
G.
Comparison to MAE and MoCo trained on Ego4D
26
G.2
Value-Based Pre- Training Ablation: Square Temporal- Dinaal--iifeeee
26
G.3
Visual Imitation Learning]
27
G.4
Embedding True Rewards Correlation
28
G.5
Embedding Distance Curves
28
G.6
Embedding Distance Curve Bumps
28
G.7 7
Embedding Reward Histograms ReealRRoool Dataset
32
G.8
Embedding Reward Histograms (Ego4))
32
Amiitaions and Future Work
32
A ADDITIONAL BACKGROUND
A.1 GOAL-CONDITIONED REINFORCEMENT LEARNING
This section is adapted from Ma et al. (022b). We consider goal-conditioned Markov decision process from visual state space: M (0, A. G, r, T, no, 7) with state space O, action space A. reward r(o, g). transition function of T(o, a). the goal distribution p(g), and the goal-conditioned initial state distribution uo g), and discount factor 7 E (0, 1). We assume the state space and
15

---

Published as a conference paper at ICLR 2023
the goal space G to be defined over RGB images. The objective of goal-condittoned RL is to find goal-conditioned policy т : O x (A) that maximizes the discounted cumulative return:
((T) (eg
Σ >'r(org))
(8)
The goul-conditioned state-action occupancy distribution d' (50) Ax of
dh (0, == Prrro o, = a ou volo;g), ( 0+11 T(or,
t=U
which captures the goal-conditioned visitation frequency of state-action pairs for policy n. The
state-occupancy distribution then marginalizes over actions: d (o;g) = Σ. d" (o, 0; g).. Then, follows that (a o, 4" (a.0o))
state-action occupancy distribution must satisfy the Bellman flow constraint in order for it to an occupancy distribution for some stationary policy w:
2= (a:g)
Σ d(o, (g: = (1 7)mo((:)) + > T(s ā, 6,a
0,
(10)
We write (o, = (og)d as the joint goal-state density induced by p(g) and the policy Finally, given d", express the objective function equation as J(T) 1-y (000
A.2 INFONCE & TIME CONTRASTIVE LEARNING.
As VIP can be understood as a implicit and smooth time contrastive learning objective, we provide ad- ditional background on the InfoNCE Oord et al. 2018) and time contrastive learning (TCN) (Sermanet
et al. 2018) objective to aid comparison in Section
InfoNCE an unsupervised contrastive learning objective built on the noise contrastive estima- tion (Gutmann & Hyvärinen, 2010) principle. In particular, given an "anchor" datum x (otherwise known as context), and distribution of positives Tpas and negatives Fneg, the InfoNCE objective optimizes
SØ Tpps)
min E Exp
log
(11)
Ex,no So(I, Tneg)
where Err,, is often approximated with a fixed number of negatives in practice.
It is shown in Oord et al. (2018) that optimizing equation 11 is maximizing lower bound on the mutual information [xx Tp where, with slight abuse of notation, x and Fpos are interpreted as random variables.
TCN a contrastive learning objective that learns a representation that timeseries data (e.g., video trajectories). The original work (Sermanet er al. 2018) considers multi-view videos and perform contrastive learning over frames in separate videos; in this work, we consider the single-view variant. At a high level, TCN attracts representations of frames that are temporally close, while pushing apart those of frames that are farther apart in time. More precisely, given three frames sampled from a video sequence (ot, ,tg, Otg), where t < and or,, and repel the representation of Otg from or This idea can be formally expressed via the following objective:
ts, TCN would attract the representations of on
[|o(o,)) (oooo,)od00 #(012)ll2
min E
(oy Pigo Jw
- log Eoy D jexp (- [(o(or,) (122
(12)
Given "positive" window of K steps and uniform distribution among valid positive samples, can write equation 12 as
Soloe :4) Bor, bt (Sp(on) 0tg]
min
Bio, or +
log
(13)
in which each term inside the expectation a standalone InfoNCE objective tailored to observation sequence data.
16

---

Published as a conference paper at ICLR 2023
B EXTENDED RELATED WORK
Perceptual Reward Learning from Human Videos. Human videos provide a rich natural source of reward and representation learning for robotic learning. Most prior works exploit the idea of learning an invariant representation between human and robot domains to transfer the demonstrated
skills (Sermanet et al.) 2016 2018 Schmeckpeper al. 2020: Chen et al. 2021 Xiong et al., 202T] Zakka et al. 2022: Bahl et al. 2022). However, training these representations require task-specifte
human demonstration videos paired with robot videos solving the same task, and cannot leverage the large amount of "in-the-wild" human videos readily available. As such, these methods require robot data for training, and learn rewards that are task-specific and do not generalize beyond the tasks they trained on. In contrast, VIP do not make any assumption the quality the task-specificity of human videos and instead pre-trains an (implicit) value function that aims to capture task- agnostic goal-oriented progress, which can generalize to completely unseen robot domains and tasks.
Coll-onnitiooeed RL as Representation Learning, Our pre-training method is also related to the idea of treating goal-conditioned RL representation learning. Chebotar et al. (202T) shows that goal-conditioned Q-function trained with offline in-domain multi-task robot data learns an useful visual representation that can accelerate learning for downstream task in the same domain. Eysenbach et al. (2022) shows that goal-conditioned Q-learning with a particular choice of reward function can be understood performing contrastive learning. In contrast, our theory introduces a new implicit time contrastive learning, and states that for any choice of reward function, the dual formulation of regularized offline GCRL objective can be cast as implicit time contrast. This conceptual bridge also explains why VIP's learned embedding distance is temporally smooth and can be used as an universal reward mechanism. Finally, whereas these two works are limited to training on in-domain data with robot action labels, VIP is able to leverage diverse out-of -f--o---mmmin human data for visual representation pre-training overcoming the inherent limitation robot data scarcity for in-domain training.
Our work is also closely related to Ma et al. (2022b), which first introduced the dual offline GCRL objective based Fenchel duality (Rockafellar 1970; Nachum & Dail 2020: Ma et al. 2022a). Whereas Ma et al. (2022b) assumes access to the true state information and focuses on the offline GCRL setting using in-domain offline data with robot action labels, we extend the dual objective to enable out-of-domain, action-free pre-training from human videos. Our particular dual objective also admits novel implicit time contrastive learning interpretation, which simplifies VIP's practical implementation by letting the value function be implicitly defined instead of deep neural network as in Ma al. (2022b).
C TECHNICAL DERIVATIONS AND PROOFS
C.I PROOF OF PROPOSITION
We first reproduce Proposition 4.1 for ease of reference:
Proposition C.1. Under assumption of deterministic transition dynamics, the dual optimization problem of
max E
Σ xr((:
(o.
(14)
is
max miny Eyta) [11 -1))))BBm (())] + log Epi (exp (r(o.)) + (V((0 - (()())
(15)
where pa (n(o: g) is the noal-conditioned initial observation distribution, and D(o, g) is the goal- conditioned distribution of two consecutive observations in dataset D.
17

---

Published as a conference paper at ICLR 2023
Proof. We begin by rewriting equation 14 as optimization problem valid occte-ocyupancy distributions. To this end, we have"]
max max Batotoroa [r(o;g)] (d)
d(sto).a:0lo))2
(P) s.t. Σ d( (1 ())oo))))))))) + Σ T( 6,
\ a,j) (g)) Vo E O,g G
(16)
Fixing a choice of Ø, the inner optimization problem operates over a o-induced state and goal space, giving equation T6 Then, applying Proposition 4.2 of Ma et al. (2022b) to the inner optimization problem, we immediately obtain
max minlE [[L 7) Hotes)) [V (V((()):
(D)
+ log Ea (d(o) (exp(r(0,9) (oxp))))_eeoo)))_p(pppp0o0 + BBT a.) -V(oo)) (17)
Now, given assumption that the transition dynamics is deterministie, replace the inner expectation Er(oyla,)) with just the observed sample the offline dataset and obtain:
max min - 7) mator))
& V (1-
( (V((();:
(18)
+ +log (dp( (( + v(d())
Finally, sampling embedded states from (é(0)) ((o); o(g)) is equivalent to sampling from
D(o, o': g). assuming there no embedding collision (i.e., Φ(o) + Φ(d'), Vo * o'), which can
be satisfied by simply augmenting any by concatenating the input to the end. Then, have desired expression:
(o())) (())) max, miny Eoii Botowy) [V((()): ((9)) log Beloor (exp(r(o. + V
(19)
C.2 VIP IMPLICIT TIME CONTRAST LEARNING DERIVATION
This section provides all intermediate steps to go from equation 4 to equation First, have
ming Exca) (a ( + log Eptoo')) (exp -
(20)
We equivalently write this objective
ming Exto) (1 Eepptenall V (M(oholll)) log Epraoo))))) +40
(21)
Then,
min Eprat (1 (1-3) Epulese)
log Vo (o(olso(a)) Epicors) exp +Vooo)+(( ((o(o).
min[1 P)) (mng)
log
Epiooto)
exp - V((())
(22)
This is equation i the main text.
C.3 VIP IMPLICIT REPULSION
In this section, we formalize the implicit repulsion property of VIP objective (equation in particular, we prove that under certain assumptions, always holds for optimal paths.
We omit the human action superseript H in this derivation.
18

---

Published as a conference paper at ICLR 2023
Proposition C.2. Suppose (*(gg) - ( واg)و12 for some 6. under the assumption of deterministic dynamics (as in Proposition (4. for any pair of consecutive states reached by the optimal policy, ($t, S1+1) TT, we have that
[[o(s)) - ا(ا)112 > (| 2
(23)
Proof. First, note that
(*(sg) = max Q" (s as)a;g)
(24)
proof can be found in Section 1.1.3 of Agarwal et al. (2019). Then, due to the Bellman optimality equation, we have that
Q" a: g) - r(s.g) + 7Bx max Q" (5', g)
(25)
Given that the dynamics deterministic and equation 24 we have that (* (5 g) = r(s,g) g) + 7 (s':g)
(26)
Now, for (st, Oe, 8141) n², this further simplifies to
V (s:; g) = r(st. g) + "V'
(27)
Note that since V is also the optimal value function, given that r(st, I(s; - -,, is the negative discounted distance of the shortest path between s; ans g. In particular, since (gig
O by construction, we have that V* (s(s;; g) = Σx= 7 (this also clearly satisfies equation 27
where the shortest path (i.e., the path л takes) between 8; and g are K steps long. Now, giving that
we assume V* ('((s)))) g) can be expressed as - [10(s)) - o())112 for some o, it immediately follows that
(lo(s)) ا(g)] (((($(41) o(g))l2 ((s,, ~ RT"
(28)
The implication of this result is that at least along the trajectories generated by the optimal policy, the representation will have monotonically decreasing and well-behaved embedding distances to the goal. Now, since in practice, VIP trained on goal-directed (human video) trajectories, which are near- optimal for goal-reaching, we expect this smoothness result be informative about VIP's embedding practical behavior and help formalize out intuition about the mechanism of implicit time contrastive learning. As confirmed by our qualitative study in Section 5.3 We highlight that VIP's embedding is indeed much smoother than other baselines along test trajectories on both Ego4D and on our real-robot dataset. This smoothness along optimal paths makes it easier for the downstream control optimizer to discover these paths, conferring VIP representation effective zero-shot reward-specifieee capability that is not attained by any other comparison.
D VIP TRAINING DETAILS
D.1 DATASET PROCESSING AND SAMPLING
We use the exact same pre-processed Ego4D dataset as in R3M, in which long raw videos are first processed into shorter videos consisting of 10-150 frames each. In total, there are approximately 72000 clips and 4.3 million frames in the dataset. Within a sampled batch, we first sample a set of videos, and then sample a sub-trajectory from each video (Step in Algorithm In this formulation, each sub-trajecte uub-trajectory is treated as a video segment from the algorithm's 's perspective; this can viewed as a variant of trajectory data augmentation. As in R3M, we apply random crop at a video level within batch, so all frames from the same video sub-trajectory are cropped the same way. Then, each raw observation is resized and center-cropped to have shape 224 x 224 x 3 before passed into the visual encoder. Finally, as in standard contrastive learning and R3M, for each sampled sub tub-trajcctory {o),... or: o.411 ..., }. we also sample additional 3 negative samples (ō,, ō5+1) from separate video sequences to be included in the log-sum-exp term in L()).
D.2 VIP HYPERPARAMETER
Hyperparameters used can be found in Table
19

---

Published as a conference paper at ICLR 2023
Table 2: VIP Architecture & Hyperparameters..
Name
Value
Architecture
Visual Backhone
ResNet50 (He al. 2016)
FC Layer Output Dim
1024
Hyperparameters
Optimizer
Adam Kingma & Ba 2014
Learning rate
0.0001
Li weight penalty
0.001
L1 weight penalty
0.001
Mini-batch size
32
Discount factor
0.98
D.3 VIP PYTORCH PSEUDOCODE
In this section, we present pseudocode of VIP written in PyTorch (Paszke 2019). Algorithm As shown, the main training loop can be as short as 10 lines of code.
Algorithm 2 VIP PyTorch Pseudocode
offline dataset phi: vision architecture
training loop
for (o_0, o_tl, o_t2, g) in D:
phi_g phi (o_g)
norm (phi (o 0), phi g) v V_t1 = torch.linalg, hww.::inngg...mmpp......
v 0
norm (phi (o_t1), phi_g)
v2 torch. torch.1inalg.. www.liinng...mmmm....mmmm..
norm (phi (o_t2) phi g)
VIP loss (1-gamma) +-V mean torch. logssuepp (V (V_t++1-gama=** t2)
optimizer (eerzzero_gra((
VIP loss backward optimizer step
E SIMULATION EXPERIMENT DETAILS.
E. FRANKAKITCHER TASK DESCRIPTIONS
In this section, we describe the FrankaKitchen suite for our simulation experiments. We use 12 tasks from the v0. versior of the environment.
We use the environment default initial state as the initial state and frame for all tasks in the Hard setting. In the Easy setting. we use the 20th frame demonstration trajectory and its corresponding environment state as the initial frame and state. The goal frame for both settings is chosen to be the last frame of the same demonstration trajectory. The initial frames and goal frame for 3 camera views are illustrated in Figure
tasks and
In the Easy setting, the horizon for all tasks is 50 steps; in the Hard setting, the horizon is 100 steps. Note that using the 20th frame as the initial state is crude way for initializing the robot, and for some tasks, this initialization makes the task substantially easier, whereas for others, the task still considerably difficult. Furthermore, some tasks become naturally more difficult depending our experiment's 's emphasis on the aggregate behavior of pre pre-trained representations, instead of trying to solve any particular task well possible.
camera viewpoints. For these reasons, it is worth noting that
E.2 IN-DOMAIN REPRESENTATION PROBING
In this section, we describe the experiment we performed to generate the in-domain VIP vs. TCN comparison in Figure We fit VIP and TCN representations using 100 demonstrations from the
20

---

Published as a conference paper at ICLR 2023
(a) Idoor_close (left)
(b) 1door_close (center)
(c) 1door_closd (right)
(d) 1door open (Left)
(e) Idoor open (center)
(l) idoor open (right)
g rdoor clsse (1eft)
adoor_close (center)
(i) rdoor rdoor_cl cc
(right)
i rdoor_epen (left)
(k) zdoor_open (conter)
()) rdoor_ open (right)
(m) ccooe (left)
sdoor_close (center)
(o) adoor_close
(right)
(p) sdoor open _
(left)
(q) sdcor open (center)
(r) sdoor open (right)
Figure 8: Initial frame (Easy), initial frame (Hard), and goal frame for all 12 tasks and camera views FrankaKitchen suite.
21

---

Published as a conference paper at ICLR 2023
(a) close (left)
(b) micro_closs (center)
(c) micro_close (right)
(d) micro open (Left)
(e) micro open (center)
(l) micro open (right)
knobl Lon (1eft)
(h) knobl Ion (center)
knobl Lon (right)
(i) knobl_ofe (left)
(k) knobl_off (ceater)
() knobl_ (right)
(m) light_on (left)
(n) 1ight_on (center)
(o) 1ight_on (right)
(p) Light_off off (left)
(g) light_ off (center)
(r) light__ooo off (right)
Figure 9: Initial frame (Easy), initial frame (Hard), and goal frame for all 12 tasks and camera views our FrankaKitchen suite.
22

---

Published as a conference paper at ICLR 2023
Success Rate
Robot Pose Error
Object Pose Error
0.30
10-
0.
6.55
0.8
0.10
0.7
0.4
105
0.-
1.00
Steps
110 RDN
Paateer
CLP
Mc
Steps
Figure Trajectory optimization results with pose errors.
FrankaKitchen sdoor_open task (center view). For TCN, TCN loss without any modification; this also allows our findings in Figure 2 to extend to the main experiment section. The visual architecture ResNet34, and the output dimension us to directly visualize the learned embedding. Different from the out-of-domain version of VIP, we also do not perform weight penalty, trajectory-lest random cropping data augmentation, or additional negative sampling. Besides these choices, we use the same hyperparameters as in Table and train for 2000 batches.
R3M's implementation of the
which enables
E.3 TRAJECTORY OPTIMIZATION
We a publicly available implementation of MPP and make modification to the algorithm the default hyperparameters. In particular, the planning horizon is 12 and 32 sequences of actions proposed per action step. Because the embedding reward (equation is the goal-embedding distance difference, the score (i.e., sum of per-transition reward) of a proposed sequence of actions is equivalent the negative embedding distance (i.e., So (o(((r)) ((g)) at the last observation.
E.3.1 ROBOT AND OBJECT POSE ERROR ANALYSIS
In this section, visualize the per-step robot and object pose L2 error with respect to the goal-image poses. We report the non-cumulative curves (on the success rate as well) for more informative analysis.
E.4 REINFORCEMENT LEARNING
We use a publicly available implementation of NPC and make no modification to the algorithm or the default hyperparameters. In the Easy (resp. Hard) setting, we train the policy until 500000 (resp. 1M) real environment steps are taken. For evaluation, we report the cumulative maximum success rate on 50 test rollouts from each task configuration 150*108=540 total rollouts) every 10000 step.
REAL-WWORLD ROBOT EXPERIMENT DETAILS
F.1 TASK DESCRIPTIONS
The robot learning environment is illustrated in Figure 11 right edge of the table, and we only use the RGB image stream without depth information for data collection and policy learning.
RealSense camera is mounted on the
We collect offline data Dtack for each task via kinesthetic playback, and the object initial placement randomized for each trajectory. On the simplest CloseDrawer task, combine 10 expert demonstrations with 20 sub-optimal failure trajectories to increase learning difficulty. For the other
23

---

Published as a conference paper at ICLR 2023
Table 3; Real-world robotics tasks descriptions.
Environment
Object Type
Criterion
CloseDrawer
Articulated Object
demos failures
drawer closed enough that spring loads.
PushBottle Le
Transparent Object
20 demonstrations
the bottle parallel the goal line set by the icccream cone.
PlaceMelon FoldZowel
Soft Object
demonstrations
the watermelon toy is fully placed in the plate. the bottom half of the towel cleanly covered the top half.
Deformable Object
20 demonstrations
Figure 11: Real-robot sctup.
three tasks, we collect 20 expert demonstrations, which found are difficult enough for learning good policies. Each demonstration is 50-step long collected at 25Hz. The initial state for the robot is fixed for each demonstration and test rollout, but the object initial position randomized. The task success is determined based on visual criterion that we manually check for each test rollout. The full task breakdown is described in Table
Each task is specified via set of goal images that are chosen to be the last frame of all demonstrations for the task. Hence, the goal embedding used compute the embedding reward (equation T for each task is the average over the embeddings of all goal frames.
The tasks (in their initial positions) using a separate high-resolution phone camera are visualized in Figure 12 Sample demonstrations in the robot camera view are visualized in Figure 13
F.2 TRAINING AND EVALUATION DETAILS
The policy network is implemented 2-layer MLP with hidden sizes [256, 256). As in R3M's real-world robot experiment setup, the policy takes concatenated visual embedding of current observation and robot's proprioceptive state and outputs robot action. The policy is trained with learning rate of 0.001, and a batch size of 32 for 20000 steps.
For RWR's temperature scale, we more effective for both VIP and R3M.
0.1 for all tasks, except CloseDrawer where find
For policy evaluation, 10 test rollouts with objects randomly initialized reflect the object distribution in the expert demonstrations. The rollout horizon is 100 steps.
(a) CloseDraxer
(b) PushBottle
(c) JackPeaco
(d) FoldTowel
Figure 12: Side-view of real-robot tasks using high-resolution smartphone camera.
24

---

Published as a conference paper at ICLR 2023
(a) CloseDrawer
(b) PushBottle
(c)
(d) FoldTowel
Real-robot task demonstrations (every 10th frame) in robot camera view. The first and last frames in each row are representative of initial final goal observaions for respective task.
Figure 13:
F.3 ADDITIONAL ANALYSIS & CONTEXT
Offiine RL vs. imitation learning for real-world robot learning. Offline RL, though known as the data-driven paradigm of RL (Levine et al. 2020), is not necessarily data efficient (Agarwal et al. 2021), requiring hundreds of thousands of samples even in low-dimensions simulated tasks, and requires a dense reward to operate most effectively (Mandlekar et al. 202I I Yu et al. 2022). significantly
difficult implement and tune compared to BC (Kumar et al.] [202]] /Zhang & Jiang, 202T). As such, the dominant paradigm real-world robot learning is still learning from demonstrations (Jang al. [202]] Mandlekar et al. 2018 Ebert et al.
Furthermore, offline RI. algorithms
2021). With the advent of VIP-RWR, offline RL may finally be a practical approach for real-world
robot learning scale.
Performance of R3M-BC. Our R3M-BC, though able to solve some of the simpler tasks, appears to perform relatively worse than the original R3M-BC Nair al. (2022) on their real-world tasks. To account for this discrepancy, we note that our real-world experiment uses different software- hardware stacks and tasks from the original R3M real-world experiments, so results are not directly comparable. For instance, camera placement, an important variable for real-world robot learning, is chosen differently in our experiment and that of R3M; in R3M, a different camera angle selected for each task, whereas in our setup, the same camera view is used for all tasks. Furthermore, we emphasize that our focus not the absolute performance of R3,-BC, but rather the relative improvement R3M-RWR provides top of R3M-BC.
F.4 QUALITATIVE ANALYSIS
In this section, we study several interesting policy behaviors VIP-RWR acquire. Policy videos are included in our supplementary video.
Robust key action execution. VIP-RWR is able to execute key actions more robustly than the baselines; this suggests that its reward information helps it identify necessary actions. For example, as shown in Figure 14 on the PickP aceMe lon task, failed VIP-RWR rollouts at least have the gripper grasp onto the watermelon, whereas for other baselines, the failed rollouts do not have the watermelon between the gripper and often incorrectly push the watermelon to touch the plate' 's outer edge, preventing pick-and-place behavior from being executed.
25

---

Published as a conference paper at ICLR 2023
(a) VIP-RWR
(b) VIP-BC
(c) REM-RWR
(d) REM-RWR
Figure Comparison of failure trajectories on PickPlaccMe VIP-RWR still able reach the critical state gripping watermelon, whereas baselines fail.
MPPI Optimizer Scaling
Cumulative Success Rate
(Ego40) MBCOo (5ge40)
03-
0.21
3:15
$15
0.13
913
0.09
0.07
5.07
062
00
Trajectories per Step
Figure 15: VIP Altemative Pre-Training Algorithms Ego4D.
Task re-attempt. We observe that VIP-RWR often learns more robust policies that are able to perform recovery actions when the task is not solved on the first attempt. For instance, in both CloseDrawer and FoldTowel, there are trials where VIP-RW VIP-RWR fails to close the drawer all way or pick up the towel edge right away; in either case, VIP-RWR is able to re-attempt and solves the task (see our supplementary video). This is known advantage of offline RL over BC (Kumar et al., 2022; Levine et al. 2020); however, only observe this behavior in VIP-RWR and not R3M-RWR, indicating that this advantage of offline RI. is only realized when the reward information sufficiently informative.
G ADDITIONAL RESULTS
G.1 COMPARISON TO MAE AND MoCo TRAINED ON EGO4D
In this section, we compare to two additional representations trained on the same pre-training dataset of Ego4D. We compare them to VIP and R3M in the trajectory optimization setting and evaluate their performance as the optimization budget increases in the spirit of Figure 5 The result are shown in Figure 15] Consistent with the findings in the main text, the pre-training dataset is not the source of VIP's empirical gains; all prior state-of-art pre-training methods struggle as zero-shot reward functions. Notably, MAE
vision transformer (Dosovitskiy al. 2020) architecture backbone and exhibits an improving trend performance similar to VIP: however, MAE's absolute performance still far inferior to VIP.
G.2 VALUE-BASED PRE-TRAINING ABLATION: LEAST-SQUARE TEMPORAL-DIFFREENCE
While VIP is the first value-based pre-training approach and significantly outperforms existing methods, we show that this effectiveness is also unique to VIP and not to training value function. To this end, we show that a simpler value-basee baseline does not perform as well. In particular, consider Least-Square Temporal-Difference policy evaluation (LSTD) (Bradtke & Barto 1996
26

---

Published as a conference paper at ICLR 2023
Success Rate
Robot Pose Error
Object Pose Error
0.30
0.98
6.5
0.10
0220
1.50
143
0.
Steps
VP
LOTD
Steps
Figure VIP vs. LSTD Trajectory Opümization Comparison.
Table Visual Imitation Learning Results.
Self-Supervised LSTD (E)
Supervised
VIP
R3M-Lang (E)
MOCO
R3M
ResNet50
CLIP (Internet)
Success Rate
$1.2
41.8
44.3
Sutton & Barto 2018) to assess the importance the choice of value-training objective:
min (o,0' and (5.(0)+ -
(29)
in which we also parameterize V as the negative L2 embedding distance as VIP. Given that human videos reasonably goal-directed, the value of the human behavioral policy computed via LSTD should be a decent choice reward; however, LSTD does not capture the long-range dependency of initial goal frames (first term in equation 3),
it obtain value function that outperforms that the behavioral policy. We train LSTD using the exact same setup as in VIP, differing in only the training objective, and compare against VIP our trajectory optimization settings.
As shown in Fig. 16. interestingly, LSTD already works better than all prior baselines in the Easy setting, indicating that value-based pre-training indeed favorable for reward-specification. However, its inability to capture long range temporal dependency as in VIP (the first term in VIP's objective) makes it far less effective on the Hard setting, which require extended smoothness in the reward landscape to solve given the distance between the initial observation and the goal. These results show that VIP's superior reward specification comes precisely from ability to capture both long- range temporal dependencies and local temporal smoothness, two innate properties of its dual value objective and the associated implicit time contrastive learning interpretation. To corroborate these findings, we have also included LSTD in our qualitative reward curve and histogram analysis in App. G.5 G.7 and G.8 and finds that VIP generates much smoother embedding than LSTD.
G.3 VISUAL IMITATION LEARNING
One alternative hypothesis to VIP's smoother embedding for its superior wwward-peeeification capabil- ity is that it learns a better visual representation, which then naturally enables better visual reward function. To investigate this hypothesis, we compare representations' capability as a pure visual encoder in visual imitation learning setup. We follow the training and evaluation protocol of (Nair cct al. 2022) and consider tasks combined from Franka Kitche,, MetaWorld (Yu et al.) (020), and Adroit (Rajeswaran et al. 2017), 3 camera views for each task, and 3 demonstration dataset sizes, and report the aggregate average maximum success rate achieved during training. R3M-Lang the publicly released R3M variant without supervised language training. The average success rates all tasks are shown Table the letter inside () stands for the pre- pre-training dataset with E referring to Ego4D and / Imagenet.
27

---

Published as a conference paper at ICLR 2023
These results suggest that with current pre pre-training methods, the performance visual imitation learning may largely be a function of the pre-training dataset, as all methods trained on Ego4D, even our simple baseline LSTD, performs comparably and are much better than the next best baseline not trained on Ego4D. Conversely, this result also suggests that despite not being designed for this purely supervised learning setting, value-based approaches constitute a strong baseline, and VIP is in fact currently the state-of-art for self-supervised methods. While these results highlight that VIP effective even pure visual encoder, necessary requirement for joint effectiveness for visual reward and representation, it fails to explain why VIP is far superior to R3M in reward- -ased policy learning. As such, we conclude that studying representations' capability as a pure visual encoder may not be sufficient for distinguishing representations that can additionally perform zero-shot reward-specification.
G.4 EMBEDDING AND TRUE REWARDS CORRELATION
In this section, we create scatterplots of embedding reward vs. true reward on the trajectories MPPI have generated to assess whether the embedding reward is correlated with the ground-truth dense reward. More specifically, for each transition in the MPPI trajectories in Figure we plot its reward under the representation that was used to compute the reward for MPPI versus the true human-crafted reward computed using ground-truth state information. The dense reward in FrankaKitchen tasks is weighted sum of (1) the negative object pose error, (2) the negative robot pose error, (3) bonus for robot approaching the object, and (4) bonus for object pose error being small. This dense reward highly tuned and captures human intuition for how these tasks ought to be best solved. As such, high correlation indicates that the embedding is able to capture both intuitive robot-centric and object-centric task progress from visual observations. We only compare VIP and R3M here as a proxy for comparing our implicit time contrastive mechanism to the standard time contrastive learning.
The scatterplots over all tasks and camera views (Easy setting) are shown in Figure 17 18 and 19 VIP rewards exhibit much greater correlation with the ground-truth reward on its trajectories that do accomplish task, indicating that when VIP does solve a task, is solving the task in a way that matches human intuition. This is made possible via large-scale value pre- vee-traniig on diverse human videos, which enables VIP to extract a human notion of task- pask-prossess that transfers to robot tasks and domains. These results also suggest that VIP has the potential of replacing manual reward engineering. providing a data-driven solution to the grand challenge of reward engineering for manipulation tasks. However, VIP is not yet perfect in its current form. Both methods exhibit local minima where high embedding distances in fact map to lower true rewards; however, this phenomenon is much severe for R3M. On 8 out of 12 tasks, VIP at least has camera view in which its rewards are highly correlated with the ground-trui rewards on its MPPI trajectories.
G.5 EMBEDDING DISTANCE CURVES
In Figure 20 we present additional embedding distance curves for all methods on Ego4D and our real-robot offline RL datasets. For Ego4D, randomly sample 4 videos of 50-frame long (see Appendix G.6 for how these short snippets are sampled), and for our robot dataset, we compute the embedding distance curves for the 4 sample demonstrations in Figure [3] As shown, on all tasks in the real-robot dataset, VIP is distinctively more smooth than any other representation. This pattern less accentuated on Ego4D. This is because a randomly sampled 50-frame snippet from Ego4D may not coherently represent a task solved from beginning to completion, so an embedding distance curve is not inherently supposed to be smoothly declining. Nevertheless, VIP still exhibits more local smoothness in the embedding distance curves, and for the snippets that do solve task (the first two videos), it stands out as the smoothest representation.
G.6 EMBEDDING DISTANCE CURVE BUMPS
In this section, we compute the fraction of negative embedding rewards (equivalently, positive slopes in embedding embedding distance curves) for each video sequence and average over all video sequences in a dataset. Each sequence in our robot dataset is of 50 frames, and we use each sequence without any further truncation. For Ego4D, video sequences are of variable length. For each long sequence of more than 50 frames, we use the first 50 frames. We do not include videos shorter than 50 frames, in order to make the average fraction for each representation comparable between the
28

---

Published as a conference paper at ICLR 2023
Saiis Lulen Jowe choaadd Vasccleeet
True
-545
Emboeosing Distarce
Embeoding Disancce
Emozdding
1 cccor_close (left)
(b) 1door_close
(c) 1dcor_cloooo (right)
-10
Distarce
-A2
-20
-04
Distance
Enweddiing
(d) _door_open (left)
(e) 1ccor_open
(f) 1door_ _ (right)
Tastentor on saber dise Vismobbutt
-12-
Trus
Entooosing Distace
-M4 -12
Embodding
(g) sdoor close (left)
(h) sdoor close
ii) sdcor close (right)
Witchen stoor open 12 Viewieet
Testsutchen ocen v3 defuu
Trschttthnn sdber spen Vewndgte
OB. G-eil
A76
012
Thus Benoor
-115
Enbeaddnn ng Detarce
Entiedaing Distance
Embebddnn Distance
(i) sdoor_opon (left)
(k) sdoor_open
(1) sdoor_open (right)
Figure 17: Embedding reward vs. ground-truth human-engincered reward correlation (VIP vs. R3M) part 1.
29

---

Published as a conference paper at ICLR 2023
rooor COSO Vieweet cam
TodkMtiherr 10927 0e22 Viewoeffult
bastet en idbor cisse va Viowiight com
E33-
Thue seowrr
-0.0
Emboaddng ng Dstarce
Entooding Ditancee
Embeddinn Distance
(a) rdoor closee (left)
(b) _door_close
(c) edoor_close (right)
Insickitcher. rdoor VAAAITT..om
lastzetcheen en, /daor span-dd Mawoddelaad
laststticeer rdoor Viettngtt
Reword
Thue seoerr
True
100
Imbessinn Distarce
Entredaing Distacce
Empredaing Distance
(d) r rdoor_open (left)
(c) rdoor_open
(f) rddor_ooppppp rpoen (right)
Think Mither micro dose Veeclet cam
Tisk Kitchon wwwoomnooooooooo 3 Viceddoout
Testistcchen micro diose Vientiotr can
Dowwerr
-03
Embedd Debance
Embedding Distencee
da Emberding
nicrooccose (left)
(h) ciicooooosee
(i) cicrroclose (right)
micro open Woellef.co cam
micro Viewdehutt
spenest
nal Stw
Thu Reword 1a
-44
<de
Embeiding
micro open (left)
(k) micro_open
(1) micro_open (right)
Figure 18: Embedding reward vs. ground grouhd-truth human-enninenn engineered reward correlation (VIP vs. R3M) part 2.
30

---

Published as a conference paper at ICLR 2023
Thil: Thlt-ittren knchl cou't View Ieft
Teekittrhen Inshi 2os'l Veraoceauit
Tetbeetrben knnh1 mant Vinandote Anacin
Seaured
2. meaielle
Emboodding Distarce
Embeddinng Distance
Emboddding Distance
(a) knobl_on (left)
(b) knob on
(c) knobl_on (right)
Thil ittchen Inohl cou? View Ieft
Thil litrhen, Imobi onr23 Viencetuut
Thole kntthen kmob1 mnd View fott
4. star
Ennoooiing Distarce
Embbedding Distance
Emboodding Distanoo
(d) knobl_on (left)
(e) knobl_on
(f) knobl_ll (right)
Toscidtchen on Vew 1 geo" '/6 yemr 2
Task lobchen litt View detuut
Toseattchen Iate cn93 Viewr
Emoodd ng Distanca
Embeedding Disaano
Emboddiing Distance
(g) 1ight_on (left)
(h) light_on on
(i) light_ on (right)
Toscuutohenn off view
Thstishotenn ligtt off e Venieebut
TTisk kitthen Latn oftid Vieerioh cam
sut o-u
Trus
This
Embedd Distance
Emberding Distance
Embebding Distance
(i) light_off (left)
(k) light_off
()) light_off (right)
Figure 19: Embedding reward vs. ground ground-truth human- eeninneee reward correlation (VIP vs. R3M) part 3.
31

---

Published as a conference paper at ICLR 2023
(a) Ego4D
(b) Real-robot dataset
Figure 20: Additional embedding distance curves on Ego4D and real-robot videos.
Table Proportion of bumps embedding distance curves.
Dataset
VIP (Ours)
R3M
ResNer50
MOCO
CLIP
Ego4D
0.253 + 0.117
0.309
0.097
0.414
0.052
0.398 + 0.057
0.444 0.047
In-House Robot Dataset
0.243 + 0.066
0.323 0.076
0.366 + 0.046
0,380 + 0.052
0.438 + 0.046
two distinct datasets. Note that for Ego4D, due to its in-the-wild nature, it is not guaranteed that 50-frame segment represents one task being solved from beginning to completion, so there may be naturally bumps in the embedding distance curve computed with respect the last frame, as earlier frames may not actually be progressing towards the last frame goal-directed manner. The full results shown T VIP has fewest bumps in Ego4D videos, and this notion of smoothness transfer to the robot dataset. Furthermore, since the robot videos are in fact visually simpler and each video guaranteed to be solving one task, the bump rate actually lower despite the domain gap. While this observation generally also holds true for other representations, it notably does not hold for R3M, which trained using standard time contrastive learning.
G.7 EMBEDDING REWARD HISTOGRAMS (RBAL-ROBOT DATASET)
We present the reward histogram comparison against all baselines in Figure 21 The trend of VIP having more small, positive rewards and fewer extreme rewards either direction is consistent across all comparisons.
G.8 EMBEDDING REWARD HISTOGRAMS (EGo4D)
We present the reward histogram comparison against all baselines in Figure 22 The histograms are computed using the same set of 50-frame Ego4D video snippets as in Appendix G.6). The y-axis is in log-scale due to the large total count of Ego41) frames. As discussed, Ego41) video segments are less regular than those our real-robot dataset, and this irregularity contributes to all representations having significantly more negative rewards compared to their histograms on the real-robot dataset. Nevertheless, the relative difference ratio's 's pattern is consistent, showing VIP having far more rewards that lie in the first positive bin. Furthermore, VIP also has significantly fewer extreme negative rewards compared to all baselines.
H LIMITATIONS AND FUTURE WORK
In this section, we describe limitations within the current VIP formulation and model and some potential future directions.
32

---

Published as a conference paper at ICLR 2023
us Eritedding Seward
RaN Eribeddic Reward Drfinence
(SP Raciet Erteeding Rewond
MPPv, Raciet Eybeddin Anxard Oeference Oll
Eritedding REARI
Groedding RRAST2
Embedding Seward
Frbedding Aeword
(a) VIP vs. R3M
(b) VIP vs. ResNet
MoGa Eribedding Aeward
Modo Eroodding Reward
inbekting
Embeoding fawurd
O1oedding
Erttedting
VIP vs. MoCo
(d) VIP vs. CLIP
-01 45 05
+11 03
(c) VIP
LSTD
Figure 21: Embedding reward histogram comparison on real-robot dataset.
you Embedd Sawurd
Enbedding Boward Offhreace
Embbadd
Entadann Bewwrd
Enraacting ReARTS
(a) VIP R3M
(b) VIP vs. ResNet
Enbediing
Cmoeedinng Reaers
Embedding Ranard
Enbedsing Bear'd
(c) VIP vs. MoCo
(d) VIP vs. CLIP
Unbedding
Croadding HEAEr
VIP LSTD
Figure 22: Embedding reward histogram comparison Ego4D videos.
33

---

Published as a conference paper at ICLR 2023
VIP is currently limited to providing rewards for tasks that can be specified via goal image. While this encompasses a wide range of robotics tasks, many tasks cannot be fully expressed via a static image, such as ones that require following intermediate instructions and steps. Likewise, though not a strict assumption, we have only tested VIP with visual goals from the same domain (robots are not necessarily in the goal image). Extending VIP be compatible with even more flexible and extensive forms of goals is fruitful direction for expanding VIP's capability.
VIP current parameterizes the value function symmetric embedding distance; this assumes that the environment is reversible (i.e.. it is equally easy to get from o to and from to o), which may not hold in practice. While we did not observe this to affect practical performance, we may improve performance by parameterizing V (o;g) g) as some distance function that supports asymmetrical structures, such as quasimetrics. Extending VIP with recent method (Wang & Isola 2022) that can learn quasimetrics with finite data may be fruitful future direction.
We have also used VIP only as a frozen visual reward and representation module test its broad generalization capability. Better absolute task performance may be achieved by fine-tuning VIP on taek-specific data. Exploring how to best fine-tune VIP a promising direction for pushing VIP's limit.
Finally, we have focused on robot manipulation tasks this work, but VIP's training objective can also be used for pre- vee-rraining reward and representation for other goal-directed tasks, such visual navigation (Savva et al. 2019). Exploring how VIP can be used to solve these other embodied AI tasks is also promising avenue of future work.
34


Reviews:
[{"review_id": "OHtDZT6jvl", "reviewer": "Reviewer_DfUc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents Value-implicit pretraining (VIP) which is a method to train representations from images for use in robotic reinforcement learning.  The method uses contrastive learning to generate smooth representations of trajectories, where the start and goal states of trajectories are encouraged by the loss to be close in the representation space and intermediate observations are encouraged to interpolate smoothly between the two.  Because the representation is trained to interpolate between start end goal states, the authors argue that this interpolation is equivalent to a value function when the reward is a goal-reaching reward.  The representation can thus be used to assess distance to the goal state, and thus can be used as a dense reward.  The paper shows strong empirical results generalizing representations learned in egocentric human video data sets to robotic tasks.", "summary_of_the_review": "This is a very strong paper with an interesting approach to a difficult and important problem.  The empirical results are strong and I think this is a clear accept.", "strength_and_weaknesses": "The paper presents very compelling empirical results, and a clear derivation and description of the method.  The method is novel, and takes large steps toward solving very difficult problems in reinforcement learning and robotics.  The literature is carefully and extensively surveyed, and the contributions of VIP are clearly laid out in contrast to the relevant prior methods.  The method is also straightforward, and can be summarized by a three step algorithm, which suggests it can be easily reproduced.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "L6Ussxy7-W", "reviewer": "Reviewer_s2zb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is in the realm of generalizable reward learning and the aim is to learn value-discriminative visual representation from a diverse set of human manipulation demonstrations by contrasting conditioned on goals. The learned representation is used for constructing reward as similarity in embedded space between adjacent steps. The proposed time contrastive method differs from standard unidirectional contrastive learning, it attracts initial and final frames while repelling intermediate frames, thus presenting smoothly decreased embedding distances. The method is evaluated on trajectory optimization, online RL, and real-world offline RL tasks to demonstrate effectiveness of the visual representation over other pretraining baselines.", "summary_of_the_review": "The paper proposes an efficient method for visual reward and representation learning from out-of-domain data, which highly benefits downstream tasks. I would say it is an improvement to existing work but can be a good inspiration on how to contrastively learn a representation of demonstrations.", "strength_and_weaknesses": "Strength:\n1. The motivation of learning visual reward functions from human demonstrations without action annotations and then generalizing it to unseen robotic control tasks is intriguing.\n2. The proposed dual-objective goal-conditioned contrastive learning enables more accurate and smooth embedding distance through full demonstration.\n3. The paper is well-written and easy to follow.\nWeakness:\n1. In Section 5.2, Table 1 shows that VIP in-domain representation learning is worse than Scratch-BC and even has no task progress due to overfitting. Is VIP easily affected by the quality of the demonstration? What if sub-optimal or not diverse enough, will it hinder RL update? \n2. Does VIP exceed in-domain imitation learning metrics? It would be more convincible if you compare your metric with IL baselines trained on in-domain demonstrations in Kitchen benchmarks, for example, behavior transformer, Implicit Behavioral Cloning.\n3. Is your model able to handle non-Markovian demonstrations where different actions concatenate sequentially, instead of single clips?\n4. On your project website, robot motion in MPPI trajectory seems to be fluctuating and unstable, is there any explanation or improvement for this problem?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EpV7JFlij1", "reviewer": "Reviewer_EXzT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a method, VIP, to learn both an observation representation and reward using a self-supervised objective over an out-of-domain dataset. The representation learning problem is cast as offline goal-conditioned reinforcement learning for the tasks covered by the out-of-domain datasets, i.e. human activities. The reward used to learning the visual representation is 0 when the observed and goal states are the same otherwise it is -1. The reward function is selected to facilitate learning the rough distance between the current and goal frames. The representation learning objective is then cast as implicit time contrastive learning and roughly resembles InfoNCE, but without explicit negative examples. The value function for a given task is then the embedding distance between the current state and the goal state. The authors assess the impact of VIP against several representation learning baselines on trajectory optimization tasks, online RL tasks, and few-shot learning tasks to real-world robot manipulation tasks. Across experimental conditions, VIP outperforms the baselines.", "summary_of_the_review": "Overall the paper is well written, the results demonstrate clear gains on prior work, and both empirical and theoretical evaluation is technically sound. The ability to learn representations that enables few shot learning from out-of-domain, offline data can have a large impact on the field. The largest concern with the paper is the amount of information that was punted to the related work section. The related work section and limitations are key portions of a paper to appropriately contextualize and understand the method's contribution and, therefore, that should be included in the paper's main body.", "strength_and_weaknesses": "Strengths:\n- The authors conduct extensive experiments to assess the performance of VIP and to understand the benefits of each of the proposed components. In addition to conducting experiments in simulation, the authors assess the performance of VIP in a real world robot setting.\n- The authors evaluate useful and relevant baselines for learning observation representations. \n- The results demonstrate strong gains on prior work at the task of learning from videos of humans, which is the majority of activity-specific data available.\n- Although in the appendix, the authors provide proofs alongside their proposed VIP components.\n- The paper is well written.\n\nWeaknesses:\n- Large portions of the paper are punted to the appendix, include two-thirds of the related work section, two whole experiments, limitations, and future work. Despite working in a goal-conditioned RL setting, the authors punt the goal-conditioned RL related work to the appendix which makes it difficult to place the proposed method within the context of the relevant goal-conditioned RL prior work.\n- It would be helpful to see how VIP performs against goal-conditioned RL where the observation encoding/representation is learned online alongside policy learning. \n- For the experiments in Section 5.2, how were the hyper-parameters selected? This is important to specify to make it clear that hyper-parameters favorable to VIP were not selected. \n- The need to not assess VIP under the closed-loop reinforcement learning setting is not well motivated, especially as goal-conditioned RL is deployed in such a setting. \n- Online RL results for VIP (sparse) where the policy is trained using the VIP state representation and the environment's reward do not appear to be included in the paper. There is no reference to where the results can be round. Additionally, it would be nice to have more of a discussion about why it is okay for VIP's state representation to be ineffective when learning from the ground truth reward function. This is where a comparison against an online, goal-conditioned RL algorithm would be beneficial to see. Are no methods able to solve the task in the online setting? Or only those methods evaluated?\n- It would be helpful to specify the length of video sequences used to train the state representations and to assess the impact of video sequence length on performance. \n- It is not clear if contrastive RL (https://arxiv.org/pdf/2206.07568.pdf) is evaluated against and it would be interesting to see a discussion about the similarities and differences included in the paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0KJWYQmBIq", "reviewer": "Reviewer_2WZ8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a self-supervised visual backbone pretraining method for reward learning in control. The key idea is to formulate representation learning from egocentric videos as an offline goal-conditioned reinforcement learning problem, whose dual-form indicated a way to implement Value-Implicit Pre-training (VIP) on unlabeled human videos. In each training iteration, VIP samples a minibatch of sub-trajectories from video, and the loss function has two components, where the first one minimizes the L2 distance of the representation of initial and goal frames, and the last one optimize the difference of L2 distance of two consecutive intermediate frames (e.g. $\\cdots \\gamma (F(o_{k+1})-F(o_k))$, $F$ measures the L2 distance to the goal frame embedding.). In another word, this implicit time contrastive\nlearning push together the feature of start/end frame, and pull away the intermediate frames implicitly. \nBesides the elegant theory, the author pretrained their visual representation model on large-scale Ego4D human videos, and show the proposed pretraining idea can be combined with various on-domain solutions and significantly outperform all prior pre-trained representations on a set of simulated and real-robot tasks. A partial PyTorch code is provided in Appendix. ", "summary_of_the_review": "It is a novel idea to derive a self-supervised pre-training target from a goal-conditioned value function from RL, and the experiments are clear and supportive. I vote to accept this good paper.", "strength_and_weaknesses": "(+) The main logic of this paper is to derive a reinforcement learning target as self-supervised pre-training tasks, and use it for reinforcement learning. Therefore, it gives less domain gap in visual representation learning than in traditional CV-pretraining solutions. \n\n(+) The method is able to compare with R3M, which also works on visual representation for robot learning. VIP does not require extra annotations such as narration and text alignment, so it is more flexible and generic. \n\n(+) Experiments on visual trajectory optimization and online RL aggregate show VIP outperforms prior pre-trained representations, on both easy and hard cases.\n\n(+) On a suite of real-world robot tasks, VIP can support few-shot offline reinforcement learning with smoother embedding distance curves.\n\n(-) Although the derived pre-training target has a complicated form to implicitly optimize the value function, it is better to explain the physical meaning of each term. For example, the exponential term in log-expectation can be viewed as $$exp(||\\phi(o)-\\phi(g)||_2-\\tilde \\delta_g(o)-\\gamma||\\phi(o')-\\phi(g)||_2)=\\frac{exp((1-\\gamma)||\\phi(o)-\\phi(g)||_2-\\tilde \\delta_g(o))}{exp((\\gamma)(||\\phi(o')-\\phi(g)||_2-||\\phi(o)-\\phi(g)||_2))}$$\nthen maximizing the denominator is to maximize the one-step temporal difference, which is contradictory to the experimental observation. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%