RoboCLIP: One Demonstration is Enough to Learn Robot Policies
Authors: Sumedh Anand Sontakke
Jesse Zhang
Séb Arnold
Karl Pertsch
Erdem Biyik
Dorsa Sadigh
Chelsea Finn
Laurent Itti
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Vision and Language Models
TL;DR: We use Video-and-Language Models to generate rewards for reinforcement learning agents by specifying tasks using natural language and out-of-domain demonstrations.

Abstract: Reward specification is a notoriously difficult problem in reinforcement learning, requiring extensive expert supervision to design robust reward functions. Imitation learning (IL) methods attempt to circumvent these problems by utilizing expert demonstrations instead of using an extrinsic reward function but typically require a large number of in-domain expert demonstrations. Inspired by advances in the field of Video-and-Language Models (VLMs), we present RoboCLIP, an online imitation learning method that uses a single demonstration (overcoming the large data requirement) in the form of a video demonstration or a textual description of the task to generate rewards without manual reward function design. Additionally, RoboCLIP can also utilize out-of-domain demonstrations, like videos of humans solving the task for reward generation, circumventing the need to have the same demonstration and deployment domains. 
RoboCLIP utilizes pretrained VLMs without any finetuning for reward generation. Reinforcement learning agents trained with RoboCLIP rewards demonstrate 2-3 times higher zero-shot performance than competing imitation learning methods on downstream robot manipulation tasks, doing so using only one video/text demonstration. Visit our website at https://sites.google.com/view/roboclip/home for experiment videos.

RoboCLIP: One Demonstration is Enough to Learn Robot Policies
Sumedh A Sontakke Jesse Zhang', Sébastien M. R. Arnold', Karl P Erdem Bryok! Dorsa Sadigh", Chelsea Finn', and Laurent Itti'
Thomas Lord Department of Computer Science, University of Southern California 2 University of California, Berkeley
Stanford University Google Research
Abstract
Reward specification is notoriously difficult problem in reinforcement learning, requiring extensive expert supervision to design robust reward functions. Imi- tation learning (IL.) methods attempt circumvent these problems by utilizing expert demonstrations but typically require large number of in-domain expert demonstrations. Inspired by advances in the field of Video-and-Languaee Models (VLMs), we present RoboCLIP, an online imitation learning method that uses single demonstration (overcoming the large data requirement) in the form of a video demonstration or textual description of the task to generate rewards without manual reward function design. Additionally, RoboCLIP can also utilize out-of-domain demonstrations, like videos of humans solving the task for reward generation, circumventing the need have the same demonstration and deploy- ment domains. RoboCLIP utilizes pretrained VLMs without any finetuning for reward generation. Reinforcement learning agents trained with RoboCLIP rewards demonstrate 2-3 times higher zero-shot performance than competing imitation learning methods on downstream robot manipulation tasks, doing so using only one video/text demonstration. Visit our website for experiment videos.
Introduction
Sequential decision-making problems typically require significant human supervision and data. In the context of online reinforcement learning [Sutton and Barto, 2018], this manifests in the design of good reward functions that map transitions to scalar rewards [Amodei et al., 2016, Hadfield-Mened et al., 2017]. Extant approaches to manual reward function definition are not very principled and defining rewards for complex long- long-horizon problems is often an art requiring significant human expertise. Additionally, evaluating reward functions often requires knowledge of the true state of the environment. For example, imagine a simple scenario where the agent must learn to lift an object off the ground. Here, reward useful for task success would be proportional the height of the object from the ground - a quantity non-trivial to obtain without full state information. Thus, significant effort has been expounded in developing methods that learn reward functions either explicitly or implicitly from demonstrations, i.e., imitation learning [Pomerleau, 1988, Ng and Russell, 2000, Abbeel and Ng, 2004, Ziebart al., 2008]. With these methods, agent policies either be directly extracted from the demonstrations or trained to optimize rewards functions learned from them.
Imitation learning (IL), however, only somewhat alleviates the need for expert human intervention. First, instead of designing complex reward functions, expert supervision is needed to collect massive
corresponding author:
ssontakk @ @usc.edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Environment
0p:T
Every timestep
Enn of the episode
S3D-v
Policy
Similarity
Z'
"Robot opening dimmerr
S3D
Task Descriptor (video text)
Figure RoboCLIP Overview. Pretrained Video-and- Lan--ugg Model is used to generate rewards the similarity score between the encoding of an episode interaction of an agent its environment, z' with the encoding of a task specifier z" such as a textual description of the task or a vidco demonstrating a successful trajectory. The similarity score between the latent vectors is provided as reward to the agent.
datasets such as RT-1 1 [Brohan et al., 2022], Bridge Dataset [Ebert et al., 2021]. D4RL [Fu et al., 2020], or Robonet [Dasari al., 2019]. The performance of imitation learning algorithms and their ability to generalize hinges on the coverage and size of data [Kumar et al., 2019, 2022], making the collection of large datasets imperative. Second and most importantly, the interface for collecting demonstrations for IL is tedious, requiring expert robot operators to collect thousands of demonstrations. On the contrary, more intuitive way define rewards would be in the form of textual description (e.g.. "robot grasping object"), ) or in the form of naturalistic video demonstration of the task performed by a human actor an environment separate from the robotic environment. For example, demonstrating to robot how to open cabinet door in one's 's own kitchen is more naturalistic than collecting many thousands of trajectories via teleoperation in the target robotic environment.
Thus, there exists an ummet need for IL algorithms that 1) require very few demonstrations and 2) allow for a natural interface for providing these demonstrations. For instance, algorithms that can effectively learn from language instructions or human demonstrations without the need for full environment state information. Our key insight is that by leveraging (VLMs) Which are already pretrained on large amount of video demonstration and language pairs we do not need rely on large-scale and in-domain datasets. Instead, by harnessing the power of VLM embeddings, we treat the mismatch between single instruction' 's embedding (provided as language command or a video demonstration) and the embedding of the video of the current policy's rollout as a proxy reward that will guide the policy towards the desired instruction.
Lan-Lagg Models
To this end, we present RoboCLIP, imitation learning algorithm that learns and optimizes reward function based on single language or video demonstration. The backbone model used in RoboCLIP S3D [Xie al., 2018] trained on the Howto]00M dataset [Miech et al., 2019], which consists of short clips of humans performing activities with textual descriptions of the activities. These videos typically consist of variety of camera angles, actors, lighting conditions, and backgrounds. We hypothesize that VLMs trained on such diverse videos are invariant to these extrameous factors and generate an actor-agnostic semantically-meannngee representation for a video, allowing them to generalize to unseen robotic environments.
We present an overview of RoboCLIP in Figure 1. RoboCLIP computes similarity score between videos of online agent experience with a task descriptor, i.c., a text description of the task or single human demonstration video, generate trajectory-level rewards RoboCLIP on the Metaworld Environment suite [Yu et al., 2020] and on the Franka Kitchen Environ- ment [Gupta et al., 2019], and find that policies obtained by pretraining on the RoboCLIP reward result in 2 3x higher zero-shot task success in comparison to of-te-of-thh-art imitation learning baselines. Additionally, these rewards require no experts for specification and can be generated using naturalistic definitions like natural language task descriptions and human demonstrations.
train the agent. We evaluate
2 Related Work
Learning from Human Feedback. Learning from demonstrations is long-studied problem that attempts to learn policy from a dataset of expert demonstrations. Imitation learning (IL) methods, such as those based on behavioral cloning [Pomerleau, 1988], formulate the problem as supervised learning over state-action pairs and typically rely on large datasets of expert-collected trajectories directly demonstrating how to perform the target task [Brohan et al., 2022, Lynch et al., 2022].

---

However, these large demonstration datasets are often expensive to collect. Another IL strategy inverse RL, i... directly learning reward function from the demonstrations [Ng and Russell, 2000, Abbeel and Ng. 2004, Ziebart et al., 2008, Finn et al., 2016]. Inverse RI. algorithms are typically difficult to apply when state and action spaces are high-dimensional. Methods such as GAIL [Ho and Ermon, 2016), AIRL [Fu et al., 2017], or VICE [Fu et al., 2018] partially address these issues by assigning rewards which are proportional to the probability of given state being from the demonstration set valid goal state as estimated by learned discriminator network. However these discriminator networks still require many demonstrations or goal states to train to effectively distinguish between states from agent-collected experience and demonstration goal states. On the other hand, RoboCLIP's s use of pretrained train agents that learn to perform target tasks with just one demonstration in the form of a video or language description. Other works instead use human feedback in the form of pairwise comparisons or rankings to learn preference reward functions [Christiano al., 2023, Sadigh al., 2017, Biyik et al., 2019, Myers et al., 2021, Bryrk et al., 2022, Brown et al., 2019, Biyik et al., 2020, Lee et al., 2021, Hejna and Sadigh, 2022]. These preferences may require less human effort to obtain than reward functions, e.g., through querying humans to simply rank recent trajectories. Yet individual trajectory preferences convey little information on their own (less than dense reward functions) and therefore humans need to respond to many preference queries for the agent to learn useful reward functions. In contrast, RoboCLIP is able to extract useful rewards from a single demonstration or single language instruction.
Midlo--nd-laan and-language models allows us to
Large Vision and Language Models as Reward Functions. Kwon et al. [2023] and Hu and Sadigh [2023] propose using large language models (LLMs) for designing and regularizing reward functions that capture human preferences. These works study the reward design problem in text-based games such as negotiations or card games, and thus are not grounded in the physical world. RoboCLIP instead leverages video-and -nd--anndglaaggg models to assess if video demonstrations of robot policies align with an expert demonstration. Prior work has demonstrated that video models can be used as reward functions. For example, Chen et al. [2021] learn a visual reward function using human data and then utilize this reward function for visual model-based control of a robot. However, they require training the reward model on paired human and robot data from the deployment environment. We demonstrate that this paired data assumption can be relaxed by utilizing large-scale vision- lision-language models pretrained on large corpora of human- -eenenneeeerated data. The most well-known of these is CLIP [Radford et al., 2021], which is trained on pairs of images and language descriptions scraped from the internet. While CL.IP is trained only on images, video-language-mdde (VLMs) trained on videos of humans performing daily tasks such as S3D [Xic et al., 2018] or XCLIP [Ni et al., 2022] are also widely available. These models utilize language descriptions while training to supervise their visual understanding so that semantically similar vision inputs are embedded close together in shared vector space. A series of recent works demonstrate that these VLMs can produce useful rewards for agent learning. Fan et al. [2022] finetune CLIP on YouTube videos of people playing Minecraft and demonstrate that the finetuned CLIP model can be used
canguage-conditoone reward function to train an agent. DECKARD [Nottingham et al., 2023] then uses the fine-tuned reward function of Fan et al. [2022] to reward agent for completing tasks proposed by large- language model and abstract world model. PAFF [Ge et al., 2023] uses a fine-tuned CLIP model to align videos of policy rollouts with a fixed set of language skills and relabel experience with the best-aligned language label. We demonstrate that videos and multi-modee task specifications can be utilized to learn reward functions allowing for training agents. Additionally, we present a method to test the alignment of pretrained VLMs with deployment environments.
3 Method
Overview. RoboCLIP utilizes pretrained iideo-and-language models to generate rewards for online RL agents. This is done by providing a sparse reward to the agent at the end of the trajectory which describes the similarity of the agent's 's behavior to that of the demonstration, We utilize video-and- language models as they provide the flexibility of defining the task in terms of natural language descriptions or video demonstrations sourced either from the target robotic domain or other more naturalistic domains like human actors demonstrating the target task in their own environment. Thus, a demonstration (textual or video) and the video of an episode of robotic interaction are embedded into the semantically meaningful latent space of S3D [Xie et al., 2018], a video-and- lad-aauage model pretrained on diverse videos of human actors performing everyday tasks taken from the HowTol00M
3

---

dataset [Miech et al., 2019]. The two vectors are subsequently multiplied using a scalar product generating similarity score between the vectors. This similarity score (without scaling) is returned to the agent as reward for the episode.
Notation. We formulate the problem in the manner of POMDP (Partially Observable Markov Decision Process) with (O, S. A. 6, 0, r. T. 7) representing an observation space O, state space S, action space A, transition function Φ, emission function 0, reward function r, time horizon T, and discount factor 7, An agent in state takes an action a and consequently causes transition in the environment through o($ St, az). The agent receives the next state St+1 and reward rr r ax) calculated using the observation O₁. The goal of the agent is to learn policy и which maximizes the expected discounted sum of rewards, i.e., Σ 2"r,. Note that all of our baselines utilize the true state for reward generation and for policy learning. To examine the effect of using a baded-baeed reward, we also operate our policy on the state space while using the pixel observations for reward generation. Thus, Tt uses while л uses 8, for RoboCLIP while for all other baselines, both r. and x utilize 8ç- This of course unfair to our method, but we find that in spite of the advantage provided to the baselines, RoboCLIP rewards still generate higher zero-shot success.
Reward Generation. During the pretraining phase, we supply the RoboCLIP reward to the agent in sparse manner at the end of each episode. This is done by storing the video of an episode of the interaction of the agent with the environment into a buffer as seen in Figure 1. A sequence of observations of length 1 128 are saved in buffer corresponding to the length of the episode. S3D is trained on videos length 32 frames and therefore the episode video is subsequently downsampled to result in a vidco of length T 32. The video is subsequently center-cropped to result in frames of size (250, 250). This is done to ensure that the episode video is preprocessed to match the specifications of the HowToo000 observations ourr is encoded as the latent video vector z" using
preprocessing used to train the S3D model. Thus the tensor of sequence of T
z = S3D Siddoeico-eooott (Op.T)
The task specification is also encoded into the same space. If is defined using natural language, the language encoder in S3D encodes sequence of K textual tokens do.k into the latent space using:
zd Sext-xx-nnncoorr (du:k)
If the task description is in the form of video of length K, then we preprocess and encode it using the video-encoder in S3D just as in Equation (1). For intermediate timesteps, i.e., timesteps other than the final one in an episode, the reward supplied to the agent is zero. Subsequently, at the end of the episode, the similarity score between the encoded task descriptor z" and the encoded video of the episode z" is used as reward ,.RoboLLPPP (T). Thus the reward is:
RoboCLIP (1)
2 z"
t T=T
where zd zz corresponds to the scalar product between vectors zd and z".
Agent Training. Using RobuCLIPP defined above, we then train agent online in the deployment environment with any standard reinforcement learning (RL) algorithm by labeling each agent experi- ence trajectory with RoboCLIP after the agent collects it. In our paper, we train with PPO [Schulman et al., 2017], an on-policy RL algorithm, however, RoboCLIP can also be applied to off-policy algorithms. After training with this reward, the agent can be zero-shot evaluated or fine-tuned on true environment reward on the target task in the deployment environment.
Experiments
We test out each of the hypotheses defined in Section on simulated robotic environments. Specifi- cally, we ask the following questions:
1. Do existing pretrained VLMs semantically aliga with robotic manipulation environments?
2. Can we utilize natural language to generate reward functions?
3. Can we use videos of expert demonstrations to generate reward functions?
4. Can we use out-of-doman videos to generate reward functions?

---

5. Can we generate rewards using combination of demonstration and natural language?
6. What aspects of our method are crucial for success?
We arrange this section to answer each of these questions. Both RoboCLIP and baselines utilize PPO [Schulman et al., 2017] for policy learning.
Baselines. We use state-of-the-ar methods in inverse reinforcement learning: GAIL, or Gen- erative Adversarial Imitation Learning [Ho and Ermon, 2016] and AIRL or Adversarial Inverse Reinforcement Learning [Fu al., 2017]. Both of these methods attempt to learn reward functions from demonstrations provided to the agent. Subsequently, they train an agent using this learned reward function to imitate the expert behavior. Both methods receive a single demonstration, consistent with approach of using single video imitation. However, since they both operate on the ground environment state, we provide them with a trajectory of states, instead of images, thereby providing them privileged state information that our method does not receive.
4.1 Domain Alignment
Pretrained vision models often trained variety of human-centric activity data, such Ego4D) [Grauman et al., 2022]. Since we are interested in solving robotic tasks with view from third person perspectives, we utilize the S3D [Xie et al., 2018] VLM pretrained How 100M [Miech et al., 2019], dataset of short third-person clips of humans performing everyday activities. This dataset, however, contains no robotic manipulation data.
To analyze the alignment of the VLM to different domains, we perform confusion matrix analysis using videos from Metaworld [Yu et al., 2020]. We collect 10 videos per task with varying values of true reward. For each video, we also collect the true reward. We then compute the RoboCLIP reward for each video using VLM alignment be- tween the textual description of the task and the video. We visualize the correlations between the RoboCLIP and true rewards in the form of an n x matrix where entry (i.j) j) corresponds to the corre- lation between the true reward and the RoboCLIP reward generated for the hh task using the j"h text description. As one can see, for given task, the highest correlation in the matrix is for the correct textual description. We visualize one such simi- larity matrix in Figure 2 for Metaworld. We find that Metaworld seems to align well in the latent space of the model with more diagonal-heavy confusion matrix. The objects are all correctly identified.
Robot opening green deseett
Robot slnuing grean de dronect
Robot piskting red buttio
Rebot turning fassurt
0.49
0.45
0.42
0.33
0.07
0.10
-0.74
-084
0.69
0.73
0.54
0.41
-0.77
-0.71
0.81
0.84
Figure Domain Alignment Confusion Matrix. We perform a confusion matrix analysis on a subset of the data on collected on Metaworld [Yu et al., 2020] environments hy comparing the pair-wise similarities between the latent vectors of the strings describing the videos and those the videos. We find that Metaworld is well-aligned with higher scores along the diagonal than along the off- elements.
4.2 Language for Reward Generation
The most naturalistic way define a task is through natural language. We do this by generating sparse reward signal for the agent as described in Section 3: the reward for an episode is the similarity score between its encoded video and the encoded textual description of the expected behavior in the VLM's latent space. The reward is provided to the agent at the end of the episode. For RoboCLIP. GAIL, and AIRL, we first pretrain the agents online with their respective reward functions and then perform finetuning with the true task reward in the deployment environment. We perform this analysis on 3 Metaworld Environments: the textual descriptions, "robot closing green drawer", "robot closing black box", and "robot pushing red button" for each environment, respectively. Figure 3 plots returns on the target tasks while finetuning on the depoloyment environment after pretraining (with the exception of the Dense Task Reward baseline). Our method outperforms the imitation learning baselines with online exploration in terms of true task rewards in all environments. Additionally our baselines utilize the full state information the environment for reward generation where RoboCLIP uses only the pixels to infer state. RoboCLIP also achieves more than double zero-shot rewards in all environments - importantly,
Trawer-Close Door PooreClose and Button-Press. We use
5

---

XoboCCIP-Video
RoboCLIP-Text
FISH (Haldar 2023)
Zero Shot Rewards Batood
160
350
140
300
120
250
100
80
150
60
20
50
Dour-Close
Figure 3: Caaguage-Conditionen Reward Generation. The pretrained VLM is used generate rewards via the similarity score of the encoding of an episode of interaction an agent in its environment, z with the encoding of a task specifier z" specified in natural language. We use the strings, "robot closing black box", "robot closing green druwer" and "robot pushing red button" for conditioning for the environments respectively. We find that agents pretrained these sanguage-condttoone rewards outperform imitation learning baselines like GAIL [Ho Ermon, 2016] and AIRL [Fu et al., 2017].
Zero-Shot Rewards, /Episoee
RoboCLIP-Video
GAIL
AIRL
Dense Original
-165
-170
-20
-20
-175
-40
-40
-180
-185
-60
-60
-190
Hinge Cabinet
Slide Cabinet
Kettle
Figure 4: Using In-Domain Videos for Reward Generation. The pretrained VL.M used generate rewards via similarity score of encoding of an episode of interaction encoding of video demonstration of expert behavior in the same environment. The similarity score hetween the latent vectors is provided as reward to the agent and is used to train online RL methods. We study this setup in the Kettle, Hinge and Slide Tasks in the Franka Kitchen Environment [Gupta al., 2019]. We find that RoboCLIP reward are able learn to complete the task all three setups without any
agent in its environment,
policies trained need for external rewards using just a single in in-domain demonstration.
the MoboCLIP-trainees agent is able complete the tasks even before finetuning true task rewards.
4.3 In-Domain Videos for Reward Generation
Being able to use textual task descriptors for reward generation can only work in environments where there domain alignment between the pretrained model and the visual appearance of the environment. Additionally, VLMs are large models often with billions of parameters making computationally expensive to fine tune for domain alignment. The most naturalistic way to define a task in such setting is in the form single demonstration in the robotic environment which can be collected using teleoperation. We study how well this works in the Franka Kitchen [Gupta et al., 2019] environment. We consider access single demonstration per task whose video is used generate rewards for online RI.
Quantitative Results. We measure the zero-shot task reward, which increases as the task object (i.e., Kettle, Slide and Hinge Cabinets) gets closer to goal position. This reward does not depend on the position of the end-effector, making the tasks difficult. Figure 4 shows the baselines perform poorly as they generally do not interact with the target objects, while RoboCLIP able to solve the task using the reward generated using the video of single demonstration.
Qualitative Results. We find that RoboCLIP allows for mimicking the "style" of the source demonstration, with idiosyncrasies of motion from the source demonstration generally transferring to the policy generated. We find this to occur in the kitchen enviromment's Slide and Hinge task as seen in Figure 5. The first row of the subfigures in Figure are visualizations of the demonstration video used to condition the VLM that are trained with the generated rewards of RoboCI.I. As can be seen, the Slide demonstration consists of wide circular are of motion. This mimicked in the learned policy, although the agent misses the cabinet in the first swipe and readjusts to make contact with the handle.
reward generation. The bottom rows correspond to the policies

---

Slide Cabinet
Hinge Cabinet
Figure 5: Qualitative Inspection of Imitation. The first row in each subfigure shows the visualizations of the demonstration video used for reward generation via the VLM. The second rows are videos taken from policy recovered from training the RoboCLIP reward generated using the videns in the first rows. The quick swiping motion demonstrated the Slide demonstration is mimicked well in the resultant policy while the resultant learned policy.
wrist-rotational "trick-shot" behavior the demonstration Hinge appears
RoboCLIP-Text
CoboCLiP-Finetuee
FISH (Haldar al.
Zero Shot Rewards liplodd
1000
300
70
60
50
600
200
40
150
400
20
200
50
10
0
0
Faucet Open
Roach
Coffee Push
Figure Finetuning Harder Environments: harder environments, like Coffee- Push and Faucet-Open, we find that RoboCLIP rewards do not solve the task completely. We test whether provid- ing a single demonstration in the environment (using observations and actions) is enough to finetune this pretrained policy, setup identical to our baselines. Thus, we pre-train the RoboCIIIP reward from language and then finetune using single robotic demonstration. This improves performance 200%. See videos our website.
This effect is even more pronounced in the Hinge example where the source demonstration consists of twirling wrist-rotational behavior, which is subsequently imitated by the learned policy. The downstream policy misses the point of contact with the handle but instead uses the twirling motion to open the hinged cabinet in an unorthodox manner by pushing near the hinge. We posit that the VLMs used RoboCLIP contain a rich latent space encoding these various motions, and so even they cannot contain semantically meaningful latent vectors in the Franka Kitchen environments due to domain mismatch, they are still able to encode motion information allowing them to used for RoboCLIP with single demonstration video.
4.4 Out-of-Domain Videos for Reward Generation
Another natural way to define task is to demonstrate it yourself. To this end, try to demon- strations humans or animated characters acting in separate environments as task specification.
For this, we utilize animated videos of hand pushing red button and opening green drawer and real human video of opening fridge door (see Figure 7). The animated videos are collected from stock image repositories and the human video is collected using phone camera in our lab kitchen.

---

RoboCLIP-Videe
GAIL
AIRL
Dense Original
150
400
200
300
150
100
200
100
50
100
50
0
Press
Drawer Open
Open
Figure Using Out-of-Domain Videos for Reward Generation. Pretrained Aileo-and-Langueee Model is used to generate rewards via the similarity score of the encoding of an episode of interaction of an agent in its environment, z" with the encoding of task specifier z" in the form of video of a human or animated character demonstrating task in their environment. The similarity score hetween the latent vectors is provided reward the agent and used train online RL methods. The frames below the graphs illustrate the vidco used for reward generation.
Using the encodings of these video, we test out RoboCLIP the 3 corresponding Metaworld tasks Button-Press. Drawer-Oper Open and Door-Open, We follow the same setup as in Section 4.2 by first pretraining methods with their respective reward functions and then finetuning in the deployment environment with target task reward.
We compare the performance of the policy trained with these rewards to GAIL [Ho and Ermon, 2016]
RoboCLIP-Video
RoboCLIP-Text
RoboCL.PP-Edit
and AIRL [Fu et al., 2017] trained using the
Shot Rewards: Lowards/Episodee /Eisode
1200
same single expert demonstration as RoboCLIP on these rewards with state information. These methods are known to be Sata-hungry, requiring multiple demonstrations to train their reward func- tions. Consequently, they perform much worse than RoboCLIP, even with 2-3x worse zero-shot task performance, can be seen from Figure 7.
1000
800
600
Zero- Zero-Shot
400
200
4.5 Multimodal Task Specification
0
Using videos specify task description is pos- sible when either there is access to a robot for teleoperation as in Section 4.3 or human demonstrate a behavior in their own environment as in Section 4.4. When these are not the case, a viable alternative is to utilize multimodal demon- strations. For example, consider a scenario where the required task is to push drawer to close it, but only a demonstration for pushing a button is available. In this situation, being able to edit the video of the off-task demonstration is useful. This way, one can direct the agent to move its end-effectors to push the drawer instead of the button.
Figure 8: Multimodal Task Specification. We study whether video demonstrations of expert demonstra- tions can be used to define tasks. We use the latent embedding of video demonstration of a robot push- ing a button and subtract from it the embedding of the text "red button" and add to it the embedding of the text "green drawer". This modified latent is used to generate rewards in the Drawer-Clod We find that the policy trained using this modified vector outperforms string-only manipulation in the zero-shot setting.
environment.
We do this by algebraically modifying the encoding of the video demonstration:
z odited (push drawer) z video video (push button) fext (button) + f (drawer)
where editee (push drawer) is the vector used to generate rewards in the Drawer- Close environ- ment, z video (push button) is the vector the encoding of the video of the robot pushing a button, (button) is the encoding of the string button and 2 fee! feet (drawer) is the encoding of the string drawer. As can be seen in Figure 8, defining rewards such multimodal manner results in a higher sero-shot score than the dense task reward and also pretraining the string-only task reward.

---

4.6 Finetuning
In harder environments, and with rewards from OOD videos and language. the robot policy sometimes approaches the target object, but fails to complete the task. Thus, we tested whether providing single demonstration (using observations and actions) was enough to finetune this pretrained policy.
Thus, for this experiment we first (1) pretrain on the RoboCLIP reward from human videos or language descriptions and then (2) finetune using single demonstration. As seen in Figure 6, we find that this converts each of the partially successful policies into complete success and improves the rewards attained by the policies by 200%. This fine-tuning setup is especially useful in harder tasks like like Coffee Push and Faucet- and is competitive with state-of- oftt--of-the-art approaches like FISH [Haldar et al., 2023].
4.7 Ablations
Finally, we investigate the effects of various design decisions in RoboCLIP. First, we study the effect of additional video demonstrations on agent performance. We also examine the necessity of using pre-trained VLM. Recent works like RE3 [Seo et al., 2021, Ulyanov et al., 2018] have shown that randomly initialized networks often contain useful image priors and can be used to supply rewards to agents to encourage exploration. Therefore, we test whether a randomly initialized S3D VLM can supply useful pretraining rewards in the in-domain video demonstration setup as in Section 4.3. Finally, we study our choice of pre-trained VLM. We examine whether a pretrained CLIP [Radford et al., 2021], which encodes single images instead of videos and was trained from S3D, can be used to generate rewards for task completion. In this setup, we record the last image in episode of interaction of the agent in its environment and feed it to CLIP trained on ImageNet [Russakovsky et al., 2015] (i.e., not trained on videos). We then specify the task in natural language and use the similarity between the embeddings of the textual description of the task and the final image in the episode to generate a reward that is fed to the agent for online RI.
different dataset
As seen in Figure 9, using single video demonstration provides the best signal for pre- training. We posit that our method performs worse when conditioned on multiple demon- strations as the linear blending of multiple video embeddings, which is used due to the scalar product, does not necessarily correspond to the embedding of a successful trajectory. Crucially, we also find that using the static image version of CLIP does not provide any useful sig- nal for pretraining. The zero-shot performance is very poor, which we posit is because it does
demo
NX demo
4 demo
Untrained VLM
Static CLIP
not contain any information about the dynam-
600
ics of motion and task completion although it contains semantic meaning about objects in the frame. On the other hand, video contrastive learning approaches do contain this information. This is further evidenced by the fact that in- spite of poor domain aligmment between Franka Kitchen and the VLM, we find that encodings of in-domain video demonstrations are still good for providing a pretraining reward signal to the agent.
500
400
300
200
100
0
Figure 9: Ablations. We study the effects varying the number of demonstrations provided to the agent can have on downstream rewards. We also study the effects of the training provided the VLM on the downstream rewards. Finally, we study whether using CLIP trained on static images provides good rewards for pretraining.
5 Conclusion
Summary. We studied how to distill knowl- edge contained in large pretrained Video-and- Language-Models into online RI. agents by us- ing them to generate rewards. We showed that our method, RoboCLIP, can train robot policies using a single video demonstration or textual descrip- tion of the task, depending on how well the domain aligns with the VLM. We further investigated alternative ways to use RoboCLIP, such as using out-of-domaal videos or multimodal demonstrations. Our results showed RoboCLIP outperforms the baselines in various robotic environments.
Limitations and Broader Impact. Since we are using VLMs, the implicit biases within these large models could percolate into RL agents. Addressing such challenges is necessary, especially since it is

---

unclear what the form of biases in RL agents might look like. Currently, our method also faces the challenge of stable finetuning. We find that in some situations, finetuning on downstream task reward results in instabilities as seen in the language conditioned reward curve in Figure 8. This instability is potentially due to the scale of rewards provided to the agent. Rewards from the VLM are fairly low in absolute value and subsequently, the normalized Q-values in PPO policies are out-of-shape when finetuned on task rewards. In our experiments, this is not a big problem since the RoboCLIP reward is already sufficient to produce policies that complete tasks without any deployment environment finetuning, but this will be essential to solve when deploying this for longer horizon tasks.
Another limitation of our work is that there is no fixed length of pretraining. Our current method involves pretraining for fixed number of steps and then picking the best model according to the true task reward. This is of course difficult when deploying RoboCLIP in real-world setup as a true reward function is unavailable and human must monitor the progress of the agent. We leave this for future work.
Acknowledgments and Disclosure of Funding
This work was supported by DARPA (HRO01 12190134). C-BRIC (one of six centers in JUMP, Semiconductor Research Corporation (SRC) program sponsored by DARPA), the Army Research Office (W911NF2020052 and the Office of Naval Research (ONR) (#N00014-21-1-22)
and 1100014-11-1-2633 The authors affirm that the views expressed herein are solely their own, and do not represent the views of the United States government or any agency thereof. Thanks to Karol Hausman, Sergey Levine, Ofir Nachum, Kuang-Huet Lee and many others at Google DeepMind for helpful discussions.
References
Picter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-Firs. International Conference on Machine Learning, ICML '04, page I, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581 138385. doi: 10.1145/101533..U0 10151000......0
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: 06565, 2016.
Erdem Biyik, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach active reward learning. In Proceedings of the 3rd Conference on Robot Learning (CoRL), 2019.
Erdem Biyik, Nicolas Huynh, Mykel J. Kochenderfer, and Dorsa Sadigh. Active pefeeence-basee gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020. doi: 10.
Erdem Bıyik, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45-67 67, 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Decksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Van- houcke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Roboties transformer for real-world control at scale. In arXiv preprint arXiv:22/2.068877 2022.
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference
10

---

on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 783-792, Long Beach, California, USA, 09-15 Jun 2019. 2019.PMLR.
Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from "in-the-wild" human videos, 2021.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv: 1910. 1215, 2019.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danillidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross cross-domain datasets. arXiv preprint arXiv:2109.1s. 13396, 2021.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML'16, 16, page 49-58. JMLR.org, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprini arXiv: 11248, 2017.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Infor- mation Processing Systems, volume 31. Curran Associates, Inc., 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Yuying Ge, Annabella Macaluso, Li Erran Li, Ping Luo, and Xiaolong Wang. Policy adaptation from foundation model feedback, 2023.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-190123 19012, 2022.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 1910:1910..1156: 2019.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXivv2303.01499, 2023.
Joey Hejna and Dorsa Sadigh. Few-shot preference learning for human-in-the-let rl. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordina- tion. In 40th International Conference Machine Learning (GML),2023
11

---

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Joey Hong. Anikait Singh, and Sergey Levine. When should we prefer offiine reinforcement learning over behavioral cloning? arXiv preprint urXiv:2204.056// 2022.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In International Conference on Learning Representations (ICLR), 2023.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback- -fficiett interactive reinforcement learning via relabeling experience and unsupervised pre-training. 2021.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time, 2022.
Antoine Miech, Dimitri Zhukov, Jean- Sean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640, 2019.
Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. In 5th Annual Conference on Robot Learning, 2021.
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning. ICML '00, page 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xi- ang, and Haibin Ling, Expanding language-image pretrained models for general video recognition. In European Conference on Computer Vision (ECCV), 2022.
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. arXiv preprint arXiv:2301.12055 2023.
Dean A Pomerleau. Alvinn: An autonomous land vehicle a neural network. Advances in neural information processing systems, 1, 1988.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), July 2017. doi: 10.15607/RSS 001.0.0
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.00344, 2017.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pages 9443- 9454. PMLR, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lemptskky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446-9454, 2018.
12

---

Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: apeed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV), pages 305-321, 2018
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning. pages 1094-11000 PMLR, 2020.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence- Volume 3, AAAI08, page 1433-1438. AAAI Press, 2008 ISBN 9781577353683.
13


Reviews:
[{"review_id": "f4dGzHwal8", "reviewer": "Reviewer_QrrK", "summary": "This paper proposes using the similarity between two embedded spaces, one from real interaction and the other from visual/textual interaction, to develop the rewards.", "questions": "1) Can authors comment why the paper is more suitable to the NeurIPS readership than CoRL, RSS, ICRA, etc. readership? \n2) Can the authors comment why the simple sparse reward of the two embeddings would work? Given that the video demonstrations are not anywhere closer to the real robot interaction, this reviewer is having a hard time understanding how the model works. \n3) Looking at Fig 2, it looks like the model is not great at distinguishing \"opening\" from \"closing.\" Why has the model not learned that concept?\n4) What is the cause of variability of rewards in Fig 3? Is it mainly due to how close the video to the real or some other reason? Since the video embedding seems to be significantly more useful, what does text serve? Are the authors trying to convey that even using text (which is easier) is fine if we don't have a video.", "limitations": "Adequate ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "It's a well-written paper. The idea is simple and clearly stated. The proposed method has been sufficiently evaluated on realistic simulation settings. ", "weaknesses": "It is possible to make the paper more relevant to machine learning as it looks like a robotics paper. The intuition/rationale behind the simple sparse reward structure is not properly explained.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review! We are glad you would like to see our work published! :)\nWe address your questions below: \n1. **“Why submit to NeurIPS over robot conferences?”**:  Thanks for your suggestion! Sequential decision-making is a generally useful problem to the community and recent advances in machine learning borrow heavily from robotics literature. Consider, for instance, reinforcement learning from human feedback (RLHF), a method that is utilized to finetune most of today’s large language models (LLMs). RLHF [1] when finetuning LLMs utilizes PPO [2], both of which were primarily robotics works now applicable to LLMs. Similarly, we believe that our work could inspire related fields to utilize our approach (e.g., the policy in our work could be replaced by an LLM and the pre-trained VLM could be a hate speech detector and the LLM could be finetuned to be prevented from generating hate speech). Furthermore, robotics works like ours [4, 5] are commonly published at ICML, ICLR, NeurIPS, and related conferences.\n\n2. **“Why does the sparse VLM embedding reward work?\"**: The VLM embedding reward encourages the agent to produce trajectories that align semantically with the given demonstration. The VLM is able to do this due to its pre-training objective (embedding videos with similar captions close to each other). Due to this semantic alignment, it will encode successful agent trajectories of say, opening a door, close to the video demo of opening a door or to textual description *opening door*. This distance in the embedding space very naturally translates as a reward.\n\n3. **“Fig 2: VLM does not distinguish opening and closing in confusion matrix?”**: The alignment metric for the confusion matrix in the current version was calculated by aligning a single demonstration video’s embedding with that of a language prompt. Thinking about the reviewer comments, we noted how this is a noisy estimate of alignment due to the small sample size (of 1) and that this metric does not align with task progress.\nTherefore, we calculate a **new confusion matrix** by calculating the correlation between true and RoboCLIP reward over 10 agent trajectories. We found that this metric actually demonstrates alignment: the resultant matrix is diagonal-heavy (see below). This demonstrates that for each of the metaworld environments, the RoboCLIP reward does indeed correlate with the true reward. \n\\begin{array} {|r|r|} \\text{Video || Text} & \\text{Robot Opening Green Drawer} & \\text{Robot Closing Green Drawer} & \\text{Robot Pushing Red Button} & \\text{Robot Turning Handle}\\\\\\  \\text{Open drawer videos} & \\textbf{0.49} & 0.45 & 0.42 & 0.33 \\\\\\  \\text{Drawer Close videos} & 0.07 & \\textbf{0.1} & -0.74 & -0.84 \\\\\\  \\text{Button Press Videos} & -0.6925 & -0.7309\t & \\textbf{0.5351} & -0.41 \\\\\\  \\text{Faucet Videos} & \t-0.77 & -0.71\t & 0.81 & \\textbf{0.84} \\\\   \\end{array}\n\n4. **“Is the variance of rewards between Fig3 due to domain differences?”**:Yes, as the reviewer has predicted, domain alignment is the main cause! The video-conditioned reward is obtained from the same modality as the environment interactions, therefore resulting in higher rewards. \n\n5. **“Why use text embeddings if video embeddings work better?”**:  Yes! The point of that experiment is to show that if a video demonstration cannot be collected, using a text description of the task can still enable agent learning. This is crucial in applications where in-domain demonstrations are unavailable - for instance, teaching a robot to perform tasks in a location inaccessible to humans.\n\n[1] Deep Reinforcement Learning from Human Preference. Christiano et al. 2017.\n\n[2] Proximal Policy Optimization Algorithms Schulman et al. 2017.\n\n[3] GPT-4 Technical Report\n\n[4] Dynamics-Aware Unsupervised Discovery of Skills, Sharma et al, ICLR 2020.\n\n[5] Data-Efficient Hierarchical Reinforcement Learning, Nachum et al, NeurIPS 2018.\n"}, {"review_id": "IATBoGIHnE", "reviewer": "Reviewer_Dcaj", "summary": "This paper presents a novel approach to leverage pretrained Vision-Language Models (VLMs) for training Reinforcement Learning (RL) agents by utilizing them to generate rewards. Specifically, the authors employ S3D, pretrained on HowTo100M, as the backbone and measure the dissimilarity between online agent experience and task demonstrations to serve as the reward signal for training task-specific RL policies. They demonstrate the effectiveness of their proposed method on the Metaworld Environment and the Franka Kitchen Environment.", "questions": "1. What is RoboCLIP-Video in Section 4.2?\n2. In Section 4.6, how did you use 4 demos to generate one reward for training?", "limitations": "I appreciate that the author acknowledges the limitation of unstable fine-tuning in their work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper addresses an important and widely recognized problem, which is the grounding of VLMs.\n2. The idea behind the proposed method is straightforward and intuitive.\n3. The author thoroughly examines the performance of the proposed approach from six different perspectives: alignment, text-to-reward mapping, video-to-reward mapping, generalization to out-of-domain videos, decompositional generalizability, and an ablation study.", "weaknesses": "I have some major concerns about the experiment section of this paper. There are several results that are not convincing and can be quite confusing. \n\n1. The domain alignment between the demonstration and text description, as mentioned by the author, is indeed unsatisfactory. For instance, the policy should learn to close the green drawer instead of opening it when the task descriptor is \"open green drawer.\" Using this misleading reward can lead to misleading learning outcomes.\n\n2. In the language-to-reward section (Section 4.2), the author only demonstrates the performance on three \"easy\" tasks, which is not sufficiently convincing. Additionally, it is unclear why the proposed method outperforms dense task rewards. The dense task reward can be considered an oracle, so it raises questions about whether the performance improvement is due to insufficient training epochs or simply because the rewards themselves are inadequate.\n\n3. The qualitative results presented in Section 4.3 are challenging to comprehend. It is difficult to understand how the robot can open the cabinet without even touching it. While I can imagine what happened in between, the visualizations provided in the paper do not offer enough information to support a clear understanding of the process.\n\n4. Regarding multimodal task specification, it is puzzling why RoboCLIP-video (with a different object) yields the best performance instead of RoboCLIP-Edit. One possible interpretation is that the robot is primarily learning push-like behaviors from a video demonstration. However, this raises concerns about the overall effectiveness and usefulness of the edit method in achieving the intended goals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments! Addressing them has made this work much better! :)\nWe address each of your concerns below:\n1. **“VLM confuses open and close, the domain alignment is not good?”**: The alignment metric for the confusion matrix was calculated by aligning a single demonstration video’s embedding with that of a language prompt. This is a noisy estimate due to the small sample size and this metric does not account for task progress.\nTherefore, we calculate a **new confusion matrix** by calculating the correlation between true and RoboCLIP reward over 10 agent trajectories. We found that this metric actually demonstrates alignment: the resultant matrix is diagonal-heavy (see below). This demonstrates that for each of the metaworld environments, the RoboCLIP reward does indeed correlate with the true reward. \n\\begin{array} {|r|r|} \\text{Video \\\\\\ Text} & \\text{Robot Opening Green Drawer} & \\text{Robot Closing Green Drawer} & \\text{Robot Pushing Red Button} & \\text{Robot Turning Handle}\\\\\\  \\text{Open drawer videos} & \\textbf{0.49} & 0.45 & 0.42 & 0.33 \\\\\\  \\text{Drawer Close videos} & 0.07 & \\textbf{0.1} & -0.74 & -0.84 \\\\\\  \\text{Button Press Videos} & -0.6925 & -0.7309\t & \\textbf{0.5351} & -0.41 \\\\\\  \\text{Faucet Videos} & \t-0.77 & -0.71\t & 0.81 & \\textbf{0.84} \\\\   \\end{array}\n2. **“Not enough tasks evaluated in Sec 4.2?”** : Thanks for your suggestion! We have **added 3 visually distinct environments** in metaworld as additional experimental domains for this rebuttal  - (1) Coffee-push which involves the picking and placing of a very small coffee mug. The coffee mug is about half the size of the robot end-effectors – manipulating this object requires significant precision (please see representative videos on the website: “Finetuning with One Demonstration Section”). The other environments are (2) Faucet, which requires the robot to turn a thin handle and (3) Reach, where the agent must reach a tiny dot in space. (1) and (2) are significantly more challenging than the existing environments. \nWe also run 2 more baselines (FISH [1] and Behavior Cloning) on these new environments. In more complex environments like (1), FISH fails to learn successful policies while RoboCLIP performs well–outperforming FISH by 3x. Summary of results: \n\\begin{array} {|r|r|} \\text{Task} & \\text{RoboCLIP-Finetuned} & \\text{FISH} & \\text{BC} \\\\\\  \\text{Faucet Open} & 859.9849646 & 889.4014539 & 896.2618905 \\\\\\  \\text{Coffee Push} & \\textbf{59.00943929} & 7.099759383 & 6.183762854 \\\\\\  \\text{Reach} & \\textbf{214.0404179} & 70.05093234 & 69.42764688 \\\\   \\end{array}\n3. **“Why does the method outperform the dense task reward baseline? (Sec 4.2)”**: To clarify, the dense task reward corresponds to the reward produced by an untrained agent. This baseline allows us to compare the performance to an agent that trains on the task for the same amount of time without any prior training. This baseline indicates that the task is not trivial and that the pretraining is crucial. Thanks for pointing out this confusion, we will clarify the description in Section 4. \n4. **“Qualitative results in Sec 4.3 hard to understand? (Fig 5)”**: To clarify the qualitative results in Section 4.3, here is what we will add to the text: The first row of each of the subfigures is a  visualization of the demonstration video used to condition the VLM for reward generation. The bottom rows correspond to the policies that are obtained by training with the resultant RoboCLIP reward obtained from the above demonstration. As can be seen, the Slide demonstration consists of a wide circular arc of motion. This is mimicked in the learned policy, although the agent misses the cabinet in the first swipe and readjusts to make contact with the handle. This effect is even more pronounced in the Hinge example where the source demonstration consists of twirling wrist-rotational behavior, which is subsequently imitated by the learned policy. The downstream policy misses the point of contact with the handle but instead uses the twirling motion to open the hinged cabinet in an unorthodox manner by pushing near the hinge. We also have gifs demonstrating this exact visualization on our website under the “Style Transfer” section. The link to the site can be found in the abstract.\n5. **“Why does RoboCLIP-Video, using a different object, perform better than RoboCLIP-Edit?”** To clarify: the RoboCLIP video datapoint here is with the target object, in this case, the drawer and this datapoint is meant to serve as a “skyline”. The point of this experiment is to demonstrate that editing can also allow one to reach the performance obtained when using a “true” demonstration. We will make this clearer in Section 4.5.\n\n[1] Haldar, Siddhant, et al. \"Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations.\" arXiv preprint arXiv:2303.01497 (2023).\n\n\n**QUESTIONS:**\n1. **“What is RoboCLIP-Video?”**\nRoboCLIP video in section 4.2 corresponds to the policy obtained by training on the video reward generated using an in-domain video demonstration within metaworld. We will clarify this in the text in Section 4.2.\n2. **“Sec 4.6: How did we use 4 demos to generate 1 reward for training?”**\nWe used 4 demonstrations to generate the reward by encoding each of the demonstrations into the latent space. Then we calculated the similarity between the video of an episode and each of the 4 demonstrations and calculate the reward as a sum of each of the 4 similarity scores. We will update Sec 4.6 to explicitly state the above.\n\n"}, {"review_id": "zIrmK5Zh4I", "reviewer": "Reviewer_S3ve", "summary": "RoboCLIP is an innovative online imitation learning method that addresses the challenging problem of reward specification in reinforcement learning. Unlike traditional approaches that rely on expert supervision and extensive data, RoboCLIP utilizes a single demonstration, either in the form of a video or a textual description, to generate rewards without the need for manual reward function design. It can even leverage out-of-domain demonstrations, eliminating the requirement for demonstrations from the same domain as deployment. \n\nBasically the method compares the embedding between the video generated by the policy and video given to it for imitation to get a reward for training an RL agent. The idea is simple and intuitive. ", "questions": "I don't have any questions for the authors.", "limitations": "limitations are discussed adequately. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Although the idea is simple, the authors verify through experimentation that learning policies from videos is possible. \nThe results are great and contribute to advance in robot policy learning from videos. ", "weaknesses": "Out-of-distribution results are underwhelming. Maybe a future direction could to focus more on the object state to train better policies.\n\nNo real robot results :( \nThe whole point of something like this should be easy applicability to the real world. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments! We're glad that you would like to see our work published!\nWe address your concerns below: \n1. **“Out-of-distribution results are underwhelming? (Sec 4.4)”**: Indeed, especially with rewards from OOD videos and language, the robot policy often approaches the target object, but fails to complete the task. Thus we now added an **additional experiment**: We tested whether providing a single demonstration, the same demonstration used to train the baselines (using observations and actions), was enough to finetune this pretrained policy. Thus, for this experiment we first (1) pretrain on the RoboCLIP reward from human videos or language descriptions and then (2) finetune using a single demonstration. Through this experiment, RoboCLIP policies successfully complete the tasks **across all setups** resulting in ~3x higher rewards in 2 out of 3 environments and matching the performance on 1.  **This setup solves all of the failure policies seen previously.** Please see additional videos of this experiment on the website (“Finetuning with One Demo”) and Fig 10 in the rebuttal document. We find that this setup results in overwhelming evidence that, with RoboCLIP, one demonstration is enough to learn a policy. Summary:\n\\begin{array} {|r|r|} \\text{Task} & \\text{RoboCLIP-Finetuned} & \\text{FISH} & \\text{BC} \\\\\\  \\text{Faucet Open} & 859.9849646 & 889.4014539 & 896.2618905 \\\\\\  \\text{Coffee Push} & \\textbf{59.00943929} & 7.099759383 & 6.183762854 \\\\\\  \\text{Reach} & \\textbf{214.0404179} & 70.05093234 & 69.42764688 \\\\   \\end{array}\n2. **“No real robot results?\"**:We agree that real robot results would be exciting. We avoided training on real robots for safety reasons due to unstable behaviors that could arise when pre-training the policy on a real robot with VLM rewards. However the aforementioned single demonstration fine-tuning experiment highlights the possibility of: first pre-training with RoboCLIP in simulation, then finetuning on the real robot using **robot demonstrations**. We leave investigating this to follow-up work.\n\n\n"}, {"review_id": "LX2EfHnWqh", "reviewer": "Reviewer_EmDU", "summary": "This study seeks to address the problem of data efficiency in imitation learning, aiming to train a robot to perform manipulation tasks after only one demonstration. In order to accomplish this, the research proposes using a pre-trained visual language model (VLM) to encode the demonstration video or task description text from the expert. The use of the dot product between the learnt and demonstrated embeddings would then serve as a reward function. Experiments are conducted in Metaworld and the Franka simulation, with tasks including \"closing a door,\" \"closing a drawer,\" and \"pressing a button.\"", "questions": "Q1: In Figure 4, what is the reason behind GAIL, AIRL, and a method called \"Task Reward\" reflecting an identical reward across all three tasks?", "limitations": "See Weaknesses and Questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of utilizing a pretrained visual language model to encode expert demonstrations or text descriptions as a reward function is neat and straightforward.\n\n- This work conducts an extensive amount of robot experiments in simulation including\"closing a door\", \"pressing a button\", \"Slide Cabinet \" and so on.", "weaknesses": "- The use of dot product between expert embeddings and learned policy embeddings as the objective reward function is not adequately explained. The rationale behind this decision and its effectiveness would benefit from further elaboration.\n\n- The Domain Alignment Confusion Matrix reveals a poor similarity alignment between the text description and the video from the robot camera. This implies a potential issue, where goal descriptions in text form may not correspond accurately to the video data. \n\n\n- Moreover, the experimental performance demonstrated in the accompanying video is weak, with numerous instances of failure. \n\n\n- The selected baseline is outdated. GAIL was introduced in 2016 and AIRL was in 2017.\n\n- The proposed method fails to improve and instead performs significantly worse when presented with an increased number of expert demonstrations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review! Addressing your comments has improved the paper significantly! :)\nWe address your concerns below: \n1. **“Why use dot products between embeddings as the reward function?”**: We will add the following to the text: VLMs such as VideoCLIP and S3D are pretrained using a cosine similarity loss that aligns encodings of different modalities such as language and videos. This is done by encoding both modalities into latent spaces of the same dimensions and calculating a dot product between the vectors of each modality. The resultant dot product (a scalar) measures the similarity between the embeddings, and the model is trained to increase similarity between paired videos and language labels.\nTherefore, this dot product naturally serves as a reward function for an agent: we compute the dot product between embeddings of agent videos and the given task specification.\n2. **“Outdated Baselines?”**: The reviewer is right that there has been considerable work on learning from demonstrations since the publication of our baselines. We have therefore added 2 more baselines (FISH [1], **published ~15 days ago at RSS 2023**, and Behavior Cloning), the most effective of which is FISH. FISH demonstrates how it is possible to learn robot policies using 1 min of demo data - thereby also operating in the same low data regime. However, we find that in our extremely low data regime of **one demonstration**, RoboCLIP outperforms even FISH by 3x in 2 out of 3 environments, and matches its performance in 1 (see Fig 10 in the rebuttal doc). This comparison highlights RoboCLIP’s performance, even compared to the latest SOTA learning from demonstration baselines. Summary:\n\\begin{array} {|r|r|} \\text{Task} & \\text{RoboCLIP-V} & \\text{RoboCLIP-T} & \\text{FISH} & \\text{BC} \\\\\\  \\text{Door Close} & \\textbf{314.8915735} & 24.08013125 & 312.7388271 & 314.8853879 \\\\\\  \\text{Drawer Close} & \\textbf{922.5} & 136.25 & 475.2357946 & 127.44404 \\\\\\  \\text{Button Press} & 125.6114189 & \\textbf{131.365798} & 29.8517859 & 79.35955141 \\\\   \\end{array}\n3. **“Numerous instances of failure in the accompanying video?”**: We agree, thanks for pointing this out! Indeed, especially with rewards from OOD videos and language, the robot policy often approaches the target object, but fails to complete the task. Thus we now added an **additional experiment**: We tested whether providing a single demonstration, the same demonstration used to train the baselines (using observations and actions), was enough to finetune this pretrained policy. Thus, for this experiment we first (1) pretrain on the RoboCLIP reward from human videos or language descriptions and then (2) finetune using a single demonstration. \nThrough this experiment, **RoboCLIP policies successfully complete the tasks across all 7 Metaworld setups.** In the new challenging environments, it results in ~3x higher rewards in 2 out of 3 environments and matching the performance on 1. **This setup solves all of the failure policies seen previously.** Please see additional videos of this experiment on the website (“Finetuning with One Demo” Section) and Fig 10 in the rebuttal document. We find that this setup provides overwhelming evidence that, **with RoboCLIP, one demonstration is enough to learn a policy**. Summary of new environments:\n\\begin{array} {|r|r|} \\text{Task} & \\text{RoboCLIP-Finetuned} & \\text{FISH} & \\text{BC} \\\\\\  \\text{Faucet Open} & 859.9849646 & 889.4014539 & 896.2618905 \\\\\\  \\text{Coffee Push} & \\textbf{59.00943929} & 7.099759383 & 6.183762854 \\\\\\  \\text{Reach} & \\textbf{214.0404179} & 70.05093234 & 69.42764688 \\\\   \\end{array}\n\n4. **\"RoboCLIP performs worse with additional expert demos? (Sec 4.6)?\":**\nWe added an **additional experiment** using 2 demonstrations, finding that it actually outperforms using either a single demonstration or 4 demonstrations. However, using 4 demonstrations does indeed reduce performance. We believe that this is because the convex combination of multiple demonstrations may not result in a successful agent trajectory, thus this could be why using 4 demonstrations results in worse performance. We will add this conjecture and experiment to Section 4.6. Summary:\n\\begin{array} {|r|r|} \\text{RoboCLIP-V 1 Demo} & \\text{RoboCLIP-V 2 Demos} & \\text{RoboCLIP-V 4 Demos} \\\\\\  125.6114189 & \\textbf{421.5395214} & 51.34670371 \\\\   \\end{array}\n5. **“Domain alignment confusion matrix shows poor similarity between certain text descriptions and videos?”**: The alignment metric for the confusion matrix was calculated by aligning a single demonstration video’s embedding with that of a language prompt. This is a noisy estimate due to the small sample size and this metric does not align with task progress.\nTherefore, we calculate a **new confusion matrix** by calculating the correlation between true and RoboCLIP reward over 10 agent trajectories. We found that this metric actually demonstrates alignment: the resultant matrix is diagonal-heavy (see below). This demonstrates that for each of the metaworld environments, the RoboCLIP reward does indeed correlate with the true reward.\n\\begin{array} {|r|r|} \\text{Video || Text} & \\text{Robot Opening Green Drawer} & \\text{Robot Closing Green Drawer} & \\text{Robot Pushing Red Button} & \\text{Robot Turning Handle}\\\\\\  \\text{Open drawer videos} & \\textbf{0.49} & 0.45 & 0.42 & 0.33 \\\\\\  \\text{Drawer Close videos} & 0.07 & \\textbf{0.1} & -0.74 & -0.84 \\\\\\  \\text{Button Press Videos} & -0.6925 & -0.7309\t & \\textbf{0.5351} & -0.41 \\\\\\  \\text{Faucet Videos} & \t-0.77 & -0.71\t & 0.81 & \\textbf{0.84} \\\\   \\end{array}\n\n\n[1] Haldar, Siddhant, et al. \"Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations.\" arXiv preprint arXiv:2303.01497 (2023)\n\n\n\n\n"}]

Author rebuttal:
We would like to thank the reviewers for their insightful comments and suggestions. They have helped improve the paper significantly and we are grateful for their time and effort :). We have run 4 additional experiments to address their concerns. We summarize the findings below. For details, please see the rebuttal doc. 
1. **3 New Environments**: We have added 3 new environments: (1) Coffee-push which involves the picking and placing of a very small coffee mug. The coffee mug is about half the size of the robot end-effectors – manipulating this object requires significant precision (please see representative videos on the website: “Finetuning with One Demonstration Section”). The other environments are (2) Faucet, which requires the robot to turn a thin handle and (3) Reach, where the agent must reach a tiny dot in space. (1) and (2) are significantly more challenging than the existing environments. 
2. **Additional Baselines:** We have added 2 more baselines (FISH [1], published **~15 days ago** at RSS 2023, and Behavior Cloning), the most effective of which is FISH. FISH demonstrates how it is possible to learn robot policies using 1 min of demo data - thereby also operating in the same low data regime. However, we find that in our extremely low data regime of **one demonstration**, RoboCLIP outperforms even FISH by 3x in 2 out of 3 environments, and matches its performance in 1 (see Fig 10 in the rebuttal doc). An abridged version of the results are below:
\begin{array} {|r|r|} \text{Task} & \text{RoboCLIP-V} & \text{RoboCLIP-T} & \text{FISH} & \text{BC} \\\  \text{Door Close} & \textbf{314.8915735} & 24.08013125 & 312.7388271 & 314.8853879 \\\  \text{Drawer Close} & \textbf{922.5} & 136.25 & 475.2357946 & 127.44404 \\\  \text{Button Press} & 125.6114189 & \textbf{131.365798} & 29.8517859 & 79.35955141 \\   \end{array}
3.  **Additional Finetuning Experiment** With rewards from OOD videos and language, the robot policy often approaches the target object, but fails to complete the task. Thus we now added an additional experiment: We tested whether providing a single demonstration, the same demonstration used to train the baselines (using observations and actions), was enough to finetune this pretrained policy. Thus, for this experiment we first (1) pretrain on the RoboCLIP reward from human videos or language descriptions and then (2) finetune using a single demonstration. 
Through this experiment, RoboCLIP policies **successfully complete the tasks across all 7 Metaworld setups converting all previous failure policies into successes**. In the new challenging environments, it results in ~3x higher rewards in 2 out of 3 environments and matching the performance on 1. See additional videos of this experiment on the website (“Finetuning with One Demo”) and Fig 10 in the rebuttal document. We find that this setup provides overwhelming evidence that, **with RoboCLIP, one demonstration is enough to learn a policy.** Summary below:
\begin{array} {|r|r|} \text{Task} & \text{RoboCLIP-Finetuned} & \text{FISH} & \text{BC} \\\  \text{Faucet Open} & 859.9849646 & 889.4014539 & 896.2618905 \\\  \text{Coffee Push} & \textbf{59.00943929} & 7.099759383 & 6.183762854 \\\  \text{Reach} & \textbf{214.0404179} & 70.05093234 & 69.42764688 \\   \end{array}
4. **Improved Domain Alignment Confusion Matrix Calculation:** In the current version of the paper, the domain alignment matrix was calculated by encoding a video of a demonstration of a target task and its language description and calculating the similarity score between them using a pre-trained VLM. This metric: (1) was noisy: due to a small sample size of 1, (2) Did not capture a notion of task completion. Thus, we calculated a new domain alignment matrix by calculating the correlation between true and RoboCLIP reward over 10 agent trajectories. This resolves both (1) and (2). We find that this metric indicates alignment and does indeed demonstrate the utility of the RoboCLIP alignment score as a reward. Summary below:
\begin{array} {|r|r|} \text{Video || Text} & \text{Robot Opening Green Drawer} & \text{Robot Closing Green Drawer} & \text{Robot Pushing Red Button} & \text{Robot Turning Handle}\\\  \text{Open drawer videos} & \textbf{0.49} & 0.45 & 0.42 & 0.33 \\\  \text{Drawer Close videos} & 0.07 & \textbf{0.1} & -0.74 & -0.84 \\\  \text{Button Press Videos} & -0.6925 & -0.7309	 & \textbf{0.5351} & -0.41 \\\  \text{Faucet Videos} & 	-0.77 & -0.71	 & 0.81 & \textbf{0.84} \\   \end{array}


Thank you once again for your thoughtful reviews! The rebuttal doc is attached below. 

Meta-review: The paper proposes a valuable video demonstration-based approach to reward learning for robotic manipulation scenarios, and adjustments to this work made during the rebuttal stage have appreciably improved the submission's empirical evaluation. Thus, this work is recommended for acceptance.
Decision: Accept (poster)