SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sumedh A Sontakke<sup>1‡</sup>, Jesse Zhang<sup>1</sup>, Sébastien M. R. Arnold<sup>4</sup>, Karl Pertsch<sup>2,3</sup>, Erdem Býík<sup>1,2</sup>, Dorsa Sadigh<sup>3</sup>, Chelsea Finn<sup>3</sup>, and Laurent Itti<sup>1</sup>
<sup>1</sup>Thomas Lord Department of Computer Science, University of Southern California
$^{2}$ University of California, Berkeley
3Stanford University
4Google Research
Reward specification is a notoriously difficult problem in reinforcement learning, requiring extensive expert supervision to design robust reward functions. Imitation learning (IL) methods attempt to circumvent these problems by utilizing expert demonstrations but typically require a large number of in-domain expert demonstrations. Inspired by advances in the field of Video-and-Language Models (VLMs), we present RoboCLIP, an online imitation learning method that uses a single demonstration (overcoming the large data requirement) in the form of a video demonstration or a textual description of the task to generate rewards without manual reward function design. Additionally, RoboCLIP can also utilize out-of-domain demonstrations, like videos of humans solving the task for reward generation, circumventing the need to have the same demonstration and deployment domains. RoboCLIP utilizes pretrained VLMs without any finetuning for reward generation. Reinforcement learning agents trained with RoboCLIP rewards demonstrate 2-3 times higher zero-shot performance than competing imitation learning methods on downstream robot manipulation tasks, doing so using only one video/text demonstration. Visit our website for experiment videos.
Sequential decision-making problems typically require significant human supervision and data. In the context of online reinforcement learning [Sutton and Barto, 2018], this manifests in the design of good reward functions that map transitions to scalar rewards [Amodei et al., 2016, Hadfield-Menell et al., 2017]. Extant approaches to manual reward function definition are not very principled and defining rewards for complex long-horizon problems is often an art requiring significant human expertise. Additionally, evaluating reward functions often requires knowledge of the true state of the environment. For example, imagine a simple scenario where the agent must learn to lift an object off the ground. Here, a reward useful for task success would be proportional to the height of the object from the ground — a quantity non-trivial to obtain without full state information. Thus, significant effort has been expounded in developing methods that can learn reward functions either explicitly or implicitly from demonstrations, i.e., imitation learning [Pomerleau, 1988, Ng and Russell, 2000, Abbeel and Ng, 2004, Ziebart et al., 2008]. With these methods, agent policies can either be directly extracted from the demonstrations or trained to optimize rewards functions learned from them.
Imitation learning (IL), however, only somewhat alleviates the need for expert human intervention. First, instead of designing complex reward functions, expert supervision is needed to collect massive
*corresponding author: ssontakk@usc.edu
Figure 1: RoboCLIP Overview. A Pretrained Video-and-Language Model is used to generate rewards via the similarity score between the encoding of an episode of interaction of an agent in its environment,  $\mathbf{z}^v$  with the encoding of a task specifier  $\mathbf{z}^d$  such as a textual description of the task or a video demonstrating a successful trajectory. The similarity score between the latent vectors is provided as reward to the agent.
datasets such as RT-1 [Brohan et al., 2022], Bridge Dataset [Ebert et al., 2021], D4RL [Fu et al., 2020], or Robonet [Dasari et al., 2019]. The performance of imitation learning algorithms and their ability to generalize hinges on the coverage and size of data [Kumar et al., 2019, 2022], making the collection of large datasets imperative. Second and most importantly, the interface for collecting demonstrations for IL is tedious, requiring expert robot operators to collect thousands of demonstrations. On the contrary, a more intuitive way to define rewards would be in the form of a textual description (e.g., "robot grasping object"), or in the form of a naturalistic video demonstration of the task performed by a human actor in an environment separate from the robotic environment. For example, demonstrating to a robot how to open a cabinet door in one's own kitchen is more naturalistic than collecting many thousands of trajectories via teleoperation in the target robotic environment.
Thus, there exists an unmet need for IL algorithms that 1) require very few demonstrations and 2) allow for a natural interface for providing these demonstrations. For instance, algorithms that can effectively learn from language instructions or human demonstrations without the need for full environment state information. Our key insight is that by leveraging Video-and-Language Models (VLMs)—which are already pretrained on large amount of video demonstration and language pairs—we do not need to rely on large-scale and in-domain datasets. Instead, by harnessing the power of VLM embeddings, we treat the mismatch between a single instruction's embedding (provided as a language command or a video demonstration) and the embedding of the video of the current policy's rollout as a proxy reward that will guide the policy towards the desired instruction.
To this end, we present RoboCLIP, an imitation learning algorithm that learns and optimizes a reward function based on a single language or video demonstration. The backbone model used in RoboCLIP is S3D [Xie et al., 2018] trained on the Howto100M dataset [Miech et al., 2019], which consists of short clips of humans performing activities with textual descriptions of the activities. These videos typically consist of a variety of camera angles, actors, lighting conditions, and backgrounds. We hypothesize that VLMs trained on such diverse videos are invariant to these extraneous factors and generate an actor-agnostic semantically-meaningful representation for a video, allowing them to generalize to unseen robotic environments.
We present an overview of RoboCLIP in Figure 1. RoboCLIP computes a similarity score between videos of online agent experience with a task descriptor, i.e., a text description of the task or a single human demonstration video, to generate trajectory-level rewards to train the agent. We evaluate RoboCLIP on the Metaworld Environment suite [Yu et al., 2020] and on the Franka Kitchen Environment [Gupta et al., 2019], and find that policies obtained by pretraining on the RoboCLIP reward result in  $2 - 3 \times$  higher zero-shot task success in comparison to state-of-the-art imitation learning baselines. Additionally, these rewards require no experts for specification and can be generated using naturalistic definitions like natural language task descriptions and human demonstrations.
Learning from Human Feedback. Learning from demonstrations is a long-studied problem that attempts to learn a policy from a dataset of expert demonstrations. Imitation learning (IL) methods, such as those based on behavioral cloning [Pomerleau, 1988], formulate the problem as a supervised learning over state-action pairs and typically rely on large datasets of expert-collected trajectories directly demonstrating how to perform the target task [Brohan et al., 2022, Lynch et al., 2022].
However, these large demonstration datasets are often expensive to collect. Another IL strategy is inverse RL, i.e., directly learning a reward function from the demonstrations [Ng and Russell, 2000, Abbeel and Ng, 2004, Ziebart et al., 2008, Finn et al., 2016]. Inverse RL algorithms are typically difficult to apply when state and action spaces are high-dimensional. Methods such as GAIL [Ho and Ermon, 2016], AIRL [Fu et al., 2017], or VICE [Fu et al., 2018] partially address these issues by assigning rewards which are proportional to the probability of a given state being from the demonstration set or a valid goal state as estimated by a learned discriminator network. However these discriminator networks still require many demonstrations or goal states to train to effectively distinguish between states from agent-collected experience and demonstration or goal states. On the other hand, RoboCLIP's use of pretrained video-and-language models allows us to train agents that learn to perform target tasks with just one demonstration in the form of a video or a language description. Other works instead use human feedback in the form of pairwise comparisons or rankings to learn preference reward functions [Christiano et al., 2023, Sadigh et al., 2017, Biyik et al., 2019, Myers et al., 2021, Biyik et al., 2022, Brown et al., 2019, Biyik et al., 2020, Lee et al., 2021, Hejna and Sadigh, 2022]. These preferences may require less human effort to obtain than reward functions, e.g., through querying humans to simply rank recent trajectories. Yet individual trajectory preferences convey little information on their own (less than dense reward functions) and therefore humans need to respond to many preference queries for the agent to learn useful reward functions. In contrast, RoboCLIP is able to extract useful rewards from a single demonstration or single language instruction.
Large Vision and Language Models as Reward Functions. Kwon et al. [2023] and Hu and Sadigh [2023] propose using large language models (LLMs) for designing and regularizing reward functions that capture human preferences. These works study the reward design problem in text-based games such as negotiations or card games, and thus are not grounded in the physical world. RoboCLIP instead leverages video-and-language models to assess if video demonstrations of robot policies align with an expert demonstration. Prior work has demonstrated that video models can be used as reward functions. For example, Chen et al. [2021] learn a visual reward function using human data and then utilize this reward function for visual model-based control of a robot. However, they require training the reward model on paired human and robot data from the deployment environment. We demonstrate that this paired data assumption can be relaxed by utilizing large-scale vision-language models pretrained on large corpora of human-generated data. The most well-known of these is CLIP [Radford et al., 2021], which is trained on pairs of images and language descriptions scraped from the internet. While CLIP is trained only on images, video-language-models (VLMs) trained on videos of humans performing daily tasks such as S3D [Xie et al., 2018] or XCLIP [Ni et al., 2022] are also widely available. These models utilize language descriptions while training to supervise their visual understanding so that semantically similar vision inputs are embedded close together in a shared vector space. A series of recent works demonstrate that these VLMs can produce useful rewards for agent learning. Fan et al. [2022] finetune CLIP on YouTube videos of people playing Minecraft and demonstrate that the finetuned CLIP model can be used as a language-conditioned reward function to train an agent. DECKARD [Nottingham et al., 2023] then uses the fine-tuned reward function of Fan et al. [2022] to reward an agent for completing tasks proposed by a large-language model and abstract world model. PAFF [Ge et al., 2023] uses a fine-tuned CLIP model to align videos of policy rollouts with a fixed set of language skills and relabel experience with the best-aligned language label. We demonstrate that videos and multi-modal task specifications can be utilized to learn reward functions allowing for training agents. Additionally, we present a method to test the alignment of pretrained VLMs with deployment environments.
Overview. RoboCLIP utilizes pretrained video-and-language models to generate rewards for online RL agents. This is done by providing a sparse reward to the agent at the end of the trajectory which describes the similarity of the agent's behavior to that of the demonstration. We utilize video-and-language models as they provide the flexibility of defining the task in terms of natural language descriptions or video demonstrations sourced either from the target robotic domain or other more naturalistic domains like human actors demonstrating the target task in their own environment. Thus, a demonstration (textual or video) and the video of an episode of robotic interaction are embedded into the semantically meaningful latent space of S3D [Xie et al., 2018], a video-and-language model pretrained on diverse videos of human actors performing everyday tasks taken from the HowTo100M
dataset [Miech et al., 2019]. The two vectors are subsequently multiplied using a scalar product generating a similarity score between the 2 vectors. This similarity score (without scaling) is returned to the agent as a reward for the episode.
Notation. We formulate the problem in the manner of a POMDP (Partially Observable Markov Decision Process) with  $(\mathcal{O}, S, \mathcal{A}, \phi, \theta, r, T, \gamma)$  representing an observation space  $\mathcal{O}$ , state space  $S$ , action space  $\mathcal{A}$ , transition function  $\phi$ , emission function  $\theta$ , reward function  $r$ , time horizon  $T$ , and discount factor  $\gamma$ . An agent in state  $\mathbf{s}_t$  takes an action  $\mathbf{a}_t$  and consequently causes a transition in the environment through  $\phi(\mathbf{s}_{t+1} \mid \mathbf{s}_t, \mathbf{a}_t)$ . The agent receives the next state  $\mathbf{s}_{t+1}$  and reward  $r_t = r(\mathbf{o}_t, \mathbf{a}_t)$  calculated using the observation  $\mathbf{o}_t$ . The goal of the agent is to learn a policy  $\pi$  which maximizes the expected discounted sum of rewards, i.e.,  $\sum_{t=0}^{T} \gamma^t r_t$ . Note that all of our baselines utilize the true state for reward generation and for policy learning. To examine the effect of using a video-based reward, we also operate our policy on the state space while using the pixel observations for reward generation. Thus,  $r_t$  uses  $\mathbf{o}_t$  while  $\pi$  uses  $\mathbf{s}_t$  for RoboCLIP while for all other baselines, both  $r_t$  and  $\pi$  utilize  $\mathbf{s}_t$ . This of course is unfair to our method, but we find that in spite of the advantage provided to the baselines, RoboCLIP rewards still generate higher zero-shot success.
Reward Generation. During the pretraining phase, we supply the RoboCLIP reward to the agent in a sparse manner at the end of each episode. This is done by storing the video of an episode of the interaction of the agent with the environment into a buffer as seen in Figure 1. A sequence of observations of length 128 are saved in a buffer corresponding to the length of the episode. S3D is trained on videos length 32 frames and therefore the episode video is subsequently downsampled to result in a video of length  $T = 32$ . The video is subsequently center-cropped to result in frames of size (250, 250). This is done to ensure that the episode video is preprocessed to match the specifications of the HowTo100M preprocessing used to train the S3D model. Thus the tensor of a sequence of  $T$  observations  $\mathbf{o}_{0:T}$  is encoded as the latent video vector  $\mathbf{z}^v$  using
$$
\mathbf {z} ^ {v} = S 3 D ^ {\text {v i d e o - e n c o d e r}} \left(\mathbf {o} _ {0: T}\right) \tag {1}
$$
The task specification is also encoded into the same space. If it is defined using natural language, the language encoder in S3D encodes a sequence of  $K$  textual tokens  $\mathbf{d}_{0:K}$  into the latent space using:
$$
\mathbf {z} ^ {d} = S 3 D ^ {\text {t e x t - e n c o d e r}} \left(\mathbf {d} _ {0: K}\right) \tag {2}
$$
If the task description is in the form of a video of length  $K$ , then we preprocess and encode it using the video-encoder in S3D just as in Equation (1). For intermediate timesteps, i.e., timesteps other than the final one in an episode, the reward supplied to the agent is zero. Subsequently, at the end of the episode, the similarity score between the encoded task descriptor  $\mathbf{z}^d$  and the encoded video of the episode  $\mathbf{z}^v$  is used as reward  $r^{\mathrm{RoboCLIP}}(T)$ . Thus the reward is:
$$
r ^ {\mathrm {R o b o C L I P}} (t) = \left\{ \begin{array}{l l} 0, & t \neq T \\ \mathbf {z} ^ {d} \cdot \mathbf {z} ^ {v} & t = T \end{array} \right.
$$
where  $\mathbf{z}^d\cdot \mathbf{z}^v$  corresponds to the scalar product between vectors  $\mathbf{z}^d$  and  $\mathbf{z}^v$
Agent Training. Using  $r^{\mathrm{RoboCLIP}}$  defined above, we then train an agent online in the deployment environment with any standard reinforcement learning (RL) algorithm by labeling each agent experience trajectory with  $r^{\mathrm{RoboCLIP}}$  after the agent collects it. In our paper, we train with PPO [Schulman et al., 2017], an on-policy RL algorithm, however, RoboCLIP can also be applied to off-policy algorithms. After training with this reward, the agent can be zero-shot evaluated or fine-tuned on true environment reward on the target task in the deployment environment.
We test out each of the hypotheses defined in Section 1 on simulated robotic environments. Specifically, we ask the following questions:
1. Do existing pretrained VLMs semantically align with robotic manipulation environments?
2. Can we utilize natural language to generate reward functions?
3. Can we use videos of expert demonstrations to generate reward functions?
4. Can we use out-of-domain videos to generate reward functions?
5. Can we generate rewards using a combination of demonstration and natural language?
6. What aspects of our method are crucial for success?
We arrange this section to answer each of these questions. Both RoboCLIP and baselines utilize PPO [Schulman et al., 2017] for policy learning.
Baselines. We use 2 state-of-the-art methods in inverse reinforcement learning: GAIL, or Generative Adversarial Imitation Learning [Ho and Ermon, 2016] and AIRL or Adversarial Inverse Reinforcement Learning [Fu et al., 2017]. Both of these methods attempt to learn reward functions from demonstrations provided to the agent. Subsequently, they train an agent using this learned reward function to imitate the expert behavior. Both methods receive a single demonstration, consistent with our approach of using a single video imitation. However, since they both operate on the ground-truth environment state, we provide them with a trajectory of states, instead of images, thereby providing them privileged state information that our method does not receive.
4.1 Domain Alignment
Pretrained vision models are often trained on a variety of human-centric activity data, such as Ego4D [Grauman et al., 2022]. Since we are interested in solving robotic tasks with view from third person perspectives, we utilize the S3D [Xie et al., 2018] VLM pretrained on HowTo100M [Miech et al., 2019], a dataset of short third-person clips of humans performing everyday activities. This dataset, however, contains no robotic manipulation data.
To analyze the alignment of the VLM to different domains, we perform a confusion matrix analysis using videos from Metaworld [Yu et al., 2020]. We collect 10 videos per task with varying values of true reward. For each video, we also collect the true reward. We then compute the RoboCLIP reward for each video using VLM alignment between the textual description of the task and the video. We visualize the correlations between the RoboCLIP and true rewards in the form of an  $n \times n$  matrix where entry  $(i,j)$  corresponds to the correlation between the true reward and the RoboCLIP reward generated for the  $i^{\text{th}}$  task using the  $j^{\text{th}}$  text description. As one can see, for a given task, the highest correlation in the matrix is for the correct textual description. We visualize one such similarity matrix in Figure 2 for Metaworld. We find that Metaworld seems to align well in the latent space of the model with a more diagonal-heavy confusion matrix. The objects are all correctly identified.
Figure 2: Domain Alignment Confusion Matrix. We perform a confusion matrix analysis on a subset of the data on collected on Metaworld [Yu et al., 2020] environments by comparing the pair-wise similarities between the latent vectors of the strings describing the videos and those of the videos. We find that Metaworld is well-aligned with higher scores along the diagonal than along the off-diagonal elements.
4.2 Language for Reward Generation
The most naturalistic way to define a task is through natural language. We do this by generating a sparse reward signal for the agent as described in Section 3: the reward for an episode is the similarity score between its encoded video and the encoded textual description of the expected behavior in the VLM's latent space. The reward is provided to the agent at the end of the episode. For RoboCLIP, GAIL, and AIRL, we first pretrain the agents online with their respective reward functions and then perform finetuning with the true task reward in the deployment environment. We perform this analysis on 3 Metaworld Environments: Painter-Close, Door-Close and Button-Press. We use the textual descriptions, "robot closing green drawer", "robot closing black box", and "robot pushing red button" for each environment, respectively. Figure 3 plots returns on the target tasks while finetuning on the depoloyment environment after pretraining (with the exception of the Dense Task Reward baseline). Our method outperforms the imitation learning baselines with online exploration in terms of true task rewards in all environments. Additionally our baselines utilize the full state information in the environment for reward generation where RoboCLIP uses only the pixels to infer state. RoboCLIP also achieves more than double zero-shot rewards in all environments — importantly,
Figure 3: Language-Conditioned Reward Generation. The pretrained VLM is used to generate rewards via the similarity score of the encoding of an episode of interaction of an agent in its environment,  $\mathbf{z}^v$  with the encoding of a task specifier  $\mathbf{z}^d$  specified in natural language. We use the strings, "robot closing black box", "robot closing green drawer" and "robot pushing red button" for conditioning for the 3 environments respectively. We find that agents pretrained on these language-conditioned rewards outperform imitation learning baselines like GAIL [Ho and Ermon, 2016] and AIRL [Fu et al., 2017].
Figure 4: Using In-Domain Videos for Reward Generation. The pretrained VLM is used to generate rewards via the similarity score of the encoding of an episode of interaction of an agent in its environment,  $\mathbf{z}^v$  with the encoding of a video demonstration of expert behavior in the same environment. The similarity score between the latent vectors is provided as reward to the agent and is used to train online RL methods. We study this setup in the Kettle, Hinge and Slide Tasks in the Franka Kitchen Environment [Gupta et al., 2019]. We find that policies trained on the RoboCLIP reward are able to learn to complete the task in all three setups without any need for external rewards using just a single in-domain demonstration.
the RoboCLIP-trained agent is able to complete the tasks even before finetuning on true task rewards.
4.3 In-Domain Videos for Reward Generation
Being able to use textual task descriptors for reward generation can only work in environments where there is domain alignment between the pretrained model and the visual appearance of the environment. Additionally, VLMs are large models often with billions of parameters making it computationally expensive to fine tune for domain alignment. The most naturalistic way to define a task in such a setting is in the form a single demonstration in the robotic environment which can be collected using teleoperation. We study how well this works in the Franka Kitchen [Gupta et al., 2019] environment. We consider access to a single demonstration per task whose video is used to generate rewards for online RL.