Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
Authors: Juan Rocamonde
Victoriano Montesinos
Elvis Nava
Ethan Perez
David Lindner
Venue: ICLR 2024 poster (2024)
Keywords: deep learning
deep reinforcement learning
reward learning
vision-language models
vision transformer
transformer
CLIP
Primary area: reinforcement learning
TL;DR: Using vision-language models (VLMs) as reward models for RL tasks works well with large enough VLMs.

Abstract: Reinforcement learning (RL) requires either manually specifying a reward function, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a more sample-efficient alternative: using pretrained vision-language models (VLMs) as zero-shot reward models (RMs) to specify tasks via natural language. We propose a natural and general approach to using VLMs as reward models, which we call VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn complex tasks without a manually specified reward function, such as kneeling, doing the splits, and sitting in a lotus position. For each of these tasks, we only provide _a single sentence text prompt_ describing the desired task with minimal prompt engineering. We provide videos of the trained agents at: https://sites.google.com/view/vlm-rm. We can improve performance by providing a second "baseline" prompt and projecting out parts of the CLIP embedding space irrelevant to distinguish between goal and baseline. Further, we find a strong scaling effect for VLM-RMs: larger VLMs trained with more compute and data are better reward models. The failure modes of VLM-RMs we encountered are all related to known capability limitations of current VLMs, such as limited spatial reasoning ability or visually unrealistic environments that are far off-distribution for the VLM. We find that VLM-RMs are remarkably robust as long as the VLM is large enough. This suggests that future VLMs will become more and more useful reward models for a wide range of RL applications.

Published as a conference paper at ICLR 2024
VISION-LANGUAGE MODELS ARE ZERO-SHOT REWARD MODELS FOR REINFORCEMENT LEARNING
Juan Rocamonde FAR AI
Victoriano Montesinos Vertebra
Elvis Nava
ETH Al Center
Ethan Perez*
David Lindner*3
Anthropic
ETH Zurich
We use CLIP as reward model train MuJoCo humanoid robot (1) stand with raised arms, (2) sit in lotus position, (3) do the splits, and (4) kneel on the ground (from left to right). We specify each task using single sentence text prompt. The prompts are simple (e.g.. "a humanoid robot kneeling") and none of these tasks required prompt engineering. See Section 4.3 for details on our experimental setup.
Figure 1:
ABSTRACT
Reinforcement learning (RL) requires either manually specifying reward func- tion, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a more sample- efficient alternative: using pretrained vision-language models (VLMs) as zero- shot reward models (RMs) to specify tasks via natural language. We propose a natural and general approach to using VLMS as reward models, which we call VLM-RMs. We use VLM-RMs based on CLIP to train MuJoCo humanoid to learn complex tasks without manually specified reward function, such kneeling, doing the splits, and sitting a lotus position. For each these tasks, only provide single sentence text prompt describing the desired task with minimal prompt engineering. We provide videos the trained agents at: https::/ //s::///a //si/sssssss google e, www/wiww///cocct.c wrriew/wlm-mie rm We can improve perfor- mance by providing a second "baseline" prompt and projecting out parts the CLIP embedding space irrelevant distinguish between goal and baseline. Fur- ther, we find a strong scaling effect for VLM-RMs: larger VLMs trained with more compute and data are better reward models. The failure modes of VLM-RMs we encountered are all related to known capability limitations of current VLMs, such as limited spatial reasoning ability or visually unrealistic environments that remarkably ro-
are far off-distribution for the VLM. We find that VLM-RMs bust as long as the VLM is large enough. This suggests that future VLMs will become more and more useful reward models for a wide range of RL applications.
Additional affiliation: Vertebra
Condespondence to: juancarlosrocamonde @ gmail.com, david inf.eeeeeenf......cc
Equal contribution
Source code available https: thub. wtw.geetttteeeeeeeiiee Kesearch//1mmm

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Training reinforcement learning (RL) agents to perform complex tasks in vision-based domains can be difficult, due to high costs associated with reward specification. Manually specifying reward functions for real world tasks is often infeasible, and learning reward model from human feedback is typically expensive. To make RL more useful in practical applications, it is critical to find more sample-efficient and natural way to specify reward functions.
One natural approach is to pretrained vision-language models (VLMs), such CLIP (Rad- ford et al., 2021) and Flamingo (Alayrae et al., 2022), to provide reward signals based on natural language. However, prior attempts use VLMs to provide rewards require extensive fine-tuning VLMs (e.g., Du et al., 2023) or complex ad-hoc procedures to extract rewards from VLMs (c.g.. Mahmoudieh al., 2022). In this work, we demonstrate that simple techniques for using VLMs as zero-shot language-grae -rooundd reward models work well, as long as the chosen underlying model is sufficiently capable. Concretely, we make four key contributions.
First, we propose VLM-RM, general method for using pre-trained VL.Ms as reward model for vision-based RL tasks (Section 3). We propose concrete implementation that uses CLIP as a VLM and cos- similarity between the CLIP embedding of the current environment state and simple language prompt as a reward function. We can optionally regularize the reward model by providing "baseline prompt" that describes neutral state of the environment and partially projecting the representations onto the direction between baseline and target prompts when computing the reward.
Second, we validate our method in the standard CartPole and MountainCar RL bench- marks (Section 4.2). We observe high correlation between VLM-RMs and the ground truth rewards of the environments and successfully train policies solve the tasks using CLIP reward model. Furthermore, we find that the quality of CLIP as a reward model improves if we render the environ- ment using more realistic textures.
Third, we train a MuJoCo humanoid to learn complex tasks, including raising its arms, sitting in lotus position, doing the splits, and kneeling (Figure Section 4.3) using a CLIP reward model derived from single sentence text prompts (c.g., "a humanoid robot kneeling").
Fourth, we study how VLM-RMs' performance scales with the size of the VLM, and find that VLM scale is strongly correlated to VLM-RM quality (Section 4.4), In particular, we can only learn the humanoid tasks in Figure with the largest publicly available CLIP model.
Our results indicate that VLMs are powerful zero-shot reward models. While current models, such as CLIP, have important limitations that persist when used VLM-RMs, we expect such limitations to mostly be overcome as larger and more capable VLMs become available. Overall, VLM-RMs are likely to enable us to train models to perform increasingly sophisticated tasks from human-written task descriptions.
2 BACKGROUND
Partially observable Markov decision processes. We formulate the problem of training RL agents in vision-based tasks as a partially observable Markov decision process (POMDP). A POMDP is a tuple (S, A, 6. R, O o, do) where: S is the state space; A is the action space; [0, is the transition function; R(s, a, 8') S A x S + R is the reward function; O is the observation space; Φ(o[s) : S A(O) is the observation distribution; and do(s): S + [0, 1] is the initial state distribution.
068's,, a) S S x A
At each point in time, the environment is in state S. In each timestep, the agent takes an action a E A, causing the environment to transition to state with probability 0(s') s, a). The agent then receives of states and actions called a trajectory 7 = (So, Qu $1, .1,...) where 8; E S, and a; E A. The returns of such a trajectory
observation o, with probability o(ols') and reward r R(s, a, s'). A sequence
the discounted sum of rewards g(T; ; R) = -1 7 R(s,, O2, $141).
The agent's goal is to find a (possibly stochastic) policy π (ssaa that maximizes the expected returns G(T) By())) [g(r(T); R)). We only consider finite-horizon trajectories, i.e., |T| co.
2

---

Published as a conference paper at ICLR 2024
Vision-languaage models. We broadly define vision-language models (VLMs; Zhang et al., 2023) as models capable of processing sequences both language inputs E _ and vision inputs i E -_ Here, L is finite alphabet and _ contains strings of length less than equal to n, whereas I is the space of 2D RGB images and ISm contains sequences of images with length less than equal m.
CLIP models. One popular class of VLMs Contrastive I Language-Image Pretraining (CLIP; Radford et al., 2021) encoders. CLIP models consist of a language encoder CLIP,, L + V and an image encoder CLIP, I + mapping into the same latent space Æ These encoders are jointly trained via contrastive learning over pairs of images and captions. Commonly CLIP encoders are trained to minimize the cosine distance between embeddings for semantically matching pairs and maximize the cosine distance between semantically non- non-matching pairs.
VISION-LANGUAGE MODELS AS REWARD MODELS (VLM-RMs)
This section presents how we can use VLMs as a learning-free (zero-shot) way to specify rewards from natural language descriptions of tasks. Importantly, VLM-RMs avoid manually engineering a reward function or collecting expensive data for learning a reward model.
3.1 USING VISION-LANGUAGE MODELS AS REWARDS
Let us consider a POMDP without a reward function
y, do)). We focus on vision- based RL where the observations E O are images. For simplicity, assume deterministic observation distribution p(o])) defined by a mapping ti(s) : - O from states to image obser- vation. We want the agent perform task T based on natural language description L For example, when controlling a humanoid robot (Section 4.3) T might be the robot kneeling on the ground and might be the string "a humanoid robot kneeling".
To train the agent using RI., we need to first design a reward function. We propose to use VLM to provide the reward R(s) as:
(s) VLM((, ()),
where e E CSn is an optional context, e.g., for defining the reward interactively with a VLM. This formulation is general enough to encompass the use of several different kinds of VLMs, including image and video encoders, as reward models.
CLIP as a reward model. In our experiments, we chose a CLIP encoder as the VLM. very basic way to use CLIP to define reward function is to use cosine similarity between state's 's image representation and the natural language task description:
(L) (LIP)((())) RELI = |CLIPI (
In this case, we do not require a context c. We will sometimes call the CLIP image encoder a state encoder, it encodes an image that is direct function of the POMDP state, and the CLIP language encoder a task encoder, as it encodes the language description of the task.
3.2 GOAL-BASELINE REGULARIZATIN TO IMPROVE CLIP REWARD MODELS
While in the previous section, we introduced very basic way of using CLIP to define task-based reward function, this section proposes Goal-Buselln Regularization as a way to improve the quality of the reward by projecting out irrelevant information about the observation.
So far, we assumed we only have a task description E _Š" To apply goal- boal-baseline regularization, we require second "baseline" description b E LS" The baseline b is natural language description of the environment setting in its default state, irrespective of the goal. For example, our baseline description for the humanoid is simply "a humanoid robot," whereas the task description is, e.g., "a humanoid robot kneeling." " We obtain the goal- regularized CLIP reward model (RcL.P.Reg) by projecting state embedding onto the line spanned by the baseline and task embeddings.
3

---

Published as a conference paper at ICLR 2024
Definition 1 (1) (U)) tion b, let g the line spanned by and g. The goal-baseling regularized reward rewarduunction is given by
Regularization), Civen goal task description and baseline descrip-
b |[CUP (L),;(b) s CLP (v(e)) be the normalized encodings, and be
[[C]]]
ReLIP-Reg = ||a projz s + - aa)s g g1/2
where or is parameter to control the regularization strength.
In particular, for a = O, we recover our initial CLIP reward function ReI. On the other hand, for 1, the projection removes all components of orthogonal to b.
Intuitively, the direction from b to g captures the change from the environment's 's baseline to the tar- get state. By projecting the reward onto this direction, we directionally remove irrelevant parts of the CLIP representation. However, we can not be sure that the direction really captures all relevant in- formation. Therefore, instead of using 1, treat it as hyperparameter. However, we find the method to be relatively robust to changes in a with most intermediate values being better than 0 or 1.
3.3 RL WITH CLIP REWARD MODEL
We can now use VLM-RMs drop-in replacement for the reward signal in RL. In our implemen- tation, we use the Deep Q-Network (DQN; Mnih et al., 2015) or Soft Actor-Critic (SAC; Haarnoja al., 2018) RL algorithms. Whenever we interact with the environment, we store the observa- tions in replay buffer. In regular intervals, we pass a batch of observations from the replay buffer through a CLIP encoder to obtain the corresponding state embeddings. We can then compute the reward function as cosine similarity between the state embeddings and the task embedding which we only need to compute once. Once we have computed the reward for a batch of interactions, we can use them to perform the standard RL algorithm updates. Appendix C contains more implementation details and pseudocode for our full algorithm in the case of SAC.
EXPERIMENTS
We conduct a variety of experiments to evaluate CLIP reward model with and without goal- baseline regularization. We start with simple control tasks that are popular RL benchmarks: CartPole and MountainCar (Section 4.2). These environments have a ground truth reward function and simple, well-structured state space. We find that our reward models are highly cor- related with the ground truth reward function, with this correlation being greatest when applying goal-baseline regularization. Furthermore, find that the reward model's outputs can be signifi- cantly improved by making a simple modification to make the environment's observation function more realistic, e.g., by rendering the mountain car
mountain texture.
We then move on to our main experiment: controlling a simulated humanoid robot (Section 4.3). We use CLIP reward models specify tasks from short language prompts; several of these tasks are challenging to specify manually. We find that these zero-shot CLIP reward models are suffi- cient for RL algorithms to learn most tasks attempted with little no prompt engineering or hyperparameter tuning.
Finally, we study the scaling properties of the reward models by using CLIP models of different sizes as reward models in the humanoid environment (Section 4.4). We find that larger CLIP models are significantly better reward models. particular, we can only successfully learn the tasks presented in Figure 1 when using the largest publicly available CLIP model.
Experiment setup. We extend the implementation of the DQN and SAC algorithm from the stable -ese-aaaaeeesseee library (Raffin et al., 2021) to compute rewards from CLIP reward models instead of from the environment. As shown in Algorithm for SAC, we alternate between envi- ronment steps, computing the CLIP reward, and RL algorithm updates. We run the RL algorithm updates on a single NVIDIA RTX A6000 GPU. The environment simulation runs on CPU, but we perform rendering and CLIP inference distributed over NVIDIA RTX A6000 GPUs.
We provide the code to reproduce our experiments in the supplementary material. We dis- cuss hyperparameter choices in Appendix C, but we mostly standard parameters from
4

---

Published as a conference paper at ICLR 2024
stable-baselines Appendix C also contains table with full list of prompts for experiments, including both goal and baseline prompts when using goal-baseling baseline regularization.
4.1 How CAN WE EVALUATE VLM--MMS
Evaluating reward models can be difficult, particularly for tasks for which we do not have a ground truth reward function. In our experiments, we use 3 types of evaluation: (i) evaluating policies using ground truth reward; (ii) comparing reward functions using EPIC distance; (iii) human evaluation.
Evaluating policies using ground truth reward. If we have a ground truth reward function for task such as for the CarPole and we can train a policy using VLM-RM and evaluate it using the ground truth reward. This is the most popular way to evaluate reward models in the literature and we use it for environments where we have ground-truth reward available.
we can use evaluate policies. For example,
Comparing reward functions using EPIC distance. The "Equivalent Policy-Invarine Compar- ison" (EPIC; Gleave et al., 2021) distance compares two reward functions without requiring the expensive policy training step. EPIC distance is provably invariant on the equivalence class re- ward functions that induce the same optimal policy. We consider only goal-based tasks, for which the EPIC distance particularly easy to compute. In particular, a low EPIC distance between the CLIP reward model and the ground truth reward implies that the CLIP reward model successfully separates goal states from con-goal states. Appendix A discusses in more detail how we compute intuitively interpret it for goal-based tasks.
the EPIC distance in our case, and how
Human evaluation. For tasks without a ground truth reward function, such as all humanoid tasks in Figure need to perform human evaluations decide whether agent is successful. We define "success rate" as the percentage of trajectories in which the agent successfully performs the task in at least 50% of the timesteps. For each trajectory, have single rater? label how many timesteps were spent successfully performing the goal task, and use this to compute the success rate. However, human evaluations can also be expensive, particularly if want evaluate many different policies, e.g., to perform ablations. For such cases, we additionally collect a dataset of numan-labelled states for each task, including goal states and non-goal states. We can then compute the EPIC distance with these binary human labels. Empirically, we find this to be a useful proxy for the reward model quality which correlates well with the performance of a policy trained using the reward model.
For more details on our human evaluation protocol, we refer to Appendix Our human evaluation protocol is very basic and might be biased. Therefore, additionally provide videos of our trained agents at https: //ss://sitssssss httsssstesssssslleeeee/esooossee google. www.ccow//vvweeeeeee rm.
4.2 CAN VLM-RMs SOLVE CLASSIC CONTROL BENCHMARKS?
As an initial validation of our methods, we consider two classic control environments: Cart and MountainCar, implemented in OpenAl Gym (Brockman et al., 2016). In addition to the nCar environment, we also consider a version with a modified rendering method that adds textures to the mountain and the car so that it resembles the setting of "a car at the peak of mountain" more closely (see Figure 2). This environment allows us test whether VLM-RMs work better in visually "more realistic" environments.
default Mountai inaar
To understand the rewards our CLIP reward models provide, we first analyse plots their reward landscape. In order to obtain simple and interpretable visualization figure, we plot CLIP rewards against a one-dimensional state space parameter, that is directly related to the completion of the task. For the CartPole (Figure 2a) we plot CLIP rewards against the angle of the pole, where the ideal position is at angle 0. For the (untextured and textured) Mount annaariiCrr environments Figures 2b and 2c, plot CLIP rewards against the position of the along the horizontal axis, with the goal location being around æ 0.5.
"One of the authors.
5

---

Published as a conference paper at ICLR 2024
1.00
1.00
1.0
rewand (reccald)
(roccald)
tewanl (revalld)
11.75
0.5
0.5
0.5
0.25
0.0
0.0
0.0
-0.25
0,00
0.25
-1.0
-0.5
0.0
0.5
-1.0
-0.5
0.0
0.5
Pole ungle (radians)
position
(a) Ca CartPole
(h) Mountai nCaa (original)
(c) Mounta i i (textured)
CartPole and Mounta n.ar. We plot the CLIP reward as function of the pole angle for the CartPole (a) and as a function of the x position for the Mounta nCa (b,c). We mark the respective goal states with vertical line. The line color encodes different regularization strengths a. For the Cart the maximum reward is always when balancing the pole and the regularization has little effect. For the CouunainnCas the agent obtains the maximum reward top of the mountain. But, the reward landscape is much more well-behaved when the environment has textures and we add goal-baseling regularization this is consistent with our results when training policies.
Figure 2: We study the CLIP reward landscape in two classic control environments:
Figure 2a shows that CLIP rewards are well-shaped around the goal state for the CartPole envi- ronment, whereas Figure 2b shows that CLIP rewards for the default Mounta nCar environment poorly shaped, and might be difficult learn from, despite still having roughly the right maxi- mum.
We conjecture that zero-shot VLM-based rewards work better in environments that are more "pho- torealistic because they are closer to the training distribution of the underlying VLM. Figure 2c shows that described earlier, we apply custom textures to the MountainCat environment, the CLIP rewards become well-shaped when used in concert with the goal-baseling regularization technique. For larger regularization strength a, the reward shape resembles the slope the hill from the environment itself an encouraging result.
We then train agents using the CLIP rewards and goal-baseline regularization in all three envi- ronments, and achieve 100% task success rate in both environments (CartPole and textured MountainCaz) for most a regularization strengths. Without the custom textures, we are not able to successfully train an agent the mountain car task, which supports our hypothesis that the environment visualization too abstract.
The results show that both and regularized CLIP rewards effective in the toy RL task domain, with the important caveat that CLIP rewards are only meaningful and well-shaped for environments that are photorcalistic enough for the CLIP visual encoder to interpret correctly.
4.3 CAN VLM-RMS LEARN COMPLEX, NOVEL TASKS IN HUMANOID ROBOT?
Our primary goal using VLM-RMs is learn tasks for which it is difficult to specify a reward function manually. To study such tasks, consider the Humanoid-v4 environment implemented in the MuJoCo simulator (Todorov et al., 2012).
The standard task in this environment is for the humanoid robot to stand up. For this task, the environment provides a reward function based on the vertical position the robot's center of mass. We consider range of additional tasks for which ground truth reward function is available, including kneeling, sitting in a lotus position, and doing the splits. For a full list of tasks we tested, Table Appendix C presents more detailed task descriptions and the full prompts used.

---

Published as a conference paper at ICLR 2024
Table 1: We successfully learned out of 8 tasks we tried for the humanoid robot (cf. Figure 1). For each task, we evaluate the checkpoint with the highest CLIP reward over random seeds. We show human evaluator 100 trajectories
Success Rate
Task
Kneeling
100%
from the agent and ask them to label how many timesteps
Lotus position
100%
were spent successfully performing the goal task. Then, we 100%
Standing up
label an episode as a success if the agent is in the goal state at least 50% of the timesteps. The success rate is the frac- tion of trajectories labelled as successful. We provide more de- tails the evaluation well more fine-grained human la- bels in Appendix and videos of the agents' performance at https: betps://siters /tt////tsss////////////eee/e/eee google www/view///cocccoo...o.../.....e www.-lll--aa rm.
Arms raised
100%
Doing splits Hands on hips Standing on one leg
100%
64%
0%
Arms crossed
0%
Camera Angle
Success Rate
Textures
Original
Original Modified
36%
Original
91%
Modified
Modified
100%
(a) Original
(b) Modified textures
(c) Modified textures
& camera angle
Figure 3: We test the effect of our modifications to the standard Humanoid-v4 environment on the kneeling task. We compare the original environment (a) to modifying the textures (b) and the camera angle (c). We find that modifying the textures to more realistic is crucial making the CLIP reward model work. Moving the camera to give a better view of the humanoid helps too, but less critical in this task.
We make two modifications the default Humano d-v4 environment make better suited for experiments. (I) We change the colors of the humanoid texture and the environment background to be more realistic (based on our results in Section 4.2 that suggest this should improve the CLIP encoder). (2) We move the camera to fixed position pointing at the agent slightly angled down because the original camera position that moves with the agent can make some of our tasks impos- sible to evaluate. We ablate these changes in Figure finding the texture change critical and repositioning the camera provides modest improvement.
Table shows the human-evaluated success rate for all tasks tested. We solve 5 out of 8 tasks we tried with minimal prompt engineering and tuning. For the remaining 3 tasks, we did not major performance improvements with additional prompt engineering and hyperparameter tuning, and we hypothesize these failures are related to capability limitations in the CLIP model we use. We invite the reader evaluate the performance the trained agents themselves by viewing videos at https: sites. google. com/ / /ww/////////ee. vetvvvlrrnnnn rm.
The three tasks that the agent does not obtain perfect performance for are "hands on hips", "standing on one leg", and "arms crossed". We hypothesize that "standing on one leg" is very hard to learn or might even be impossible in the MuJoCo physics simulation because the humanoid's feet are round. The goal state for "hands hips" and "arms crossed" is visually similar to humanoid standing and we conjecture the current generation of CLIP models are unable to discriminate between such subtle differences in body pose.
While the experiments in Table use no evaluate goal-baseline regularization for the kneeling task. Figure 4a shows that 0 improves the reward model's EPIC distance to human labels, suggesting that it would also improve performance on the final task, might need more fine-grained evaluation criterion that.
regularization (i.e., a = 0), we separately

---

Published as a conference paper at ICLR 2024
RN50
VIT-I-14
ViT-H-14
ViT-bigG-14
0.7
RN5D
Success Rate
0.6
Model
0.6
0.5
BPIC
VITL-114
RN50
0%
ViT-I-14
0%
0.4
VIT-H-14 A
VIT-H-14
0%
0.4
ViT-higG-14
100%
ViT-bigG-I
0.0
0.5
1.0
8.0
8.5
9.0
a
logio number of parameters
(a) Goal-baseline regularization for different model sizes.
(b) Reward model performance by VLM training compute (a 0).
(c) Human-evaluated suc- cess rate (over 2 seeds).
Figure 4: VLMs become better reward models with VLM model scale. We evaluate the humanoid kneeling task for different VLM model sizes. We evaluate the EPIC distance between the CLIP rewards and human labels (a and b) and the human-evaluated success rate of an agent trained using differently sized CLIP reward models (c). We see strong positive effect of model scale on VLM- RM quality. In particular, (c) shows we are only able to learn the kneeling task using the largest CLIP model publically available, whereas (b) shows there a smooth improvement in EPIC distance compared to human labels. (a) shows that goal- bal-lbe regularization improves the reward model across model sizes but it is more impactful for small models.
4.4 How DO VLM-RMs SCALE WITH VLM MODEL SIZE?
Finally, we investigate the effect of the scale of the pre-trained VLM on its quality as a reward model. We focus on the "kneeling" task and consider 4 different large CLIP models: the original CLIP RN50 (Radford et al.. 2021), and the ViT-I-14, ViT-H-14, and ViT-bigG-14 from OpenCLIP (Cherti et al., 2023) trained on the LAION-5B dataset (Schuhmann al., 2022).
In Figure 4a we evaluate the EPIC distance to human labels of CLIP reward models for the four model scales and different values of a, and we evaluate the success rate of agents trained using the four models. The results clearly show that VLM model scale is key factor in obtaining good reward models. We detect a clear positive trend between model scale, and the EPIC distance of the reward model from human labels. On the models we evaluate, we find the EPIC distance to human labels is close to log-linear in the size of the CLIP model (Figure 4b).
This improvement in EPIC distance translates into improvement in success rate. In particular, we observe a sharp phase transition between the ViT-H-14 4 and VIT-bigG-14 CLIP models: can only learn the kneeling task successfully when using the VIT-bi ArT-bigG-1- 4 model and obtain 0% success rate for all smaller models (Figure 4c). Notably, the reward model improves smoothly and predictably with model scale as measured by EPIC distance. However, predicting the exact point where the RL agent can successfully learn the task is difficult. This is a common pattern in evaluating large foundation models, as observed by Ganguli al. (2022).
RELATED WORK
Foundation models (Bommasani al., 2021) trained on large scale data can learn remarkably gen- eral and transferable representations of images, language, and other kinds of data, which makes them useful for large variety of downstream tasks. For example, pre-trained vision-language encoders, such CLIP (Radford et al., 2021), have been used far beyond their original scope, e.g., for image generation (Ramesh et al., 2022; Patashnik et al., 2021; Nichol et al., 2021), robot control (Shridhar al., 2022; Khandelwal et al., 2022), or story evaluation (Matiana et al., 2021).
Reinforcement learning from human feedback (RLHF; Christiano et al., 2017) is a critical step in making foundation models more useful (Ouyang et al., 2022). However, collecting human feedback expensive. Therefore, using pre-trained foundation models themselves to obtain reward signals for RL finetuning has recently emerged as key paradigm in work on large language models (Bai
8

---

Published as a conference paper at ICLR 2024
et al., 2022). Some approaches only require small amount of natural language feedback instead of a whole dataset of human preferences (Scheurer et al., 2022; 2023; Chen et al., 2023). However, similar techniques have yet to be adopted by the broader RL community.
While some work uses language models to compute a reward function from a structured environ- ment representation (Xie et al., 2023; Ma et al., 2023), many RL tasks visual and require using VLMs instead. Sumers et al. (2023) use generative VLMs to relabel the goal of agent trajectories for hindsight experience replay, but not for specifying rewards. Cui et al. (2022) use CLIP to pro- vide rewards for robotic manipulation tasks given a goal image. However, they only show limited success when using natural language descriptions to define goals, which is the focus of our work. Mahmoudich et al. (2022) are the first to successfully use CLIP encoders as a reward model con- ditioned on language task descriptions in robotic manipulation tasks. However, to achieve this, the authors need to explicitly fine-tune the CLIP image encoder on a carefully crafted dataset for robotics task. Instead, we focus on leveraging CLIPP's zero-shot ability to specify reward functions, which is significantly more sample-efficiett and practical. Fan et al. (2022) train CLIP model to provide a reward signal in Minecraft environments. But, that approach requires a lot of labeled, environment-speecin
data. Du et al. (2023) finetune a Flamingo VLM (Alayrac al., 2022) to act as a "success detector" for iision-based RL tasks tasks. However, they do not train RL policies using these success detectors, leaving open the question of how robust they under optimization pres- sure. Concurrently to our work, Sontakke et al. (2023) successfully use a VLM to provide reward signals for RL agents in robotics settings. However, they focus on specifying the reward with video demonstrations and only show basic results with natural language task descriptions.
In contrast to these works, we do not require any finetuning to use CLIP reward model, and we successfully train RL policies to achieve a range of complex tasks that do not have an easily-specified ground truth reward function.
CONCLUSION
We introduced method use vision-language models (VLMs) as reward models for reinforcement learning (RL), and implemented it using CLIP as a reward model and standard RI. algorithms. We used VLM-RMs to solve classic RL benchmarks and to learn to perform complicated tasks using a simulated humanoid robot. We observed a strong scaling trend with model size, which suggests that future VLMs are likely to be useful as reward models in an even broader range of tasks.
Limitations. Fundamentally, our approach relies on the reward model generalizing from a text description to reward function that captures what human intends the agent to do. Although the concrete failure cases we observed are likely specific to the CLIP models we used and may be solved by more capable models, some problems will persist. The resulting reward model will be misspecified if the text description does not contain enough information about what the human intends or the VLM generalizes poorly. While we expect future VLMs to generalize better, the risk of the reward model being misspecified grows for more complex tasks, that are difficult to specify in a single language prompt. Therefore, when using VLM-RMs in practice it will be crucial to use independent monitoring ensure agents trained from automated feedback act as intended. For complex tasks, it will be prudent to use a multi-step reward specification. e.g., by using VLM capable of having a dialogue with the user about specifying the task.
Future Work. There are many possible extensions of our approach that may improve performance but were not necessary in our tasks. For example, finetuning VLMs for specific environments is a natural next step to make them more useful as reward models. To move beyond goal-based supervi- sion, future VLM-RMS could encode videos instead of images. To move towards specifying more complex tasks, future VLM-RMs could use eaaoogeeeeebbeeddd VLMs.
For practical applications, it will be important to ensure robustness and safety of the reward model. Our work can serve a basis for studying the safety implications of VLM-RMs. For instance, future work could investigate the robustness of VLM-RMs against optimization pressure by RL agents.
More broadly, we believe VLM-RMs open up exciting avenues for future research to build useful agents on top of pre-trained models, such as building language model agents and real world robotic controllers for tasks where do not have reward function available.
9

---

Published as a conference paper at ICLR 2024
AUTHOR CONTRIBUTIONS
Juan Rocamonde designed and implemented the experimental infrastructure, ran most experiments, analyzed results, and wrote large parts of the paper.
Victoriano Montesinos implemented parallelized rendering and training to enable using larger CLIP models, implemented and ran many experiments, and performed the human evaluations.
Elvis Nava advised on experiment design, implemented and ran some of the experiments, and wrote large parts of the paper.
Ethan Perez proposed the original project and advised on research direction and experiment design.
David Lindner implemented and ran early experiments with the humanoid robot, wrote large parts of the paper, and led the project.
ACKNOWLEDGMENTS
We thank Adam Gleave for valuable discussions throughout the project and detailed feedback on early drafts, Jérémy Scheurer and Nora Belrose for helpful feedback early Adrià Garriga-Alonso for help with running experiments, and Xander Balwit for help with editing the paper.
We are grateful for funding received by Open Philanthropy, Manifund, the ETH Al Center, Swiss National Science Foundation (B.FG. CRSII5-17773 and 315230 189251), ETH project funding (B.F.G. ETH-20 19-01), and the Human Frontiers Science Program (RGY0072/2019).
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. In Advances Neural Information Processing Systems, 2022.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08003: 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108. 072558 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gy. arXiv preprint 1rXiv:1606.01 01540, 2016.
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback, 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 281 2818-2829, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017.
Yuchen Cui, Scott Nickum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foun- dation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, 2022.
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. arXiv preprint arXiv:2303.07280 07280. 2023.
10

---

Published as a conference paper at ICLR 2024
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open open-ended embodied agents with internet-scale knowledge. In Advances in Neural Information Processing Systems, 2022.
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference Fairness, Accountability. and Transparency, pp. 1747-1764, 2022.
Adam Gleave, Michael D Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differ- ences in reward functions. In International Conference on Learning Representations, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Muchine Learning, 2018.
Apoory Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but ef- fective: CLIP embeddings for embodied AI. In IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2022.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
Parsa Mahmoudich, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In International Conference on Machine Learning, 2022.
Shahbuland Matiana, JR Smith, Ryan Teehan, Louis Castricato, Stella Biderman, Leo Gao, and Spencer Frazier. Cut the carp: Fishing for zero-shot story evaluation. arXiv preprint arXiv:2110.03111, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 122iv:2112.10741 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP: Text- driven manipulation of StyleGAN imagery. In IEEE/CVF International Conference on Computer Vision, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 12-268):188, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06122, 2022.
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak. Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. arXiv preprint arXiv:2303. 16755, 2023.
11

---

Published as a conference paper at ICLR 2024
Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION- 5B: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and where pathways for robotic manipulation. In Conference on Robot Learning, 2022.
Sumedh A Sontakke, Jesse Zhang, Sébastien MR Amold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demonstration is enough to learn robot policies. In Advances in Neural Information Processing Systems, 2023.
Theodore Sumers, Kenneth Marino, Arun Ahuja, Rob Fergus, and Ishita Dasgupta. Distilling
internet-scale vision-language models into embodied agents. January 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Tewa2Reward: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309. 1489, 2023.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey, arXiv preprint arXiv:2304.cm 00685, 2023.
12

---

Published as a conference paper at ICLR 2024
A COMPUTING AND INTERPRETING EPIC DISTANCE
Our experiments all have goal-based ground truth reward functions, i.e., they give high reward goal state is reached and low reward if not. This section discusses how this helps to estimate EPIC distance between reward functions more easily. As side-effect, this gives us an intuitive understanding of EPIC distance in our context. First, let define EPIC distance.
Definition 2 (EPIC distance; Gleave et al. (2021)). The Equivalent-Policy Invariant Comparison (EPIC) distance between reward functions R1 and R2 is:
1
DEPIC
V2 - ((CR)))))
(4)
where -): is the Pearson correlation w.r. a given distribution over transitions, and C(R) is the canonically shaped reward, defined as:
((R)(s,a,s') = R(s, a') BRRRss A, - R(s,,, 7,,,,,,),,,,55)7
For goal- baled tasks, we have reward function R(s, a, s') R(s') 1s (s'). which assigns reward of to "goal" states and 0 to "non-goal states based on the task T. In our experiments, we focus on goal-based tasks because they are most straightforward to specify using image-text encoder VLMs. We expect future models to be able to provide rewards for more general class of tasks, e.g., using video encoders. For goal- goal-based tasks computing the EPIC distance is particularly convenient.
Lemma (EPIC distance for CLIP reward model). Let (CLIP,, CLIP be pair of state and task encoders as defined in Section 3.1. Let Reup be the CLIP reward function as defined in eq. (2), and R(s) 1T (s) be the ground truth reward function, where ST is the set of goal states for our task 1. Let μ be probability measure in the state space, let pl:;)) )) be the Pearson correlation under measure μ and Var(-) the variance under measure p. Then, we can compute the EPIC distance of CLIP reward model and the ground truth reward as:
DEPIC V2 /1 /11pp(Reup
p(Reup R),
--V Var(R) (Vac(Rer))
(CLIPI(I) (Jst (L))((()))))(()) Jsç (M))(()))
pl (RRur, R)
where SF = ST.
Proof. First, note that for reward functions where the reward of a transition (s, s') only depends on s', the aunonically-shaee reward simplifies
C(R) = R(s') + (E[((S)) - E(R(S') - -E(R(()))
R(s') E[R(S')].
Hence, because the Pearson correlation is location-invarian we have
C(C(()),C((B)) p(R,, R2).
= P(Y = 1) and recall that Var[Y] = p(1-p) 1-1-pp p). Then, we can simplify the Pearson correlation between continuous variable X and Bernoulli random variable Y as:
Cov [X,Y]
E E[X]R[y] B[X = 11p
p(X,))=
[] Var[Y]
Var X YY
V Var X Var[Y]
E[X = 11p E 11p² E[X = 0J(1 -p)p
Var X] /TVVao Var[Y[]
E[X|Y = 1pp(1 p) E 0] p)p
Var[XX] Var[Y]
Var
[E[X|Y = 1] E[X = 0))
Var X
13

---

Published as a conference paper at ICLR 2024
Combining both results, we obtain that:
Var(R)
(((())))),C(R))) V Var((Rrp))
Es [HCLP) -gsş [RCLi]]
our ground truth reward function of the form R(s) 11 (s) and we denote # as the optimal policy for reward function R, then the quality of Raw depends entirely on the Pearson correlation p(RruI R). If p(RcuP, R) is positive, the cosine similarity of the task embedding with embeddings for goal states & E ST is higher than that with embeddings for non-goal states E ST. Intuitively, p(RCLP, R)) is measure of how well CLIP separates goal states from non-goal states.
In practice, we use Lemma evaluate EPIC distance between a CLIP reward model and a ground truth reward function.
Note that EPIC distance depends on state distribution μ (see Gleave et al. (2021) for further discussion). In our experiment, use either uniform distribution over states (for the toy RL environments) or the state distribution induced by a pre-trained expert policy (for the humamoid experiments). More details how collected the dataset for evaluating EPIC distances can be found in the Appendix B.
B HUMAN EVALUATION
Evaluation on tasks for which we do not have a reward function was done manually by one of the authors, depending on the amount of time the agent met the criteria listed in Table 2. See Figures and for the raw labels obtained about the agent performance.
We further evaluated the impact of goal-baseline regularization on the humanoid tasks that did not 0, cf. Figure 8. In these cases, goal baseline regulariza-
succeed in our experiments with tion does not improve performance. Together with the results in Figure 4a, this could suggest that goal-baseline regularization more useful for smaller CLIP models than for larger CLIP models. Alternatively, it is possible that the improvements regularization are too small to lead to noticeable performance increases in the trained agents for the failing humanoid tasks. Unfortunately, a more thorough study of this was infeasible due to the cost associated with human evaluations.
the reward model obtained by goal-baseline
Our second type of human evaluation is compute the EPIC distance of reward model to pre- labelled set states. To create a dataset for these evaluations, we select all checkpoints from the training with the highest VLM-RM reward of the largest and most capable VLM used. We then collect rollouts from each checkpoint and collect the images across all timesteps and rollouts into a single dataset. We then have human labeller (again an author of this paper) label each image according to whether it represents the goal state or not, using the same criteria from Table 2. We use such a dataset for Figure 4. Figure shows more detailed breakdown of the EPIC distance for different model scales.
C IMPLEMENTATION DETAILS & HYPERPARAMETER CHOICES
this section, we describe implementation details for both our toy RL environment experiments and the humanoid experiments, going into further detail on the experiment design, any modifications make to the simulated environments, and the hyperparameters we choose for the RL algorithms we use.
Algorithm shows pscudocode of how we integrate computing CLIP rewards with batched RL algorithm, in this case SAC.
C.I CLASSIC CONTROL ENVIRONMENTS
Environments. We use the standard CartPole and ainCar environmemn environments implemented in Gym, but remove the termination conditions. Instead the agent receives negative reward
14

---

Published as a conference paper at ICLR 2024
Task
Condition
Kneeling
Agent must be kneeling with both knees touching the floor. Agent must be losing balance nor kneeling the air. Agent seated down in the lotus position. Both knees are on the floor and facing outwards, while feet must be facing inwards.
Lotus position
Standing up
Agent standing up without falling.
Arms raised Doing splits
Agent standing up with both arms raised.
Agent the floor doing the side splits. Legs stretched the floor.
Hands hips
Agent standing up with both hands the base of the hips. Hands must not be on the chest.
Arms crossed
Agent standing up with its crossed. If the agent has its hands just touching but not crossing, it is not considered valid. Agent standing up touching the floor only with one leg and with- out losing balance. Agent must not be touching the floor with both feet.
Standing on one leg
Table 2: Criteria used to evaluate videos of rollouts generated by the policies trained using CLIP rewards on the humanoid environment. A rollout is considered a success the agent satisfies the condition for the task at least 50% of the timesteps, and failure otherwise.
Kneeling
Lotus position
Standing up
Arms raised
100%
100%
100%
100%
Percentage Eghodes
60%
60%
60%
60%
20%
20%
20%
20%
25
75
100
50
75
100
25
50
75
100
0
25
75
100
Arms crossed
Hands on hips
one leg
Doing splits
100%
100%
100%
100%
Percentage Eghoodes
80%
80%
80%
80%
60%
60%
60%
60%
-0%
40%
40%
40%
20%
20%
20%
20%
25
100
Successful Steps per Episode
Successful Steps per Episode
Succeseful Steps per Episode
Successful Steps per Rpisode
Figure 5: Raw results of our human evaluations. Each histogram is over 100 trajectories sampled from the final policy. One human rater labeled each trajectory in one of five buckets according to whether the agent performs the task correctly 0, 25, 50, 75, or 100 steps out of an episode length of 100. To compute the success rate in the main paper, we consider all values above 50 steps "success".
15

---

Published as a conference paper at ICLR 2024
RN50
VIT-L-14
VI- 14
biThigG-
Porceragg of Expicddss
100%
Percentage Episodes
100%
Percenagss of Eppioodes
100%
1008
30%
30%
30%
Percentage
50%
0%
0%
0%
0%
0
25
50
75
100
75
100
1
25
50
75
100 Epiiode
75
100
Socoessful
Episode
Successful Stops Episode
Sucoessful
Successful Stops Episode
Figure 6: Raw results our human evaluations for the model scaling experiments. The histograms computed the same way in Figure 5, but the agents were trained for differently sized CLIP models on the humanoid "kneeling" task. As the aggregated results in Figure 4c in the main paper suggest, there is stark difference between the agent trained using the ViT-H-14 model and the ViT ViT-bigG-I- model.
goal goal state
goal state (meun)
not goal (mean)
ViT-L-14
ViT-H-14
ViT-bigG-14
Inequency
Frequency
Frequency
Prequeccy
100
100
1010
100
0-
Standurdized Reward
Standardived Reward
Standardized Reward
Standardized Reward
The rewards distributions of (human labelled) goal states vs. non-goal states become more separated with the scale of the VLM. We show histograms of the CLIP rewards for differently labelled states in the humanoid "kneeling" task. The separation between the dotted lines, showing the average of each distribution, is the Pearson correlation described in Appendix A. This provides a clear visual representation of the VLM's capability.
Figure 7:
Task
Goal Prompt
Baseline Prompt
CartPole
"pole vertically upright on top of the cart"
"pole and cart"
MountainCar
"a car the peak of the mountain, next the yellow flag"
"a car in the mountain"
Kneeling
"a humanoid robot kneeling"
"a humanoid robot"
Lotus position
"a humanoid robot seated down, meditat- ing in the lotus position"
"a humanoid robot"
Standing up Arms raised
"a humanoid robot standing up"
"a humanoid robot"
Humanoid
"a humanoid robot standing up, with both arms raised"
"a humanoid robot"
Doing splits
"a humanoid robot practicing gymnastics, doing the side splits"
"a humanoid robot"
Hands on hips
"a humanoid robot standing up with hands
"a humanoid robot"
on hips"
Arms crossed "a humanoid robot standing up. with its
"a humanoid robot"
arms crossed"
Standing on one leg
"a humanoid robot standing up on one leg"
"a humanoid robot"
Table 3: Goal and baseline prompts for each environment and task. Note that we did not perform prompt engineering, these are the first prompts we tried for every task.
16

---

Published as a conference paper at ICLR 2024
Arms crossed
Hands on hips
Standing one leg
100%
100%
100%
0 0.0
0r 0.0
= 0.0
80%
80%
80%
Percentage
60%
60%
60%
40%
40%
40%
20%
20%
20%
0%
0%%
0%
25
50
75
100
75
100
100%
100%
100%
a 0.2
0.2
0.2
80%
80%
Percentage
60%%
60%
60%
40%
40%
40%
20%
20%
0%
0%
0%
25
50
75
100
25
50
75
100
75
100
100%
100%
100%
0.4
0.4
0.4
80%
80%
80%
Percentage
60%
60%
60%
40%
40%
40%
20%
20%
20%
0%
0%
0%
25
50
75
100
25
50
75
100
25
50
75
100
100%
100%
100%
a 0.6
0.6
= 0.6
80%
80%
80%
Percentage
60%
60%
40%
40%
40%
20%
20%
20%
0%
25
50
75
100
25
50
75
100
0
25
50
75
100
Successful Steps per Episode
Successful Steps per Episode
Successful Steps per Episode
Figure 8: Human evaluations for evaluating goal-baseline regularization in humanoid tasks. The his- tograms are computed the same way as in Figure 5. We show the humanoid "arms crossed", "hands on hips", and "standing shows one of the tasks and the rows show regularization strength values a 0.0, 0.2, 0.4, 0.6. The performance for 0 and a 0.2 seems comparable and larger values for a degrade performance. Overall, we don't 't find goal- baal-naseline regularization leads to better performance on these tasks.
leg" tasks that failed in experiments with
0. Each column
17

---

Published as a conference paper at ICLR 2024
Algorithm SAC with CLIP reward model.
Require:
Task description I, encoders CLIP,, and CLIP,, batchsize B
Initialize SAC algorithm
3i <- (L)
D Precompute task embedding Initialize buffers
B + [I D +
repeat
Sample transition (st, a, 1 using current policy
Append (st. 8t+1) unlabelled buffer B
[B] 2 B then
for (st, St,0,,rea 1) in do
In practice this loop batched > Compute state embedding Compute CLIP reward
3a + --(P)((((s))
ReLPP: + 27 /./ -|
Optionally apply goal-baseline regularization (Definition 1)
Remove (8t, (e2 8141) from unlabelled buffer B
Append (st, at, RCLIPE, St+1) to labelled buffer '
Perform standard SAC gradient step using replay buffer D
until convergence
for dropping the pole in CartPole and a positive reward for reaching the goal position in MountainCar: We make this change because the termination leaks information about the task completion such that without removing the termination, for example, any positive reward function will lead to the agent solving the Cart.Poe le task. As result of removing early termination condi- tions, we make the goal state in the MountainCar an absorbing state of the Markov process. This ensure that the estimated returns are affected by anything policy might do after reaching the goal state. Otherwise, this could, in particular, change the optimal policy or make evaluations much noisier.
RL Algorithms. We use DQN (Mnih et al., 2015) for CartPole, only environment with discrete action space, and SAC (Haarnoja et al., 2018), which is designed for continuous environ- ments, for MountainCar. For both algorithms, saablebbaselines: (Raffin et al., 2021).
standard implementation provided by
DQN Hyperparameters. We train for 3 million steps with fixed episode length of 200 steps, where we start the training after collecting 75000 steps. Every 200 steps, we perform 200 DQN updates with a learning rate of 2.3e 3. We save a model checkpoint every 64000 steps. The Q-networks are represented by 2 layer MLP of width 256.
SAC Hyperparameters. We train for 3 million steps using SAC parameters 7 = 0.01, Y = 0.9999, learning rate 10 4 and entropy coefficient 0.1. The policy is represented by 2 layer MLP of width 64. All other parameters have the default value provided by stable-basel ines3.
We chose these hyperparameters in preliminary experiments with minimal tuning.
C.2 HUMANOID ENVIRONMENT
For all humanoid experiments, we use SAC with the same set of hyperparameters tuned on prelimi- nary experiments with the kneeling task. We train for 10 million steps with an episode length of 100 steps. Learning starts after 50000 initial steps and we do 100 SAC updates every 100 environment steps. We use SAC parameters T = 0.005, 7 = 0.95, and learning rate 10-4, We save a model checkpoint every 128000 steps. For our final evaluation, we always evaluate the checkpoint with the highest training reward. We parallelize rendering over GPUs, and also use batch size B = 3200 for evaluating the CLIP rewards.
18


Reviews:
[{"review_id": "88COuHCem5", "reviewer": "Reviewer_kRKZ", "summary": "This paper proposes the VLM-RM method, which uses a pre-trained CLIP model as a reward for training vision-based agents. VLM-RM models the reward as the alignment between a language instruction specifying the task and the current observation image while regularizing relative to a baseline prompt to remove irrelevant information from the reward. VLM-RM is empirically validated in classic control tasks and reaching various poses in MuJoCo.", "questions": "- What exact values are selected for $\\alpha$ in Fig. 4a? I recommend putting an indicator in this plot to show which values are used. \n- What is the optional context $c$ in Eq. 1 used for? I don't see it referred to later in the paper.\n- Can the paper include the RL learning curves for all the experiments showing the number of samples versus either the predicted CLIP return or a ground truth metric, like true reward in the classic control tasks or EPIC in the humanoid tasks? Given there is a ground truth reward in the classic control tasks, can the authors also include that as a reference for the VLM-RM learning curves? It is important to see the training stability and efficiency under the CLIP model reward.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The proposed VLM-RM method is simple and works well. VLM-RM provides a way to use CLIP models easily as a zero-shot reward signal from text without having to finetune the CLIP model. Furthermore, the prompts used for the reward model in VLM-RM are simple and intuitive, highlighting the ease of applying VLM-RM.\n- The experiments show the applicability of VLM-RM. In CartPole and MountainCar, the CLIP reward predictions align with the true success state, and we see the goal-baseline regularization helping for MountainCar. VLM-RM is also able to learn a variety of behaviors in the high-dimensional humanoid task. While the success of the humanoid experiments was evaluated by just one of the authors, the videos on the website convincingly show the correct behaviors.\n- The work clearly explores the limitation of VLM-RM that the environments should be visually realistic for CLIP to provide a meaningful learning signal. In MountainCar, realism produces better alignment between CLIP and the true success state (Fig. 2c). In Fig. 3, we see the impact of modified textures and the camera placement on performance. I believe the limitation around the realism of the observations is more a limitation of the environments rather than VLM-RM. \n- The work shows evidence that VLM-RM scales to better performance with larger, more capable CLIP models. Fig. 4, shows the humanoid kneeling task is only possible with the largest CLIP model.", "weaknesses": "- The paper states that the \"CLIP rewards are only meaningful and well-shaped for environments that are photorealistic enough for the CLIP visual encoder to interpret correctly,\" yet the paper focuses on control environments without realistic visuals. Why not instead focus on more visually realistic simulation benchmarks and not have to modify the environment to make the rendering more realistic to fit the algorithm? \n- It's unclear if the goal-baseline regularization is necessary. The primary experiments in Table 1 don't use any goal-baseline regularization. The analysis in Fig. 4a shows the maximum or minimum amount of goal regularization to be best. Additionally, to what degree can the same effects of the goal-baseline regularization be incorporated in the goal prompt? A more detailed goal prompt can just specify to ignore the irrelevant information. \n- While the paper shows the strong final results of VLM-RM, the RL training stability of VLM-RM is unclear. Is it easy for RL methods to learn from the VLM-RM reward? How does this compare to a ground truth reward? See my further comments under the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "61j0M65NUW", "reviewer": "Reviewer_CqGG", "summary": "This paper investigates the use of pre-trained visual-language models for specifying reward models for reinforcement learning agents via natural language. In its simplest form, a single sentence text prompt is passed to a CLIP model along with the pixel observation of the agent to produce a score that translates to the reward. A more sophisticated method is also proposed, labelled goal-baseline regularisation, with the intention of removing irrelevant information about the environment from the observation embedding. Specifically, it involves projecting the observation embedding on to the the line spanned by the task embedding and a baseline embedding, the latter of which originates from a generic description of the environment. The CLIP alignment is then calculated using an interpolation of the original observation embedding and the projection embedding. The approach is validated in the context of two simpler task, CartPole and MountainCar, as well as more complex ones in the Humanoid environment. For the first two, it is shown that the CLIP-induced reward aligns well with the ground-truth reward (which is not provided to the agent) and that agents trained with the former can learn to perform well in the initial task. In the Humanoid environment, agents are trained on complex tasks with no available ground-truth reward, and are shown to perform well in most tasks as judged by a human evaluation. It is shown that modifying the backgrounds and textures of the visual observations to make them more realistic can have a large effect on the success of agents. Furthermore, scaling experiments are run that demonstrate that using more powerful VLMs leads to better agent performance.", "questions": "- Could the authors elaborate on the ‘minimal prompt engineering’ required to find the single sentence prompts for each task? Also, are there any results demonstrating the robustness of the method to syntactic variations in the prompt?\n- Suggested citation if authors believe it is relevant (using VLMs to improve exploration in RL): https://openreview.net/forum?id=-NOQJw5z_KY", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles the important and challenging problem of reward specification in reinforcement learning and effectively demonstrates a promising approach using VLMs.\n- The method is clearly explained and theoretically simple to implement.\n- There is valuable insight derived from the ‘tricks’ that get the method to work, namely the goal-baseline regularisation and the re-rendering of the observations to make them more realistic.\n- Interesting idea to evaluate performance on tasks with no ground truth reward via EPIC distance with human-evaluated reward.\n- The scaling experiments that show that performance improves with the size of the VLM are very encouraging for the scalability of the method.", "weaknesses": "- Risk of bias in human experiments. While this is duly acknowledged in the paper, the fact that the human analysis was conducted by one of the authors of the paper is a potentially strong source of bias in the human evaluation.\n- It is stated that ‘minimal prompt engineering’ was required to find the right single text prompt for the reward function but the process for discovering the right prompt and the robustness of the method with respect to noise in the prompt / semantic variations is not discussed.\n- There is some evidence in the experiments of reward misspecification with the VLM, a theoretical limitation acknowledged by the authors in the conclusion. It is noted that when the regularisation strength is high for realistic MountainCar that the CLIP reward function reflects the shape of the mountain. The fact that going up the small hill to the left gives high CLIP reward is useful because it encourages a policy where the mountain car oscillates in the valley until it has enough momentum to reach the top right (where the ground truth reward lies), but this is somewhat of a fortunate coincidence as there is actually a mismatch here with the ground-truth reward, which only rewards reaching the top right.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gbDquq1n5M", "reviewer": "Reviewer_rUuN", "summary": "Using VLM to generate intrinsic rewards is a new venue since manually specifying reward functions for real-world tasks is often infeasible. In this paper, the authors propose VLM-RM, a method for using pre-trained VLMs as a reward model for vision-based RL tasks. In more detail, they introduce a baseline as regularization for the final reward. The baseline aims to remove the irrelevant part in the CLIP embeddings. This irrelevant info means the natural language description of the environment setting in its default state, irrespective of the goal. As for the experiment part, the authors validate their method in the standard CartPole and MountainCar RL benchmarks.", "questions": "None.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Designing a more RL-friendly reward is an interesting research direction. It can help the generalization of policy learning. \n2. Removing the unimportant part from the CLIP embeddings seems to be a reasonable direction for improving the performance of VLM rewards.\n3. The paper is easy to follow.", "weaknesses": "1. The contribution is limited and the motivation is not clear. Intuitively, I can also claim that the background information of the language description is not useless since different instructions may have different meanings in different environments. I encourage authors to provide more theoretical analysis to support the effectiveness of the proposed baseline if any. In addition, simply proposing one regularization is not novelty enough unless the authors can prove it can boost performance in a wide range of tasks. \n\n2.  The experiments are not sufficient. Taking Figure 3 as an example, more seeds and more tasks should be included for baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IPjmrwcWEf", "reviewer": "Reviewer_13eD", "summary": "The paper proposes to use pre-trained CLIP embeddings as 0-shot language-conditioned rewards for RL. It further proposes a \"baseline regularization\" technique that aims to normalize rewards by removing features from the reward function that are irrelevant to the task. The paper demonstrates learning of humanoid posing tasks for which ground truth rewards are hard to specify and discusses several approaches for evaluating the reward models in such scenarios without groundtruth rewards.", "questions": "- how does the evaluated approach for Table 1 differ from prior works like Mahmoudieh et al that also use 0-shot CLIP embeddings for language-conditioned reward computation?\n\n- in Mahmoudieh et al. the non-finetuned model does not work at all as a reward function -- is the fact that the model in the submission works well purely based on the chosen task domain? \n\n\n## Summary of Review\nOverall, the paper shows some interesting learned behaviors with a simple method, but the novelty over prior work is not well demonstrated in the experimental results. No comparisons to baselines are performed and the introduced algorithmic changes are not used in the experiments. As such, I do not recommend acceptance of the paper in it's current form. I do think the paper has potential and I encourage the authors to\n- add experimental results that demonstrate that the proposed baseline regularization leads to improved policy performance\n- add experiments on at least one additional non-toy domain, e.g. a robotic manipulation task\n- demonstrate experimentally the correlation between the offline EPIC reward distance metric and policy performance once trained on the reward function\n\n# Post-Rebuttal Comments\n\nThank you for answering my review!\nI want to emphasize that I do not devalue empirical works in any way -- there are lots of examples for very high impact empirical works that shed light on shortcomings of the existing literature and provide practical implementation advice to \"make things work\".\n\nThe critical elements for such papers are that they provide technical \"tricks\" that are specific to the core algorithm yet demonstrate that they help across a wide range of applications of this algorithm (ie are not overfit to one task / environment etc.).\n\nBased the on the rebuttal the paper claims three main “tricks” to make things work:\n   - (1) model size —> this one makes sense and is validated with experiments\n   - (2) reward formulation —> this one is not validated experimentally\n   - (3) choice of RL optimizer algorithm —> this one seems orthogonal to the core question of how to formulate reward functions and is also not experimentally validated\n\nFurther, I reiterate my concern that all experiments are conducted in a single non-toy environment — this is in my opinion insufficient to claim that the paper introduces “a general way to make CLIP-rewards work”.\n\nIf the main scope of the paper should be “tricks to make CLIP-rewards work”, the writing would need to change substantially, the experimental evaluations mentioned above would need to be added and more non-toy enviornments would need to be evaluated.\n\nFinally, it remains confusing that the main technical contribution is not used in the main experimental evaluation and the rebuttal did not provide a convincing reason for this.\n\nThus, in summary I do not see my concerns addressed by the rebuttal and maintain my score.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem of learning reward functions for RL is important, since for many applications, e.g. real world robot learning, it is hard to specify ground truth reward functions without serious instrumentation of the environment. The paper shows impressive results on several humanoid posing task that require a level of granularity unseen in previous 0-shot CLIP reward papers.\n\nThe proposed \"baseline regularization\" technique for learned rewards is interesting and novel to my knowledge. I also found the discussion of using EPIC distance to offline-score the reward model interesting.\n\nOverall, the paper is well-written and easy to follow. I appreciate that the authors provide qualitative videos of the learned behaviors.", "weaknesses": "The proposed approach of using a CLIP model 0-shot to specify language-conditioned rewards is not novel, and the relevant references are cited in the related work section (Cui et al., Mahmoudieh et al.). The main technical novelty is in proposing the baseline regularization. However, the main experiment on learning humanoid posing tasks does not use this regularization at all. As such, it appears that the used approach in these experiments is identical to those proposed in prior work (Mahmoudieh et al), just applied on a different task.\n\nIn the same vein, the paper is lacking any comparison to prior work. This may be because the method is identical to prior work, just applied on a different domain? I do think it is valuable to show that CLIP-based rewards can work 0-shot in more challenging domains, but without technical novelty this contribution seems insufficient to warrant acceptance.\n\nFigure 4 shows that the proposed technical novelty of the baseline normalization does improve EPIC distance between the learned and a human-provided reward function. However, the paper lacks evidence that this improvement in EPIC distance indeed translates in an improvement in policy performance once trained on the reward function (the performed evaluation is a single step function from 0% to 100% success, so it's not informative enough to show good correlation, an environment with a more finegrained reward function may be required).\n\nA minor weakness is, that the experiments were performed in a clean, simulated environment, without distractors or other moving objects. Thus it is unclear whether the reward functions would be robust to such distractors (though admittedly prior works also did not evaluate realistic scenes).\n\nFinally, albeit impressive, the experiments are only performed on one non-toy environment (the humanoid). So even if the authors demonstrated that the proposed baseline regularization helps to improve policy performance, it would be good to add evaluations on at least a second non-toy environment, e.g. a robotic manipulation task, to prove that the effects are consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IBq2QyxNia", "reviewer": "Reviewer_Q69H", "summary": "The paper investigates using pretrained vision-language models (VLMs) as zero-shot reward models to specify tasks via natural language. Specifically, they use CLIP models to train a MuJoCo humanoid to learn complex tasks, e.g., kneeling, doing the splits, and sitting in a lotus position. Besides, they also propose a technique called goal-baseline regularization to improve performance. They also study the scaling effect of VLMs for being a reward function.", "questions": "When using the VLMs as the reward function for a task, one may need to consider if one should use it as a dense reward signal or a terminal reward, can the authors elaborate on how they considered this question?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Experiments and conclusion** The authors conduct extensive and solid experiments from the standard CartPole and MountainCar environments to the relatively complex MuJoCo humanoid environment to validate a natural idea of utilizing VLMs as a reward function to learn the agent. Besides, some in-depth studies are also well conducted, such as the scaling effect of VLMs, and various humanoid tasks specified by human language. The conclusion is interesting. \n\n**Method**  Goal-Baseline Regularization is a novel and smart technique proposed in this work to project out irrelevant information about the observation and improve the practicability of the CLIP-based reward model.\n\n**Writing**  The writing is clear and easy to follow; each contribution is explicitly listed and highlighted, and the section/subsection titles make navigation effortless.", "weaknesses": "**Real world impact**  Although the work shows the potential of VLM-RMs, all conducted experiments have been based on relatively simple and synthetic tasks, rather than from the real world. To further increase the potential impact of this work, it would be beneficial to consider some more tasks with real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)