Zero-Shot Reward Specification via Grounded Natural Language
Authors: Parsa Mahmoudieh
Sayna Ebrahimi
Deepak Pathak
Trevor Darrell
Venue: ICLR 2022 Submitted (2022)

Abstract: Reward signals in reinforcement learning can be expensive signals in many tasks and often require access to direct state. The alternative to reward signals are usually demonstrations or goal images which can be labor intensive to collect. Goal text description is a low effort way of communicating the desired task. Goal text conditioned policies so far though have been trained with reward signals that have access to state or labelled expert demonstrations. We devise a model that leverages CLIP to ground objects in a scene described by the goal text paired with spatial relationship rules to provide an off-the-shelf reward signal on only raw pixels to learn a set of robotic manipulation tasks. We distill the policies learned with this reward signal on several tasks to produce one goal text conditioned policy.

Under review as a conference paper at ICLR 2022
ZERO-SHOT REWARD SPECIFICATION VIA GROUNDED NATURAL LANGUAGE
Anonymous authors Paper under double-blind review
ABSTRACT
Rewards in reinforcement learning are difficult design for tasks and often require access to underlying state information. The alternatives to reward signals are usually demonstrations or goal images which can labor-intensive to collect. Describing the goal as a text description is low effort way of communicating the desired task to an agent. However, goal-text conditioned policies so far have been trained in scenarios either with rewards that assume access to states or with labeled expert demonstrations. In this work, we devise a model that leverages CLIPP ground objects in scene described by the goal-text paired with spatial relationship rules to provide an off-the-shelf reward signal using only raw pixels as input. We distill the policies learned with this reward signal several tasks obtain single text conditioned policy that can be applied to solve new tasks at deployment, We show the effectiveness of this framework on of robotic manipulation tasks.
INTRODUCTION
Previous efforts have explored image-based goal specification, with significant successes in visual navigation and manipulation tasks (Pathak et al., 2018; Nair et al., 2018; Fu et al., 2018; Singh et al., 2019). Yet existing image-based goal specifica- tion paradigms are limited because they are typ- ically limited to particular scene instance an environment, whereas semantic goal comprises multiple possible scene configurations. Reinforce- ment learning offers one of the most appealing premises in the study of AI: from a reward signal alone, algorithms which learn optimal policies that maximize expected reward can learn to perform navigation, dexterous manipulation, and host of other impactful tasks. However, discovering or specifying reward function for given task is often a very challenging problem, especially when one is considering agents that can learn from un- instrumented environments, e.g., from raw image observations alone. We wish have an agent that can learn purely from pixels, with no access the underlying state of the environment at any point during learning or task execution. Achieving this goal without access to an instrumented reward function has been exceedingly challenging.
Training Phase
Goal text
Rewards
Actions
ZSRM
Agent
Env
Train ....
Distill
gool 1,...
Multi
Deployment Test Phase
Actions
New goal text
Multi
Eny
Obs
We devise a method that learns tasks using only goal text from user and images from the envi- ronment training time. We train several agents several tasks and distill their policies into multi-task policy that is conditioned on goal text and current ob- servation. The distilled policy is able to generalize to unseen tasks at test time. We assume access environment reward, state, demonstrations, goal images.
Figure 1:
One can use image-based reward specification to cause robot agent navigate to particular chair next to a specific tall plant, but that agent may not always succeed the generic task of "go to chair next to a tall flowering plant": e.g., if the goal specification image shows red chair next

---

Under review as a conference paper at ICLR 2022
E
It
State Extractor
SP1 ...
E CLIP
Reward Generator
Obj Noun Phrases s...k
g
Goal Description
Trivial Parser
Desired Obj Spatial Relationships
We compute reward of image observation conditioned goal language description, with no access state, demonstrations, goal images. We first parse the goal description into object noun phrases desired object spatial relationships. We then leverage CLIP's image and language encoders Ey and Ez. to locate the object noun phrases in the image observation. Using the object states and the desired object spatial relationships, reward is generated to train the agent for the desired task in its environment.
Figure 2:
plant with yellow flower the agent may navigate away from flower, depending on the model's underlying image representation. To be sure that the true goal is properly specified irrespective of the invariances of the model's underlying perceptual representation, a user may have to provide a set of goal image examples that cover the variation of the target concept, potentially very expensive undertaking collect generate.
with blue chair next to red
We advocate semantic reward specification via grounded natural language, where a user describes a goal configuration in the world using natural language description referring entities in the world. This direction has long been a "holy grail" of AI research and presumed capability of AI science fiction-the the ability to instruct robot with natural language-yer yet attempts have been limited by the state of the art in grounded language perception. It also falls under the general umbrella of leveraging large-scale passive data bootstrap embodied learning, where we rely on language as a means to provide necessary reward shaping missing in visual data alone.
Several previous efforts train reward functions or policies that take natural language as input for goal description (Oh et al., 2017; Bahdanau et al., 2018; Zhou & Small, 2020; Goyal et al., 2020; Fu et al., 2019; Hermann et al., 2017; Shao et al., 2020). They all however rely on reward signals that have access state of the system demonstrations of the task distribution they are training There works that use human videos to learn reward functions to train their agent with (Shao et al., 2020; Sermanet al., 2018; 2016), but they require curated dataset of humans performing the tasks.
Recently however, the advent of large-scale multimodal training data together with large capacity language and vision deep learning models has significantly advanced the state of the art. A steady series of innovations have advanced grounded language modeling, from early work on multimodal translation and fusion models (Barnard et al., 2003; Quattoni et al., 2007; Guadarrama et al., 2016), to large-scale joint embedding models (Frome et al., 2013; Radford et al., 2021). to the plethora of multimodal transformer models currently under investigation (Su al., 2019; Lu et al., 2019: 9; Chen et al., 2019; Hu & Singh, 2021). CLIP, in particular, demonstrated transformative advance on zero-shot object recognition (Radford et al., 2021).
We could specify goal state to robot by simply offering a description of the goal configuration natural language and using the CLIP embedding dot product with an observed image evaluate proximity to goal state. Surprisingly, this can work in simple cases, for examples see the top example Figure 5. But for more complex goals performance can poor, especially with complex captioned scenes involving spatial relationships and domains that differ from web images (as shown in (Radford et al., 2021)) and the bottom example in that figure.
To overcome these limitations, we propose factored grounding of 'what" vs 'where/how' aspects of goal state (Figure 2), and offer a novel apatial-salience scheme for the former. We argue that existing (c.g., CLIP-like) models can be used to ground the what aspects of goal quite effectively, including appropriate attribute and concept-level generalization, while separate where/how module ground spatial relationship aspects of goal configuration. After training many tasks with this
2

---

Under review as a conference paper at ICLR 2022
reward scheme we learn language conditioned policy that execute new tasks in zero-shot fashion - from a natural language description of task without having to train a new policy for every new task.
In summary, our contributions are as follows:
1. A zero-shot reward function to learn set of robotic manipulation tasks raw pixels without access to state. (Zero-shot in the sense that no goal images or human demonstrations are provided at training time or test time.)
2. A distilled danguage-conditionn multi-task policy on several tasks learned using the zero-shot reward function which is able to deploy new unseen tasks.
2
METHOD
2.1 BASE ZERO-SHOT REWARD MODEL: DOT-PRODUCT VISUOLINGUISTIC REWARD
Our base model is the most intuitive way use the CLIP model to compute reward. Our base model simply computes reward by taking a dot product between the goal language input feature and image observation feature through CLIP's language and image encoders respectively: Y2 = (It) (g) A subset of tasks can be learned with this reward model. We visualize the limits of this model on two environments in Figure 5.
2.2 FULL ZERO-SHOP REWARD MODEL: SPATIALLY GROUNDED VISUOLINGUISTIC REWARD
CLIP-Saliency for Phrase Grounding We have developed a simple method which factors phrase grounding and spatial relationship processing, which can be used to specify reward function that operates only on image content using natural language description. Figure 2 illustrates our full model leverages CLIP's encoders in a very different way than the base model. Our model first parses the goal language description g into object noun phrases ...o. and the desired object interaction. The object noun phrases are passed through CLIP's language encoder and the current image observation is passed through CLIP's image encoder. We use the language encodings as class embeddings of each object noun phrase to perform a saliency analysis (using Grad-CAM (Selvaraju al., 2017)) on the image encoding of the observation on the last convolutional layer (specifically, the ReLU layer of CLIP's ResNet-50 backbone). Saliency models such as Grad-CAM generally output heatmap of the features that indicate class exists in the current image input. The state extractor in Figure 2 computes the "state" of each object using the argmax of the saliency heatmap. See Figure 3.
Spatial Grounding Heuristics Our model determines task completion using a spatial grounding rule based reward function that
Table 1: Spatial Grounding Heuristics
takes the object pixel coordinates of one or more camera views and desired spatial rela- tionship as input. Our spatial heuristics can
Spotial Relationship
Roward Criteria
Ohjil on
Ohj2
O
Obji on the right of Obj2
o >
be used describe a broad set of manipula- tion tasks to push or place objects to semantic locations. A reward of 1 is given to the RL algorithm for every timestep that the condi-
Objil on top of Olij
10 - of <
& of <o, < 03 tez
Otji below Okj2
1 02 < & o < te2
Obji between Ubj2, Ubj
mmn oz) of maa(0? 0g)
Obji in fromt Dbj2
0 0
Otji behind Okg2
o 0
tions of the reward criteria are met for a given spatial relationship. The oracle reward uses true x.y,z spatial positions of the objects to determine if the desired spatial relationship is reached and also outputs to the RL algorithm for every timestep the conditions of the task are met.
Obji cluse to Obj2
[0 0a <e
Obji inside Oet2
[0] 02
Our spatial heuristics are fully defined in Table 1. The first set (left of, right of, on top of, below, and in between) assume coordinates in a front camera view and the semanties are defined in that camera view with positive in the upward direction and positive x in the right direction. The second set (in front of & behind) has access left camera view with coordinate x2 pointing towards the right
3

---

Under review as a conference paper at ICLR 2022
Figure 3: a) Mask R-CNN object detection results b) Mask R-CNN dctection results far view c) Mask R-CNN instance segmentation results on far view d) Grad-CAM result with 'red block' text input (white being highest intensity) e) Grad-CAM result with 'yellow block' text input (white being highest intensity)
which is towards the front in the first camera view and y2 pointing upward similar to the first camera view. The second camera is needed to know if the object is placed in the front or behind another object correctly. The third set (close to & inside of) require access degree downward tilt towards the ground. This camera view is needed to see if the object is getting closer to another object in two orthogonal directions at once where as a left only or front view only allows you to determine one dimension of closeness. For "inside of"" 45 degree camera helps the agent see if the object going inside another object without occlusion. The threshold for "inside of"" is much smaller than for "close to" since the centroid can come much closer when an object goes inside container object.
front camera view with 45
2.3 LANGUAGE CONDITIONED MULTI-TASK POLICY
Now that we have a method for learning tasks in a zero-shot reward fashion, we would like to be able to not require training multi-task policy that takes in the goal text description of an unseen task and executes the task without needing any samples from the environment.
policy for every task and ideally have language conditioned
To approach this goal, we first learn several tasks via reinforcement learning using our zero-shot reward model. We then create large dataset of the rollouts of those polices and pair each trajectory with the goal text description of the tasks it was trained to learn. We then use behavioral cloning (supervised learning for predicting actions) to learn policy that takes images and text goal task description as input and actions target outputs. We use CLIP's language encoder as the text goal embedding that is fed to the multi-task policy. addition we use image augmentation techniques to aid behavioral cloning in learning more robust policies.
3 EXPERIMENTS
3.1 PHRASE GROUNDING AND BASE HERO-SHOT REWARD MODEL VISUALIZATION
Object detectors are usually not used off the shelf and need to be fine-tuned with domain data. In Figure we show pretrained Mask R-CNN (He et al., 2017) outputs on different camera views for the block stack en- vironment. As you can in the first subfigures, the blocks are not proposed as objects with Mask R-CNN from both far and close camera views. This demonstration of the need for in-domain fine-tuning of object detectors to work in the environment you want to use. Our Grad-CAM output from CLIP however, highlights ex- actly the objects are interested in from the object noun phrases that describes each object in the last two subfigures.
way to extract object states, however, they
Double Inverted Pendulum
Calealyrt
Dry Timaatag
Figure 4: We showcase the Base Reward Model in the Double In- verted Pendulum doing slightly hetter than oracle reward. We av- erage results over 3 seeds per task.
The CLIP model has been trained much larger dataset than Mask R-CNN and has language component that allows us to request for phrase of interest. With Mask R- CNN however, even if the object proposals were good, it would not necessarily classify objects it has not been trained for and therefore
the location of the object
4

---

Under review as a conference paper at ICLR 2022
Reseent Besgoree Inverted Penoulum
6275
0
Image index
Easo Sewa 'Imuge of yullon oiuect lie is blue sbject'
2
3
4
Image index
Figure 5: We visualize the results of the base reward model which is trivial dot product between the goal language description and image observation. The top row (green box) displays successful utilization of base reward model and bottom row (red box) shows a failure case, The x-axis represents image index.
SawyerSimRobbt-PPshinnn
FetchSim Stabot-Stacking
Placs
Aug Regarive Centrold oN De2
AUr Scurc
40
28
Use T'mesteps
(gt
(dot-product visuolinguistic
CuriesityRLL RL
Ours-ful (spatially grounded visuolinguistic reward)
Figure We showcase our Full Reward Model performing almost as well as oracle reward on SawyerSimRohor. Pushing (pushing blue puck close to a yellow puck), Rottt--tttttni (stacking yellow block on top of red block), and FetchSimRobot-Place Placcig (placing yellow block on the right of red block). We average results seeds per task.
output filtered of object proposals. In other words, we would not know which object is which if the detector hasn't been trained with the label of objects we care about.
In Figure show what our base reward model outputs on two goal descriptions: 1. inverted pendulum 2. yellow object close to a blue object. For the first goal description we observe that the product increases the pendulum becomes more inverted from either side as desired. For second goal description we observe that as the blue object gets closer to the yellow object the dot product increases except for the closest image where it dips which results in an undesired output. We observed through these two goal descriptions and many others that the base reward model is not sufficient for recognizing object spatial relationships. The encoders are good identifying what objects are in the image however, which is what leverage for our full reward model.
3.2 FULL ZERO-SHIOT REWARD MODEL RESULTS
In Figure we show how our full reward model performs on three manipulation tasks. We train each task using our full zero-shot reward model output as reward for the PPO reinforcement learning algorithm (Schulman et al., 2017). We then train with the oracle reward that has access to state and the base zero-shot reward model for comparison. Curiosity reward (Pathak al., 2017; Burda
5

---

Under review as a conference paper at ICLR 2022
et al., 2018) is also used as baseline since it only has access to images similar to method for computing reward, but has only been successful for videogames such as Atari and Mario or Locomotion where exploring new states leads progressing through the task (going further in levels of game for example). It is less privileged however, in that it does not use language input for task specification.
Curiosity learns the pushing pucks close together task but then starts learning separation of the pucks which reemphasizes that curiosity is only useful for tasks where exploring new dynamics leads to going farther in the task. Curiosity also has some trouble learning the double inverted pendulum task because the dynamics of the pendulum swinging can be hard to predict and therefore have misleading higher reward. Curiosity also does not learn the other manipulation tasks (stacking and placing) as those are more complex tasks that are harder to reach by exploration. For Double inverted pendulum (Fig. 4) our base reward model does better than oracle by chance which we speculate is because the oracle reward was originally designed for state input and was not tuned for learning image to reward mapping. Our base reward model fails for pushing, stacking, and placing, which take "an image of yellow block on top of a red block", "an image of a yellow object close to a blue object", and "an image of yellow block the right of a red block" language input for those tasks respectively. For pushing, stacking, and placing our full reward model performs almost as well as oracle sparse reward which has privileged access to state.
The oracle reward function for Gouble-Inverted-Penie minus velocity penalty. The oracle for SawyerSimmRooot-Pee one when the centroid distance between two pucks are below threshold. The oracle reward for DetchSimRooot-Staccee
is alive bonus minus distance penalty is sparse reward that outputs
is sparse reward that outputs one when a yellow block is within horizontal and vertical threshold distance of a red block. The oracle reward for SaacchSioobboPPacciiiii sparse reward that outputs one when a yellow block is correctly placed on the right of red block. See Figs. 3 & 5 for image observation examples of FetchSimmRoot and SawyerSimRobot.
is a
3.3 MULTI-TASK POLICY RESULTS
In order avoid hav- ing to train a new policy for every new task we want to learn, we train a multi-task policy with set of training tasks and then show that it can gen- cralize to a set of un- seen test tasks by lever- aging CLIP's language model to encode the goal text description of the tasks as condition- ing input to our multi- task policy.
Seen distribution
Unseen distribution
train tasks
test tasks
train tasks
test tasks
(episode reward stats)
mean
s.c.
mean
s.c.
mean
s.c.
mean
s.c.
No Conditioning
22.71
1.13
12.74
0.89
17.60
1.08
9.01
0.87
Primitive Code Cond.
35.06
1.29
16.60
0.99
25.03
1.14
12.66
0.85
Language Cond.
43.62
1.30
20.08
1.08
31.28
1.23
14.79
1.02
Table 2: Multi-task Policy Performance Results: We report multi-task average and standard error performance across all 18 training tasks and all 18 test tasks (unseen object color in goal text or image) for seen and unseen initial state distributions with no conditioning, primitive code conditioning, and language conditioning. The values reported are the episode reward averaged across 50 seeds per task policy rollout. The tasks robotic arm manipulation of objects to different target semantic relationships of cach other in the FetooSi environment from only images. While the policies were trained using our zero-shot reward function, the reward metric reported is oracle reward
In the FetchSimRRbot environment we train 18 tasks with PPO each for 200K environ- ment steps with our zero-shot reward model described in the previous section and show generalization results for 18 unseen test tasks by training a language conditioned policy with behavior cloning on rollouts of the 18 training tasks. We collect 5000 timesteps per task which is around 50 trajectories. We do this for pickplace-left, pickplace-right, and stack tasks for different object combinations. The object training colors are red, green, and yellow, and the object test color is blue. So the multi-task policy has never seen blue block in text input or image input.
For each task average episode rewards over 50 seeds. We then average across all training tasks to get the train metric and across all test tasks to get the test metric respectively. We compare our language conditioned policy with policy trained without task labels, and one trained with primitive

---

Under review as a conference paper at ICLR 2022
code the conditioning input. The latter baseline simply labels the training tasks with integers 0 to 17. Since it only has primitive codes for the training tasks it uses CLIP's language embedding to find the closest primitive code it was trained with to the test language goal description.
We use the same oracle reward that has direct access to state in figure 6 to measure the performance of our multi-task models. Every timestep the objects are in the correct target text description the agent is rewarded point. The objects never start in the correct target state. Therefore it always takes several timesteps for the policy move the objects the correct state. All episodes timeout after 100 timesteps or if the one of the objects falls off the table. In RL training and in policy rollout collection for behavior cloning each object has a one block width of variation in starting point location. We tested on same distribution of start point variation interval (seen initial state distribution) and double the start point variation interval (unseen initial state distribution).
In Table 2
we see that no conditioning has the lowest performance as expected since there is full ambiguity in what task to perform given only an image. However, since the no conditioning model has been trained across many different object colors going to different target states it has learned general displacement of blocks to different semantic locations randomly that can sometimes correct semantic location by chance during testing. For the training tasks it has higher performance than testing tasks since it can memorize to execute some of the training tasks that have almost the exact same initial states sampled during testing those in the behavior cloning dataset and thereby execute correctly as memorized.
moved to the
The primitive code conditioned model has much higher performance than the no conditioning model since it can disambiguate what task it needs to execute. Surprisingly language conditioning performs significantly better than primitive code conditioning in training task performance. We conjecture this is due to language conditioning being a more structured representation of the task, providing a better loss manifold from which the model can learn better perceptual motor representation of the task distribution. In addition the training tasks are sampling initial states from a continuous distribution meaning almost no two policy rollouts have the same initial state. So some amount of generalization needs to be had even in the seen initial state distribution to have high performance on the training tasks.
The language conditioned model is significantly better in test task performance over the primitive code conditioning since can use the language embedding to infer the target task determined by the goal text description. The primitive code conditioning model has access to the language embedding only to determine the closest training primitive code to the target task description. It compares the text description embedding of all the training tasks to the test task text description embedding and chooses the training task primitive code corresponding to the smallest 1.2 distance. It's interesting to observe that the primitive code conditioning does better than conditioning on the test task because the closest primitive code can sometimes lead to a successful execution of the test task since it has some signal towards the correct task. One such example that the test goal description task "a blue block on the right of a yellow block" is mapped to the primitive code of the training goal description task of " red block on the right of yellow block".
In Table 2 we also show the same trend in the models performing at double the interval of initial starting locations of the objects seen at training (unseen intial state distribution): the no conditioning policy performing the poorest on train and test tasks, the primitive code conditioned model performing significantly better on both, the language conditioned model performing the best at training and test tasks via its language structured specification of tasks.
We train the multitask policy with ResNet18 image encoding that is concatenated with CLIP language encoding both of which are pushed through an fc layer before concatenation. The concatenated vector is then inserted into two fe layers before predicting actions. We apply an 1.2 regression loss for continuous action space. The policy is trained using Adam optimizer with AMS grad with learning rate of le-4. The images are augmented with Py Torch RandomResizeddrrpp of 0.95 to 1.0 area and 0.98 to 1.02 aspect ratio randomization and resized to original image dimensions of 128x 288128.. All the multitask models are trained for 300 epochs.
7

---

Under review as a conference paper at ICLR 2022
4 RELATED WORK:
Goal conditioned policies Goal conditioned policies allow the user to specify the agent's goal. States (Schaul et al., 2015; Andrychowiez et al., 2017) and Images (Pathak et al., 2018; Nair et al., 2018; Fu et al., 2018; Singh et al., 2019) are one way of specifying goal. However, they assume that user has access images or state to minimize human labor and instrumentation, and use language which provides a natural form of supervision.
photo or state of final task given the agent. We assume no access to goal
Goal text conditioned policies Several previous efforts train reward functions or policies that take natural language input for goal description (Oh et al., 2017: Bahdanau et al., 2018; Zhou & Small, 2020; Goyal et al., 2020; Fu et al., 2019; Hermann et al., 2017; Shao et al., 2020). They all however rely on reward signals that have access to state of the system or demonstrations of the task distribution they are training on.
Learning reward functions There are works that human videos to learn reward functions to train their agent with (Sermanet et al., 2018; 2016; Shao et al., 2020). We however, don't need curated dataset of humans performing the tasks we want our agent train with. Having humans perform all tasks we are interested in may not scale well with the amount of labor needed for recording and curating those datasets. CLIP has the advantage of learning a caption model from 400 million image, text pairs from the publicly available sources on the internet WIT.
No Environment Reward There have been recent work methods that use reward signal from the environment to train for specific tasks. Two notable ones are Curiosity (Pathak et al., 2017) and DIAYN (Eysenbach al., 2018). Curiosity has only been successful on video games with discrete action space and DIYAN has mostly been successful on direct state input. Our method however, can learn subset of manipulation tasks in continuous action space on raw pixels and has the advantage of specification from user goal text.
Utility of large vision language models in RL There has been recent work that leveraged CLIP to do many complex manipulation tasks (Shridhar et al., 2021). They however, have access to labeled expert demonstrations for training their policy. We assume no access demonstrations or goal images at training or test time.
DISCUSSION
In this work, we presented a method for learning a set of object manipulation tasks without access to state of the system to compute reward from text goal description alone. Our method doesn't 't use goal images or demonstrations at training time or test time. We devised a zero-shot reward model that leverages language vision model (CLIP) that has been trained on a very large dataset of captioned images on the internet to compute progress (reward) towards a goal text description. We this zeroshot-reward model collect data on many tasks then supervise a language conditioned multi-task policy that can execute new tasks without need of extra training.
Our work opens avenues for many directions such as learning spatial relationships from data or leveraging separate datasets for spatial relationships. Pioneering work along these lines was proposed in (Guadarrama et al., 2013; Golland et al., 2010), who showed that spatial relationships could be grounded in a learning- based system. We encourage pursing the integration of these approaches to the grounding of spatial prepositions with the noun-phrase grounding capability we have demonstrated based on our CL.IP-saliene method. current method also does not specify state of the system other than semantic position such as object pose or closed door.
Finally we must address the ethical perils inherent in our leverage of models trained on large-scale vision and language datasets. Such datasets are well known to suffer from dataset bias that can cause failure or unintended harm (Buolamwini & Gebru, 2018). While the near-term risks appear to be limited with the robotic applications presently envisioned, practitioners should continuously monitor systems for bias against underrepresenteed groups and ensure that robotic systems work across all socioeconomic domains. Techniques for bias assessment and debiasing should be employed whenever possible to ensure this remains the case.
8

---

Under review as a conference paper at ICLR 2022
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv preprint arXiv: 2017. 8
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning understand goal specifications by modelling reward. arXiv preprint wrXiv:1806.01444 2018. 2,8 8
Kobus Barnard, Pinar Duygulu, David Forsyth, Nando De Freitas, David M Blei, and Michael I Jordan. Matching words and pictures. 2003.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91. PMLR. 2018. 8
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprini
2018. 5
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations, 2019. 2
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint TrXiv:/802.06000, 2018. 8
Andrea Frome. Greg Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. 2013. 2
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. arXiv preprint arXiv:1805.11688 2018. 1,8
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement leamning for vision-based instruction following. arXiv preprint arXiv:1902.07742, 2019. 2,
Dave Golland, Percy Liang, and Dan Klein. game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 conference on empirical methods in natural language processing, pp. 410-419, 2010. 8
Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Pix12r: Guiding reinforcement learning using natural language by mapping pixels rewards. arXiv preprint 1554::2007..55433, 2020. 2,
Sergio Guadarrama, Lorenzo Riano, Dave Golland, Daniel Go. Yangqing Jia, Dan Klein, Pieter Abbeel, Trevor Darrell, al. Grounding spatial relations for human-robot interaction. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1640-1647. IEEE, 2013. 8
Sergio Guadarrama, Erik Rodner, Kate Saenko, and Trevor Darrell, Understanding object descriptions in roboties by open-vocabulary object retrieval and detection. The International Journal Robotics Research, 35(1-3):265-28) 2016. 2
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017. 4
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in simulated 3d world. arXiv preprint arXiv: /766.0670 2017. 8,
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified transformer.
arXiv preprint arXiv:2 10l.:2002//000 10772 2021. 2

---

Under review as a conference paper at ICLR 2022
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task -agoostci visiolinguistic representations for vision-and- lan-gagueg tasks. arXiv preprint
2019.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In NeurIPS, 2018. 1, 8
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning, pp. 2661 2661-2670. PMLR, 2017. 2,
Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-7772222 2787. PMLR, 2017. 5,
Deepak Pathak, Parsa Mahmoudich, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp.
2050-2053, 2018. 1,8
Ariadna Quattoni, Michael Collins, and Trevor Darrell. Learning visual representations using images with captions. In 2007 IEEE Conference Computer Vision and Pattern Recognition, pp. 1-8.
IEEE, 2007.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint S.Kiv:2003.000200 2021. 2
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In ICMI., 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017. 5
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint arXiv:1612.0669/ 2016. 2, 8
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Tome--ontrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018. 2,
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learning manipulation concepts from instructions and human demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2020. 2,
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. arXiv preprint arXiv:2109. 2021. 8
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv: /904.07854, 2019. 1,
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-ling linguiste representations. arXiv preprint arXiv: 085.00 2019. 2
Li Zhou and Kevin Small. Inverse reinforcement learning with natural language goals. arXiv preprint orXiv:2008.0692. 2020. 2, 8
10


Reviews:
[{"review_id": "ageyidlqlb1", "reviewer": "Reviewer_GnSJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\nThe paper tackles and interesting and important problem in 0-shot reward specification. Solving 0-shot reward specification, particularly from language is essential to getting RL to work in new environments. Additionally the approach proposed in this work is creative and performs well in the domains considered from images.\n\n*Weaknesses*\n\nThe weaknesses in this work are (1) the generality of the proposed approach, (2) the experimental domains/comparisons, and (3) coverage of prior work.\n\n(1) The paper proposes an interesting approach to 0 shot reward specification in explicitly mapping an instruction to nouns and spatial relations, using CLIP to localize a noun, then using the position and spatial relation with heuristics to define the reward. However I think there are a number of limitations of this approach that the paper should discuss further. \n\nFirst, parsing the instruction into objects and into a fixed set of spatial relations does seems limited to programmatic language and the specific tasks in the paper (e.g. in the paper the instruction “an image of a yellow block on top of a red block\" directly maps to the \"on top of\" rule). But this likely will not be trivial in the case where the instruction is in the form of natural language. Even for the \"inverted pendulum\" example shown earlier in the paper, its not clear how a parser would map that to objects and a specific spatial relation. \n\nSecond, the approach of extracting the object positions with Grad-CAM is a clever idea, but it does seem very specific to environment, tasks, and viewpoints considered in the paper. In a more interesting visual scene with more than 2 objects its not clear that the Grad-CAM would provide as good object localization. The paper also mentions that for this environment very specific camera viewpoints are needed for the GRAD-CAM localization to work. Moreover there are many tasks (e.g. opening a cabinet) where its not clear how to heuristically define a reward based on the Grad-CAM localization, or what part of the cabinet the saliency map will map to.\n\nIt does seem that most of the reward specification is actually coming in the form of hardcoded rules, and the main use of CLIP is in localizing objects. \n\n(2) The experiments show that on some simple table-top manipulation tasks the proposed approach can nearly match an Oracle reward function. This is an interesting result, however says more about the ability to use CLIP to localize objects than it does about the particular reward function approach, since the heuristic reward is the same between both this method and the Oracle. The curiosity driven baselines also are not that informative since they do not get the language instruction. Some more interesting comparisons would be (a) how the method compares to goal-image specifications and (b) how it might compare to using an off the shelf image-captioning approach. It would also be valuable to see if this approach can handle (a) more complex natural language instructions, and (b) more interesting environments with more complex visuals than the 2 solid colored blocks. \n\n(3) There are a number of missing related works [1,2,3,4] which also learn language conditioned skills on robots. Particularly [1-3] also learn to ground language to goals or rewards. \n\n[1] MacGlashan et al. Grounding English Commands to Reward Functions. RSS 2015.\n[1] Arumugam et al. Grounding natural language instructions to semantic goal representations for abstraction and generalization. Autonomous Robots 2019.\n[3] Nair et al. Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation. CoRL 2021\n[4] Lynch and Sermanet. Language conditioned imitation learning over unstructured data. RSS 2021. ", "summary_of_the_paper": "The paper presents a hybrid learning/rule-based approach to language-conditioned reward specification, particularly for robotic control tasks from images. The core contribution of the work is in specifying the reward 0-shot. Specifically, the method parses a language instruction into an object noun phrase and desired spatial configuration. Then it uses CLIP embedding of each object phrase as a query, and uses Grad-CAM to get a saliency map in the image which is used to localize the target object. Finally a set of heuristics are used to compute the reward for a particular spatial configuration given the target object(s) coordinates. The reward can then be used to train either single or multi-task policies. Experiments suggest that in Simulated Fetch robot domains this approach can nearly match an Oracle reward.  ", "summary_of_the_review": "Overall the paper presents an interesting and creative solution to the important problem 0-shot language specification for robot manipulation from images. However, the proposed method seems to be task/environment/viewpoint specific, and the paper should further discuss these limitations. The paper can also improve its experimental domains, baseline comparisons, and coverage of related work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XRmBKWKTXYY", "reviewer": "Reviewer_PiKW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea to use natural language grounded in the scene as the reward function is an interesting idea and a goal for language grounded RL. However, I have some concerns/questions.\n\n1. Definition of zero-shot. The reward function is determined by the heuristics defined in table 1. These heuristics encode the knowledge about the goal. So, the reward function is not entirely zero-shot as the heuristics encode the structure of the goal.\n\n2. The use of natural language is limited. Originally, I expected to see the reward function that is directly computed from the CLIP image embed and text embed. It is reasonable that the paper shows the dot product doesn’t represent the spatial relationship well. However, the final proposal, i.e. using a CLIP-based saliency map and a separate spatial module, doesn’t leverage the vision-language model well. The use of CLIP here is closer to an object detector. While this paper showed that the pretrained mask RCNN doesn’t provide good detection, this still doesn’t show change the fact that CLIP is used as an object detector. In this case, any color detector may work as well too. Then, what does a pretrained vision-language model help in the reward specification? This separation also limits the possibility to extend to other language descriptions beyond spatial relationships.\n\n3. CLIP-base result in Figure 6. Even though the reward using a simple dot product may not be good in all cases, why does Figure 6 show the base result to be always flat? It doesn’t seem likely that RL agent that uses this reward function doesn’t learn anything from the beginning.\n\n4. Alternative CLIP baseline? While the dot product version doesn’t work. I’m wondering if there is an alternative formulation. For example, turns the reward specification into classification of objects of different spatial relationships. This would remove the need for separate spatial heuristics.", "summary_of_the_paper": "This paper demonstrates how to use pretrained CLIP model as a reward function for an RL agent. This approach enables flexible goal specification using language. The proposed reward generation method uses CLIP to identify the relevant objects and a separate module to compute the reward based on a specified spatial relationship. This separation produces a better reward function than using the dot product between the embedded image vector and language vector. ", "summary_of_the_review": "The idea of the paper is interesting. However, the final implementation doesn’t really need the pretrained vision-language model.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Sf9J7hCAU5h", "reviewer": "Reviewer_b31Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work proposes a simple and novel solution to the zero-shot reward specification problem which is an important step towards getting RL agents to work in new environments and tasks. The experiments show promising results and support the author’s claims.\n\nWeakness:\n- Experiments have been performed on only simple scenarios and environments.\n- The natural language queries are simple and the authors don't talk about translating to more complex tasks \n- Using Clip based text embedding with a dot product in the reward function performs worse than the heuristic-based reward function, but since the embedding based approach is translatable, more experiments could have been performed to understand why the reason for the lower performance\n- Missing details on the construction of goal-text. Is it constructed in a way that is easy for the parser to be parsed?\n- Missing related works in the space of learning language conditioned reward function or learning to perform language conditioned tasks [1].\n\n[1] Fu J, Korattikara A, Levine S, Guadarrama S. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint arXiv:1902.07742. 2019 Feb 20.", "summary_of_the_paper": "The paper proposes an approach for zero-shot reward specification using natural language groundings without expert demos and state information. The work proposes using CLIP’s image+language encoder to find the saliency maps and separately encoding the target locations for the object. These objects are identified by parsing the task sentence. The work proposes automatically getting a reward function by again using the sentence. The work claims that automatically parsing the goal text and using a heuristic to get a reward function works better than using the CLIP's language encode to goal text and get the rewards with a dot product.", "summary_of_the_review": "The experiments in the paper are on toy examples and the approach proposed has several limitations for it to translate to other tasks, for example, the complexity of the environment and the task specification. But the techniques proposed are simple and can be extended to more complex cases with further research.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OsW_FKM_UuA", "reviewer": "Reviewer_fEgi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles the problem of specifying goal configurations via language text. This is a more general (and therefore, more challenging) setting that is agnostic to the specific instance of the target objects/configurations presented via the goal image. The empirical evidence presented by the authors seems to support their claims.\n\nHowever, given below are some of my concerns with the paper.\n+ The goal text descriptions seem to be severely under-specifying tasks. For instance, consider an example of a goal text from the paper: “image of an inverted pendulum”. This doesn’t really say a lot about the desired task and it might hold true for several configurations within a delta tolerance of the actual goal state. That makes me wonder if the kind of goal-text descriptions being considered in the paper can be treated as good proxies for specifying tasks? I would imagine this problem becoming severe with task complexity when the approach is deployed to settings with slightly more complicated state spaces and task configurations. For instance, consider grid-world navigation. What would a goal description that leads to a high correlation between grounding of objects in the current observation and progress towards the goal be for the task of going to location X in the grid?\n\n+ The Introductory section of the paper makes the case against reward engineering by saying that it utilizes the underlying state information. The spatial grounding heuristics used in the proposed approach are getting computed based on the GT locations of objects in the scene. Doesn’t that count as having access to state information and “engineering” rewards? How do the authors reconcile this? Moreover, computing the rewards based on whether spatial relationships (as decoded from the goal text) are being satisfied or not involves the use of multiple camera views: this brings in additional complexity from a real-world deployment perspective.\n\n+ It is not clear how their approach fundamentally differs from the reward structure used here (https://arxiv.org/pdf/1904.04404.pdf). The reward is derived from the IoU and classification accuracies of predictions of the model and the policy gets trained depending on how well the agent’s perception can localize the target object. One might say that they warm start their training with expert trajectories which the authors do not, but to be fair, the cited paper works with a much more complex state space (3D indoor scenes). Ignoring that, the general principle of deriving a reward based on localization/grounding of a target semantic concept holds true here as well.\n\n+ The paper is missing a lot of critical information such as:\n  + There is no information of how the goal text descriptions are derived. Is it by using some sort of captioning model on the image goal description or are they manually annotated for every task?\n  + The paper doesn’t have any details on how the grounding of semantics and spatial relationships from the goal text onto the visual observations translates to numeric rewards. That seems to be an important step in their approach.\n\nAlso listed below are some minor suggestions and clarification questions:\n+ In Sec 2.2, the authors say that the goal text is parsed into object noun phrases and object interactions. What does the latter mean? Are these the spatial relationships that the referred objects must satisfy?\n+ Did the authors try fine-tuning the Mask-RCNN? Is the comparison shown in Fig. 3 between non fine-tuned Mask-RCNN and CLIP? If so, do the authors have a sense for the degree of overlap between the vocabulary of classes between MaskRCNN/CLIP and their environments?\n+ In Sec 3.2, the language makes it seem like your model is progressively trained using GT reward first and then your proposed visio-linguistic rewards. This could get a little confusing, consider rewording to ensure that it’s clear that one of them is your approach and the others are baselines.", "summary_of_the_paper": "The paper proposes a framework to train policies for tasks that are specified via language text without the use of any expert trajectories or underlying state information to engineer reward functions. The authors leverage a state-of-the-art visual grounding model (CLIP) to ground object nouns from the text into the current visual observation in order to derive a proxy for the rewards signal. Through some toy experiments they demonstrate performance of a simple baseline that implements this concept as well as enhancements to the baseline (saliency grounding + spatial relationship based rewards) that overcomes drawbacks of the naive approach. They also train their language-conditioned on several tasks to obtain a generic policy that can learn to solve an arbitrary task, given paired text descriptions and trajectory rollouts from the training tasks.", "summary_of_the_review": "Overall, I am not entirely convinced that the approach of simply grounding text descriptions onto image observations can serve as a reliable proxy for rewards. In addition to that, the paper, in its current format, has a lot of missing details. Therefore, my vote is to reject this paper", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript describes a method that turns sentences into reward functions by recognizing objects, parsing sentences into a simple formalism, and then grounding the parse in the recognized objects to form a reward for an agent.

1. The title and much of the manuscript are written in a way that reviewers found confusing. It would seem from the title and most of the text that the method integrates language models, CLIP specifically, into RL in a novel way to provide zero-shot rewards. But this is not the case. CLIP is used purely as an object detector. Yes, the method requires a good object detector and CLIP provides that, but any good object detector that can handle arbitrary phrases would have done.

2. The overall setup of the work: extract the state of the world and then parse sentences to formulate rewards by grounding parts of the parse into parts of the world state has been explored widely in robotics. Reviewers provided citations going back several years, but many others exist.

I would encourage the authors to rewrite the manuscript around their central contributions and downgrade their use of CLIP and language models in general to a minor technical footnote. Similarly refocusing related work on the robotics literature and demonstrating how this approach differs and improves on the state of the art there could result in a strong contribution.
Decision: Reject