Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach for zero-shot reward specification using natural language groundings without expert demos and state information. The work proposes using CLIP’s image+language encoder to find the saliency maps and separately encoding the target locations for the object. These objects are identified by parsing the task sentence. The work proposes automatically getting a reward function by again using the sentence. The work claims that automatically parsing the goal text and using a heuristic to get a reward function works better than using the CLIP's language encode to goal text and get the rewards with a dot product.
main_review: The work proposes a simple and novel solution to the zero-shot reward specification problem which is an important step towards getting RL agents to work in new environments and tasks. The experiments show promising results and support the author’s claims.

Weakness:
- Experiments have been performed on only simple scenarios and environments.
- The natural language queries are simple and the authors don't talk about translating to more complex tasks 
- Using Clip based text embedding with a dot product in the reward function performs worse than the heuristic-based reward function, but since the embedding based approach is translatable, more experiments could have been performed to understand why the reason for the lower performance
- Missing details on the construction of goal-text. Is it constructed in a way that is easy for the parser to be parsed?
- Missing related works in the space of learning language conditioned reward function or learning to perform language conditioned tasks [1].

[1] Fu J, Korattikara A, Levine S, Guadarrama S. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint arXiv:1902.07742. 2019 Feb 20.
summary_of_the_review: The experiments in the paper are on toy examples and the approach proposed has several limitations for it to translate to other tasks, for example, the complexity of the environment and the task specification. But the techniques proposed are simple and can be extended to more complex cases with further research.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a framework to train policies for tasks that are specified via language text without the use of any expert trajectories or underlying state information to engineer reward functions. The authors leverage a state-of-the-art visual grounding model (CLIP) to ground object nouns from the text into the current visual observation in order to derive a proxy for the rewards signal. Through some toy experiments they demonstrate performance of a simple baseline that implements this concept as well as enhancements to the baseline (saliency grounding + spatial relationship based rewards) that overcomes drawbacks of the naive approach. They also train their language-conditioned on several tasks to obtain a generic policy that can learn to solve an arbitrary task, given paired text descriptions and trajectory rollouts from the training tasks.
main_review: The paper tackles the problem of specifying goal configurations via language text. This is a more general (and therefore, more challenging) setting that is agnostic to the specific instance of the target objects/configurations presented via the goal image. The empirical evidence presented by the authors seems to support their claims.

However, given below are some of my concerns with the paper.
+ The goal text descriptions seem to be severely under-specifying tasks. For instance, consider an example of a goal text from the paper: “image of an inverted pendulum”. This doesn’t really say a lot about the desired task and it might hold true for several configurations within a delta tolerance of the actual goal state. That makes me wonder if the kind of goal-text descriptions being considered in the paper can be treated as good proxies for specifying tasks? I would imagine this problem becoming severe with task complexity when the approach is deployed to settings with slightly more complicated state spaces and task configurations. For instance, consider grid-world navigation. What would a goal description that leads to a high correlation between grounding of objects in the current observation and progress towards the goal be for the task of going to location X in the grid?

+ The Introductory section of the paper makes the case against reward engineering by saying that it utilizes the underlying state information. The spatial grounding heuristics used in the proposed approach are getting computed based on the GT locations of objects in the scene. Doesn’t that count as having access to state information and “engineering” rewards? How do the authors reconcile this? Moreover, computing the rewards based on whether spatial relationships (as decoded from the goal text) are being satisfied or not involves the use of multiple camera views: this brings in additional complexity from a real-world deployment perspective.

+ It is not clear how their approach fundamentally differs from the reward structure used here (https://arxiv.org/pdf/1904.04404.pdf). The reward is derived from the IoU and classification accuracies of predictions of the model and the policy gets trained depending on how well the agent’s perception can localize the target object. One might say that they warm start their training with expert trajectories which the authors do not, but to be fair, the cited paper works with a much more complex state space (3D indoor scenes). Ignoring that, the general principle of deriving a reward based on localization/grounding of a target semantic concept holds true here as well.

+ The paper is missing a lot of critical information such as:
  + There is no information of how the goal text descriptions are derived. Is it by using some sort of captioning model on the image goal description or are they manually annotated for every task?
  + The paper doesn’t have any details on how the grounding of semantics and spatial relationships from the goal text onto the visual observations translates to numeric rewards. That seems to be an important step in their approach.

Also listed below are some minor suggestions and clarification questions:
+ In Sec 2.2, the authors say that the goal text is parsed into object noun phrases and object interactions. What does the latter mean? Are these the spatial relationships that the referred objects must satisfy?
+ Did the authors try fine-tuning the Mask-RCNN? Is the comparison shown in Fig. 3 between non fine-tuned Mask-RCNN and CLIP? If so, do the authors have a sense for the degree of overlap between the vocabulary of classes between MaskRCNN/CLIP and their environments?
+ In Sec 3.2, the language makes it seem like your model is progressively trained using GT reward first and then your proposed visio-linguistic rewards. This could get a little confusing, consider rewording to ensure that it’s clear that one of them is your approach and the others are baselines.
summary_of_the_review: Overall, I am not entirely convinced that the approach of simply grounding text descriptions onto image observations can serve as a reliable proxy for rewards. In addition to that, the paper, in its current format, has a lot of missing details. Therefore, my vote is to reject this paper
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper demonstrates how to use pretrained CLIP model as a reward function for an RL agent. This approach enables flexible goal specification using language. The proposed reward generation method uses CLIP to identify the relevant objects and a separate module to compute the reward based on a specified spatial relationship. This separation produces a better reward function than using the dot product between the embedded image vector and language vector. 
main_review: The idea to use natural language grounded in the scene as the reward function is an interesting idea and a goal for language grounded RL. However, I have some concerns/questions.

1. Definition of zero-shot. The reward function is determined by the heuristics defined in table 1. These heuristics encode the knowledge about the goal. So, the reward function is not entirely zero-shot as the heuristics encode the structure of the goal.

2. The use of natural language is limited. Originally, I expected to see the reward function that is directly computed from the CLIP image embed and text embed. It is reasonable that the paper shows the dot product doesn’t represent the spatial relationship well. However, the final proposal, i.e. using a CLIP-based saliency map and a separate spatial module, doesn’t leverage the vision-language model well. The use of CLIP here is closer to an object detector. While this paper showed that the pretrained mask RCNN doesn’t provide good detection, this still doesn’t show change the fact that CLIP is used as an object detector. In this case, any color detector may work as well too. Then, what does a pretrained vision-language model help in the reward specification? This separation also limits the possibility to extend to other language descriptions beyond spatial relationships.

3. CLIP-base result in Figure 6. Even though the reward using a simple dot product may not be good in all cases, why does Figure 6 show the base result to be always flat? It doesn’t seem likely that RL agent that uses this reward function doesn’t learn anything from the beginning.

4. Alternative CLIP baseline? While the dot product version doesn’t work. I’m wondering if there is an alternative formulation. For example, turns the reward specification into classification of objects of different spatial relationships. This would remove the need for separate spatial heuristics.
summary_of_the_review: The idea of the paper is interesting. However, the final implementation doesn’t really need the pretrained vision-language model.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a hybrid learning/rule-based approach to language-conditioned reward specification, particularly for robotic control tasks from images. The core contribution of the work is in specifying the reward 0-shot. Specifically, the method parses a language instruction into an object noun phrase and desired spatial configuration. Then it uses CLIP embedding of each object phrase as a query, and uses Grad-CAM to get a saliency map in the image which is used to localize the target object. Finally a set of heuristics are used to compute the reward for a particular spatial configuration given the target object(s) coordinates. The reward can then be used to train either single or multi-task policies. Experiments suggest that in Simulated Fetch robot domains this approach can nearly match an Oracle reward.  
main_review: *Strengths*

The paper tackles and interesting and important problem in 0-shot reward specification. Solving 0-shot reward specification, particularly from language is essential to getting RL to work in new environments. Additionally the approach proposed in this work is creative and performs well in the domains considered from images.

*Weaknesses*

The weaknesses in this work are (1) the generality of the proposed approach, (2) the experimental domains/comparisons, and (3) coverage of prior work.

(1) The paper proposes an interesting approach to 0 shot reward specification in explicitly mapping an instruction to nouns and spatial relations, using CLIP to localize a noun, then using the position and spatial relation with heuristics to define the reward. However I think there are a number of limitations of this approach that the paper should discuss further. 

First, parsing the instruction into objects and into a fixed set of spatial relations does seems limited to programmatic language and the specific tasks in the paper (e.g. in the paper the instruction “an image of a yellow block on top of a red block" directly maps to the "on top of" rule). But this likely will not be trivial in the case where the instruction is in the form of natural language. Even for the "inverted pendulum" example shown earlier in the paper, its not clear how a parser would map that to objects and a specific spatial relation. 

Second, the approach of extracting the object positions with Grad-CAM is a clever idea, but it does seem very specific to environment, tasks, and viewpoints considered in the paper. In a more interesting visual scene with more than 2 objects its not clear that the Grad-CAM would provide as good object localization. The paper also mentions that for this environment very specific camera viewpoints are needed for the GRAD-CAM localization to work. Moreover there are many tasks (e.g. opening a cabinet) where its not clear how to heuristically define a reward based on the Grad-CAM localization, or what part of the cabinet the saliency map will map to.

It does seem that most of the reward specification is actually coming in the form of hardcoded rules, and the main use of CLIP is in localizing objects. 

(2) The experiments show that on some simple table-top manipulation tasks the proposed approach can nearly match an Oracle reward function. This is an interesting result, however says more about the ability to use CLIP to localize objects than it does about the particular reward function approach, since the heuristic reward is the same between both this method and the Oracle. The curiosity driven baselines also are not that informative since they do not get the language instruction. Some more interesting comparisons would be (a) how the method compares to goal-image specifications and (b) how it might compare to using an off the shelf image-captioning approach. It would also be valuable to see if this approach can handle (a) more complex natural language instructions, and (b) more interesting environments with more complex visuals than the 2 solid colored blocks. 

(3) There are a number of missing related works [1,2,3,4] which also learn language conditioned skills on robots. Particularly [1-3] also learn to ground language to goals or rewards. 

[1] MacGlashan et al. Grounding English Commands to Reward Functions. RSS 2015.
[1] Arumugam et al. Grounding natural language instructions to semantic goal representations for abstraction and generalization. Autonomous Robots 2019.
[3] Nair et al. Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation. CoRL 2021
[4] Lynch and Sermanet. Language conditioned imitation learning over unstructured data. RSS 2021. 
summary_of_the_review: Overall the paper presents an interesting and creative solution to the important problem 0-shot language specification for robot manipulation from images. However, the proposed method seems to be task/environment/viewpoint specific, and the paper should further discuss these limitations. The paper can also improve its experimental domains, baseline comparisons, and coverage of related work.
review_title: null

Meta-review: This manuscript describes a method that turns sentences into reward functions by recognizing objects, parsing sentences into a simple formalism, and then grounding the parse in the recognized objects to form a reward for an agent.

1. The title and much of the manuscript are written in a way that reviewers found confusing. It would seem from the title and most of the text that the method integrates language models, CLIP specifically, into RL in a novel way to provide zero-shot rewards. But this is not the case. CLIP is used purely as an object detector. Yes, the method requires a good object detector and CLIP provides that, but any good object detector that can handle arbitrary phrases would have done.

2. The overall setup of the work: extract the state of the world and then parse sentences to formulate rewards by grounding parts of the parse into parts of the world state has been explored widely in robotics. Reviewers provided citations going back several years, but many others exist.

I would encourage the authors to rewrite the manuscript around their central contributions and downgrade their use of CLIP and language models in general to a minor technical footnote. Similarly refocusing related work on the robotics literature and demonstrating how this approach differs and improves on the state of the art there could result in a strong contribution.
Decision: Reject