Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes an RL-based approach to language understanding that is able to map instructions to reward functions and the corresponding policy using only offline data. Given a set of potentially sub-optimal demonstrations in the form of state-action sequences, each paired with a crowd-sourced instruction, the proposed framework learns a binary reward as a classifier that is a function of initial and current states, as well as language. Using model-based RL, this reward is combined with the learned transition function to learn the corresponding policy in an offline fashion. The method is evaluated in simulation and shown to outperform recent RL-, behavioral cloning- and goal image-based language understanding baselines. The paper then presents results for a handful of tasks with a physical robot arm.
strengths_and_weaknesses: STRENGTHS

+ The use of language is a compelling alternative to goal images in terms of specifying task goals. As the paper notes, there a number of limitations with requiring that goals be specified in terms of images, which are alleviated when using language as the goal specification.
+ The ability to learn language-guided policies entirely from offline demonstration has advantages over contemporary methods that learn reward functions from offline data, but then require on-robot learning.

WEAKNESSES

- Despite suggestions to the contrary, the approach seems limited to tasks that can be defined in terms of a single change in state vs. those that require reasoning over sequences of states (e.g., determining whether the agent has "turned the faucet to the left" requires reasoning over sequences of states and not a single state-pair).
- The paper should more clearly compare how this approach differs from related RL-based approaches to language understanding. While many of these do require that the resulting policies be learned online, they do not require access to state-action trajectories as is the case here, which can be expensive to collect as in the case requiring teleoperated demonstrations. Further, by learning policies in a model-free manner, they are not subject to the bounds on optimality posed by model-based RL.
- The paper should clarify which existing methods assume that the underlying demosntrations were provided by experts. This does not seem to be the case for symbol grounding-based approaches to language understanding, for which the grounding of the symbols to actions does not need to be optimal. Indeed the standard means by which these methods collect training data is to crowd-source annotations for existing demonstrations that need-not be optimal (e.g., see Tellex et al 2011).
- From a language understanding perspective, experimental evaluation is underwhelming. As noted below, the results seem more of an assessment of visuomotor control than of language understanding given (a) the simple nature of the instructions and (b) the limited evaluation of zero-shot performance.
- In its current form, the paper over-states the the method's utility on real robots. It is nice to see experiments beyond simulation, but the simplistic nature of the language suggests that the primary challenge here is on the visuomotor control rather than on the language understanding.



REFERENCES

[1] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 1507–1514, San Francisco, CA, August 2011.

summary_of_recommendation: The idea of learning language-conditioned policies entirely from sub-optimal demonstrations is interesting. However, the instructions are much simpler than those considered by other language understanding methods and the generalizability is unclear. It is nice to see the method demonstrated on a physical robot, however it seems that this is more due to the visuomotor policy learning than it is language understanding.

UPDATE BASED ON AUTHOR RESPONSE

I thank the authors for their detailed response, which resolved some of the questions and concerns that I initially raised in my review.

I like the overall motivation, but the significance of the contributions aren't clear. This is not the first paper to use a classifier to define a reward function (see below). I agree with the authors and other reviewers on the importance using language as a means of specifying abstract goals (with much of the work in language understanding focused on language that provides a finer grained specification of goals). However, as the authors note, the language here is rather simple. It seems that the primary contribution is the ability to learn visuomotor policies offline with language as an alternative images for specifying goals (i.e., language is sort of a second-class citizen). That's not to say that the contributions aren't sufficient, but that the paper should make them clear.


[2] Konidaris et al., Robot Learning from Demonstration by Constructing Skill Trees, IJRR 2012

[3] Kulkarni et al., Hierarchical Deep Reinforcement Learning: Ingegrating Temporal Abstaction and Intrinsic Motivation, NeurIPS 2016
issues: * The paper incorrectly states that references [27] and [28] are limited to simulated robots
* The following work is relevant

Blukis, V., Terme, Y., Niklasson, E., Knepper, R. A., & Artzi, Y. (2019). Learning to map natural language instructions to physical quadcopter control using simulated flight. arXiv preprint arXiv:1910.09664


* Line 106: It is not clear from the text why learning from offline data enables the learning of language-conditioned policies on real robots, Can the authors elaborate?
* Is the space of language associated with each task assumed to be known?
* Lines 170--172: With regards to the faucet example: If the agent rotated the faucet by 10 deg and then moved it back and forth by 5 degrees, would it not look like it was rotated at each time step under this definiition, even though I wouldn't be rotating it.
* The positive selection strategy seems arbitrary. Why is this labeling scheme reasonable?
* The paper mentions that identifying negative examples is challenging, yet the proposed approach is quite simple.
* For the evaluations in Section 5.1, is there a difference between the instructions that were seen during training. The description of the zero-shot experiments in Section 5.2 suggests that the sentences were the same,
.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work proposes a new approach to specify tasks using natural language. Motivated by the drawbacks of goal images, specifically over- and under-specification, the authors propose to instead represent reward using a language-conditioned classifier: does the transition between the initial and final states (images) satisfy the linguistic task description? In order to learn such a classifier, a dataset of state transitions and their corresponding natural language descriptions is required to ground the task description in the robot’s low-level observations. To scale this process, the authors use sub-optimal datasets that are later annotated with natural language. They show the proposed language-conditioned reward function can be combined with model-based RL in both real and simulated domains to successfully execute several tasks. Further, the authors analyze the performance of their method under different types of linguistic variations.

strengths_and_weaknesses: This paper addresses an important problem in robotics: that of task specification. Natural language is a good candidate as has been shown by previous works in the literature and the current paper. One strength of this paper is the insight to formulate the reward function as a classification problem. This type of weak supervision allows the authors to scale their data collection, while still being able to flexibly define a wide variety of tasks. Along with this proposal, the authors show the reward, when learned in this way, can be successfully optimized within a model-based RL framework. Another strength of this paper is the robot evaluations which show the feasibility of such an approach in the real world. In addition, this paper is well organized and the video was helpful as an overview of the approach. Along with these strengths, there are a few weaknesses that could be addressed to further improve the paper.

The proposed approach addresses some of the downsides of using goal images. While I agree this work can address the “task overspecification” problem, it is not immediately clear to me how the framework can be used to solve some non-goal reaching tasks (as alluded to in lines 37 or 171). The types of tasks that can be represented are limited by the reward being a function of the initial and final states. This seems to preclude some types of tasks such as those involving constraints on the trajectory (e.g., move a cup without spilling) or temporal (but not indefinite) tasks (e.g., turn the faucet three times). This paper would be strengthened by a discussion about the limitations of this type of task specification as well as clarification on how the model can be used to continuously rotate a faucet (line 171, is the planning approach repeatedly applied indefinitely to continue maximizing reward?).

In addition, I have included some minor comments and questions below regarding the clarity of the paper:
- Is there any intuition for why all the methods do worse on the “open door” task? (Figures 6 and 7)
- In the appendix, why are different visual planning models used for sim and real? This insight may be useful to those working on similar domains.
- It is mentioned that delta end-effector control is used, how far does each action move the gripper? How long is each trajectory typically?
- For the sim experiments, the data is collected using a random policy. How often does this policy lead to “interesting” data that corresponds to a task?

summary_of_recommendation: I recommend this paper be accepted to CoRL. It tackles the difficult problem of task specification and offers a useful perspective to the community. The results back up the claims of the paper and the methodology is well described. 
issues: The main issue with this paper is missing discussion about the limitations of the proposed reward specification method.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an offline multi-task reinforcement learning (RL) framework for language-conditioned tasks. First, exploration data is collected with a robot through random and often suboptimal interactions. Then this data is annotated with crowdsourced natural language labels that describe what was achieved, if anything at all. These labels are used to train a reward classifier which is subsequently used for model predictive control to achieve language-specified instructions during test time. Experimental results in simulation show that the  language-conditioned policies outperform image-goal baselines by 25%, and can even handle some unseen paraphrases of instructions. The method is also evaluated on a real-robot setup. 

strengths_and_weaknesses: Strengths:
- Overall, using language to condition policies instead of high-dimensional image-goals is an apt formulation. From an user-interaction perspective, asking users to specify image-goals is restrictive. Often the user might be expected to solve the manipulation task before the robot is asked to do so in order to generate a target image for the model, which is unreasonable. Language is also effective at capturing the semantic essence of the task on what needs to be achieved without having to extract this high-level information from images. 
- Prior works in language-conditioned RL often assume access to extensive amounts of language annotations to guide the policies during rollouts. The sample inefficiency of online RL methods are at odds with collecting annotations, which is expensive and time-consuming. In this work, the language labels are instead collected in an offline fashion to circumvent this issue. 
- Generally, the paper is well written and the method is easy to understand. The simulated experiments include a good set of baselines and comparisons. The method is also demonstrated on a real-robot.


Weaknesses:
- While the offline setting is beneficial for collecting language-labels, it’s also restrictive in certain ways. Initially, the robot starts off with random and highly-suboptimal behavior to collect data. With random exploration, it’s unlikely that the robot will generate meaningful trajectories that can be used to fully-exploit the compositional nature of language instructions and the ability to sequentially chain commands for long-horizon tasks. Consider the task of “setting the table” where the robot has to gather several objects (like plates and forks) and arrange them in a specific manner. This is unlikely to emerge through random exploration, and thus the language-goals in this framework are mostly restricted to quick actions like “open the drawer”. 
- Another concern is also the scalability of the approach for real-world applications. The real-robot experiments in Section 5.3 used a dataset of 3000 episodes (150000 frames) for a single desk environment. While the approach was tested with unseen paraphrases, in all the evaluations the training and testing environments, i.e. the IKEA desk, were identical, as is in a typically RL setting. It’s unclear if the reward-function is just learning a one-to-one mapping from language to prior states seen during training, or it’s actually learning something broadly applicable to other settings beyond the single desk. If the reward function is indeed restricted to the specific desk environment, then scaling this approach to a wide-range of real-robot tasks would be very difficult. 


summary_of_recommendation: This paper presents an interesting multi-task offline-RL method for language-conditioned tasks. While the method circumvents the issue of collecting expensive online language-annotations, it’s restricted to a specific environment and skills learned through random exploration. 

**Post Rebuttal**  
I thank the authors for the follow-up responses. It's great that the authors plan on releasing the dataset. Unlike WikiText, I am not sure how reusable this dataset would be without the physical setup (robot, objects, and scene). However, the simulated setup should certainly help in reproducing the results. Overall, I am still positive about the work, and will keep my score unchanged.
issues: Minor
- What was the action-space used for data-collection and execution? SE-3? SE-2? Were there any hand-crafted heuristics for the exploration process?
- How long did the random-exploration data collection process take per task?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 
This paper addresses the problem of learning to follow natural language instructions to solve vision-based manipulation tasks. Specifically, it assumes the availability of a large offline dataset of robot interaction that could be sub-optimal or even random with crowd-sourced descriptions that describe robot's behaviors. To solve this problem, this paper proposes to learn a classifier from the offline dataset that takes an initial state, a final state, and a description of the trajectory as input and outputs if the final state fulfills the description as well as learn a forward visual dynamics model (an action-conditioned video prediction model). To learn an effective classifier, many techniques are proposed, including learning from negative samples, data augmentation, and employing a pre-trained language model. During testing, the proposed framework rollouts action sequences with the learned visual dynamics model and evaluate them based on the reward provided by the classifier conditioned on the task language instruction and the predicted final states. Then, the action sequence that maximizes the classifier reward is selected to be executed. The proposed framework is compared against a variety of baselines that use a different task specification (i.e. a goal image) or leverage the offline data differently (e.g. multi-task RL or imitation learning). The experiments in simulation and on a real Franka Emika Panda robot show that the proposed framework outperforms baselines and can reasonably generalize to unseen natural language instructions. I believe this work studies a promising research direction (i.e. natural language instruction following with offline datasets), proposes an interesting framework, introduces effective techniques, and presents comprehensive evaluations. Therefore, I would like to see this work presented in CoRL.
strengths_and_weaknesses: 
## Paper strengths and contributions

**Motivation and intuition**
- This paper gives clear and organized reasons why representing tasks using goal states could be problematic.
- Leveraging natural language descriptions to instruct robots is easy and natural especially for non-expert users and flexible.
- Employing an offline dataset could potentially be more sample efficient. Also, learning a binary classifier should require much less data than multi-task RL and imitation learning.

**Technical contribution**
To effectively learn a binary classifier that can reliably predict if the transition between a pair of initial and final states align with a natural language description, this paper proposes several techniques, including:
- Leveraging intermediate states (close to initial or final states) to obtain more positive examples.
- Defining and sampling negative examples with mismatched initial-final state pairs and language instructions as well as swapped initial-final state pairs. The effectiveness is shown by comparing LORL to LORL (-Neg) in Table 2.
- Employing data augmentation in the image state space (e.g. affine transformations and color jitter) and the latent space of the language instruction (i.e. noise).
- Using a pre-trained language model (a Distilbert sentence encoder). This allows generalization to unseen instruction as shown in Table 1 and Figure 7.

**Clarity**
The writing is amazingly clear and the paper organization is easy to follow. I would even say this paper has the best writing among all the papers (from five major conferences) that I have reviewed so far this year. Reading this paper is enjoyable. I really appreciate the authors' efforts in polishing the writing.

**The amount of the offline data required**
In my opinion, learning a binary classifier should require way fewer data compared to learning a multi-task RL policy or imitation learning. Therefore, I believe the idea of learning just a classifier instead of a policy from the offline dataset can reduce the amount of required data. Is this the main reason why the proposed framework outperforms the baselines that also learn from language instructions?

**Annotator expertise for collecting offline datasets**
To create an offline dataset that contains robot trajectories and their descriptions, one intuitive way would be first generating language instructions that describe certain desired behaviors and asking human workers to control robots to fulfill the instructions. This requires not only access to robots but also the workers' expertise in controlling robots. Therefore, it is less practical to collect datasets this way on a large scale. 

On the contrary, this paper instead first collects robot trajectories and then asks annotators to provide descriptions narrating the trajectories. This greatly alleviates the expertise needed for the annotators. Yet, producing meaningful behaviors can still be challenging.

**Ablation study**
The proposed framework employs several techniques and the provided ablation studies justify the effectiveness of some of them, including: 
- The use of negative examples with mismatched initial-final state pairs and language instructions as well as swapped initial-final state pairs, shown in Table 2.
- The use of a pre-trained language model, shown in Table 1 and Figure 7.

**Experimental results**
- The presentation of the experimental results is clear.
- The proposed framework is evaluated in both a simulated environment and with a real Franka Emika Panda robot. 
- The baselines are sufficient
    - To verify that the issues of representing tasks using goal states/images), a baseline (Pixel) that uses a ground truth goal-image is considered.
    - To compared against different ways of leveraging offline datasets, two baselines are considered:
        - Language-conditioned behavior-cloning (LCBC): leverages the offline datasets by performing imitation learning with language instructions.
        - Language-conditioned RL (LCRL): a multi-task RL policy that conditions on language instructions and learns from sparse reward.
- The proposed framework outperforms those baselines by a large margin.  

**Reproducibility**
Given the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible. 


## Paper weaknesses and questions

**Initial-final state pairs or videos**
To evaluate if a behavior aligns with a language description, this paper proposes to represent the behavior using an initial-final state pair.
Yet, doesn't this also have the issue of representing a goal using an image (e.g. task over-specification and under-specification)? For example, the task of rotating a faucet indefinitely (mentioned in the introduction) can't be dealt with using initial-final state pairs. Therefore, I wonder if it would be better to represent a behavior using the entire video.

**Pixel baseline**
- Pixel uses L2 pixel distance as a planning cost. I wonder if it would work better using a learned latent distance (even it needs extra learning to learn a latent space) or perceptual loss.
- To verify if representing a goal using an image has the task over-specification, it would make sense to use a masked goal image to only show the parts of the image that are relevant to the task. If this masked goal baseline performs better, the task over-specification does exist.

**Learning from nonmeaningful behaviors**
This paper mentions many times that the proposed framework can learn from nonmeaningful behaviors (i.e. trajectories collected from random policies). I have a few questions regarding this statement as follows.
- L314-317 "We filter out episodes for which annotators wrote “do nothing” or indicated they could not understand what the robot was doing.": why is it necessary to filter those data then? 
- Aren't data collected from random policies mainly contain those "do nothing" behaviors? 
- So, my guess is that the classifier would not learn well if the majority of the offline dataset is nonmeaningful due to its imbalance. However, this is what you would get from a random policy. If this is the case, I think the authors oversell this a little bit and I would suggest the authors tone down and just claim that it can learn from sub-optimal data but not random data. Since most random data might not even contain interactions with objects, so it still needs to assume that behaviors are most meaningful in some ways.
- Following the previous point, I would like the know the performance of the model if those data are not filtered out.

**Performance of the classifier**
The experimental results only present the performance of the full pipeline of the proposed framework. I believe it would be informative to also evaluate the performance of the classifier alone. 
- I suggest the authors split the offline dataset into training/validation/testing datasets so that the training dataset can be used to train the classifier, the validation dataset can be used to determine the hyperparameters, and the testing dataset can be used to evaluate the performance of it.
- In this case, it would be easier to conduct ablation studies to verify if those techniques employed to train the classifier are effective (see below).

**Missing ablation study**:
This paper employs several techniques to train the classifier, but some ablation studies are missing to evaluate if those techniques are useful.
- The noisy labeling scheme is used to create more positive examples. 
- There are two types of negative examples introduced to train the classifier, including mismatched initial-final state pairs and language instructions as well as swapped initial-final state pairs. It would be informative to see which one contributes more to the performance gain.

**Collecting the offline dataset**
Is some sort of instruction template used for collecting instructions? This is probably the case so that 50,000 episodes only share 2311 unique instructions and this is also why there are experiments on generalizing from scripted language instructions (probably instructions templates) to unseen natural language. If this is the case, the authors should explicitly mention this in the paper.

**The amount of the offline data required for the multi-task RL policy and the behavioral cloning model**
As mentioned earlier in the strengths section, I assume that learning a binary classifier requires fewer data compared to learning a multi-task RL policy or a behavioral cloning model conditioned on language instructions. Therefore, I suspect that the performance gap might be attributed to insufficient data for learning the multi-task policy and BC model. To verify that, we can increase the data in the offline dataset and see if the performance of the proposed framework is the same yet the performance of the multi-task policy and BC model increases.

**Related work**
It would make the related work section more comprehensive by including some prior works that explore using formal languages (i.e. programs) as a task representation, including
- Modular multitask reinforcement learning with policy sketches
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
- Programmable agents
- Program Synthesis Guided Reinforcement Learning
- Program Guided Agent 
- Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving
- Reinforcement Learning of Implicit and Explicit Control Flow in Instructions

summary_of_recommendation: 
I believe this work studies a promising research direction (i.e. natural language instruction following with offline datasets), proposes an interesting framework, introduces effective techniques, and presents comprehensive evaluations. While I still have some minor concerns and questions (see above), I would like to see this work presented in CoRL. However, I would consider myself more a machine learning and reinforcement learning guy rather than robotics so my evaluations mainly come from learning perspectives.
issues: 
Described in the strengths and weaknesses section.
review_title: null

Meta-review: All reviewers agree that the presented approach is interesting and
that natural language descriptions pose compelling alternatives to goal images wrt task goal specification.

The paper would benefit from a discussion regarding limitations and scalability (especially wrt to real world applications).
Furthermore some claims and statements, as pointed out by the reviewers, should be clarified and referenced (where appropriate).

Most of the criticism can be addressed trough clarifications.

## Post-Rebuttal Update:
The authors successfully addressed and clarified the reviewers concerns and incorporated their feedback into an improved version of the manuscript. 
Decision: Accept (Poster)