Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contributions of this paper could be concluded as following:
(1) Proposed a transformer-based model to build mapping from natural language instructions to real scenes with pixel-wise Q values and masks which are compatible with both supervised learning and reinforcement learning methods.
(2) Apply the proposed transformer-based model on multi-step rearrangement tasks and compare it with a baseline model:Unet-based VGP model in metrics of block acc and IES
(3) Give out a dataset based on Bisk dataset mentioned in paper with both real and simulated samples to for training model for multi-step rearrangement task in more realistic scenes

strengths_and_weaknesses: Strengths:
1.	Innovatively proposed the method to mapping input image with natural language instructions with pixel-wise Q-value and masks which is feasible in practice and useful
2.	The proposed model was validated that it can converge fast with much fewer samples than traditional transformer model in training phase.
3.	Design a complete experiment setting to compare propose model with baseline in aspects of training efficiency, genralization toward different taks types et al.
Weakness:
1.	The mapping lack explainability, a theoritical explaination or citiation should be presented to clear the reasonability of propose mapping method
2.	Data ablation experiments should be more comprehensive. Fisrt, generalizability should be taken into consideration, a different dataset could be used to validate the propose model can still improve fast or not. Second, robustness should be validated. For example, add some noisy into train set.
3.	The released dataset need to validated in several aspects such as data type balance,reasonablity of insturctions and size

summary_of_recommendation: I recommend this paper for its contributions on proposing  a transformer-based model to solve multi-step rearrangement task in realisitc simulated environment as it provides an innoative and feasible method to map natrual language instructions to input scene images and the experiments shows that it could be more effective and efficient in training than most transformer-based models.
issues: 1.	Explainability of propose pixel-wise mapping method
2.	A more comprehensive validation of effiency of propose transformed method
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work leverages recent vision models to perform natural language guided block stacking.  They use synthetic language and a real crowdsourced resource (filtered for physical viability).  Limited experiments (due to covid) are performed on a physical manipulator.  The task requires understanding spatial language expressions for both identifying the referent and target location.
strengths_and_weaknesses: **Strengths**
1. Novel model architecture for the task with good comparison baselines
2. Analysis of simple stacking and rows in both synthetic and real images.
3. Curation of the Bisk et al. dataset for robotic tasks and evaluation on this real language setting
4. Manipulation performed with simulated UR-5
5. Basic manipulation demonstration of simplest setting on physical manipulator

**Weaknesses**
1. Insufficient analysis of the difficulties of real language (e.g. behavior in abstract language settings)
2. Limited physical manipulator experiments which also means limited analysis of the causes for the model's transfer errors to the real setting.
summary_of_recommendation: The authors present and demonstrate the utility of novel models for language guided control/actions based on recent advances.  They do basic comparisons against language and task complexity and include experiments both on simulated and real control platforms. 
issues: The primary limitation of the current work is limited error analysis which makes it difficult to diagnose what the next steps are for the research community and ideally more physical experiments (though this is not a limitation that can be expected to be addressed in the current climate.)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes using natural language instructions for multi-step rearrangement tasks, by mapping images and instructions to masks over pick or place locations. The model is factorized into 2 components — a pretrained SPOT-Q model independent of language, and a language-conditioned model proposed by the authors. The predictions from these two components are combined to obtain the final action.
strengths_and_weaknesses: Strengths:
- Experiments are conducted both with synthetic and real images, as well as templated and crowd-sourced language. The results show that the proposed method outperforms previous methods on both visual goal prediction and on task completion metrics.
- The factorized model is interesting, and allows reusing pretrained models, thereby reducing data requirements.

Weaknesses:
- The authors discuss the advantages of using a factorized model, but do not discuss the limitations. For example, language might have more information than "where to act", like "put the red block *gently* on the blue block".
- Some analysis of how the performance changes as the no. of steps in the task is increased could be informative.
- It seems like the natural language experiments (based on the dataset from Bisk et al) involve moving only a single block. If this is true, it doesn't represent the "multi-step" rearrangement setting being addressed in the paper, and needs to be made explicit.
- The experiments on compositionality aren't particularly informative — it seems pretty intuitive that the model will learn concepts like "red block" and "blue block" independently. A more interesting question is — can the model learn to compose different object attributes? E.g. if it is only shown "small red blocks" and "large blue blocks" during training, can it generalize to "large red blocks"?
- The experiments with the physically feasible subset are not convincing — (1) are the differences statistically significant?, (2) could the difference be because the physically feasible dataset is smaller in size?
summary_of_recommendation: The proposed approach is sound, and the experimental results show that it improves over existing methods, but some additional experiments could make the paper better.
issues: - More convincing analysis of physically feasible subset.
- Clarify whether the natural language setting is single-step or multi-step.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 
This paper addresses the problem of learning to perform block rearrangement tasks described by natural language instructions. To this end, the paper proposes a framework that decouples the problem into two sub-problems: where to act and how to act. Specifically, it proposes a Transformer-based model to map images and language instructions to masks over grasp and or place locations. Then, a pre-trained policy performs actions according to its Q function that provides pixel-wise Q values and predicted masks. The experiments show the proposed framework can reliably solve the task. Ablation studies verify design choices such as language models (GloVe and BERT), the encoder architecture (UNet and Transformer), etc. I believe this work studies a promising research direction (i.e. language instruction guided agents) and proposes a reasonable framework to address the problem. Yet, I am concerned with the narrow scope of the problem and the limited novelty and technical contributions.
strengths_and_weaknesses: 
## Paper strengths and contributions

**Motivation**
- Leveraging natural language descriptions to instruct robots is easy and natural especially for non-expert users and flexible.
- Employing natural language instructions for learning a multi-task policy allows leveraging semantic similarity across tasks, which potentially allows generalizing to unseen tasks.
- Decoupling learning where to act and how to act could potentially be more sample efficient compared to learning an RL policy from scratch purely from reward signals. Also, such as framework is more interpretable since we can tell if it fails to predict correct locations to apply actions or it just cannot precisely perform actions.

**Technical contributions**
- Predicting segmentation maps indicating where to grasp or place seems effective
- Employing a Transformer to process natural language instruction yields better performance compared to UNet.

**Ablation study**
Ablation studies justify some of the design choices:
- UNet vs. Transformer: the results show that employing a Transformer model yields better performance compared to an UNet baseline.
- GloVe vs. BERT: the BERT-based Transformer has slightly higher block accuracy and performs well when trained from states.

**Experimental results**
The experimental results show that: 
- The proposed Transformer-based visual goal prediction (VGP) model can predict segmentation maps for grasping and placing reasonably well in terms of block accuracy and idealized error score (IES).
- The proposed framework achieves a higher task completion percentage with a lower average number of actions per trial.
- The proposed framework can learn from thousands of simulated training images and successfully be fine-tuned with around 2k real images to achieve reasonable performance (with an IES around 0.7).
- The proposed framework can deal with unseen color pairs.

## Paper weaknesses and questions

**Novelty**
The novelty, from both the views of theoretical analysis aspect and empirical contribution perspective, is not clear or not explicitly mentioned.  Therefore, I do not find enough novelty from any aspects while the efforts put in for setting up and evaluating the proposed system are highly appreciated. To me, this seems more like a system paper than a learning paper.

**Two-stage pipeline**
This paper proposes to decouple the block rearrangement problem into where to act and how to act. While I do recognize the advantage of this decoupling scheme, there are still some issues:
- Error propagation: errors produced in each stage can accumulate and lead to undesired behaviors.
- The intermediate representations: intermediate representations (such as segmentation maps in this case) need to be chosen beforehand and the chosen one may not be the best for the entire system. The justification suggesting that a segmentation map is a good representation for this task is missing from this paper. Some other choices, such as object detection boxes, could have been evaluated.

**Limited problem scope**
While the proposed framework and ideas are applicable to many problem families, the only task considered in this paper is block rearrangement. I encourage the authors to apply the proposed framework to other domains to increase the significance of this work.

**Related work**
It would make the related work section more comprehensive by including some prior works that explore using formal languages (i.e. programs) as a task representation, including
- Modular multitask reinforcement learning with policy sketches
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
- Programmable agents
- Program Synthesis Guided Reinforcement Learning
- Program Guided Agent 
- Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving
- Reinforcement Learning of Implicit and Explicit Control Flow in Instructions

**RL only baseline**
It is not fair to compare the proposed framework against the RL-only baseline this way since the RL-only baseline does not leverage the segmentation map supervision and does not learn from other tasks like "SPOT-Q". Some fairer comparisons could be: (1) learning the RL policy with an auxiliary task of predicting segmentation maps, (2) learning a multi-task RL policy to solve the block rearrangement tasks with multiple configurations.

**Unseen color pairs**
While the ability to generalize to unseen color pairs (e.g. stack the red block on the blue block) is evaluated in this paper, the evaluation setup is very limited. It limits the number of unseen color pairs to only 1 but all other 55 combinations are seen. I would say holding out at least 10-20% makes more sense.

**Grid downscaleing factor**
Instead of predicting segmentation maps with the same size as the input images, this work proposes to downscale it. However, the downscaling factor would depend on the size of the object of interest and therefore it is not flexible.
summary_of_recommendation: 
I believe this work studies a promising research direction (i.e. language instruction guided agents) and proposes a reasonable framework to address the problem. Yet, I am concerned with the narrow scope of the problem and the limited novelty and technical contributions.
issues: 
Described in the strengths and weaknesses section.
review_title: null

Meta-review: After the author response and the discussion phase, the reviewers agree that there is room for improvement, but appreciate both the key idea of the paper of using pre-trained models to efficiently learn language-conditioned tasks and the effort of demonstrating results going from real language all the way to low-level robot control. The results on a real robot are also appreciated and make it particularly well-suited for CoRL. I recommend accept.

--------------------------------------

This paper seems to be somewhat borderline. It presents an interesting method to a relevant problem, includes some ablations to understand the importance of different design choices, and includes experiments with natural language along with some experiments on a real manipulator.

The paper also has several weaknesses, including the following:
* The experiments are limited to block re-arrangement tasks, rather than a more general class of rearrangement tasks. It does not seem like the method is limited to such simple block-based problems, but the experiments do not validate its generality.
* There are a number of experiments that would strengthen the empirical analysis, including a stronger RL baseline, more analysis on the failure cases with natural language, and more experiments on a real robot. (see comments from reviewers kWqo and h8PP). Including this more detailed analysis and comparisons would make it possible for the reader to identify/understand important future steps in this research area.
* The related work section does not discuss the relation of this work to a number of relevant works. Beyond the works mentioned by reviewer kWqo, there are additional prior works that study rearrangement & manipulation tasks from language instructions [1,2,3].

The final result of the paper will depend on the author response and reviewer discussion. If some of the weaknesses can be addressed, then it will most likely be above the bar.

* [1] Learning to Understand Goal Specifications by Modeling Reward. ICLR 2019.
* [2] Language as an Abstraction for Hierarchical Deep Reinforcement Learning. NeurIPS 2019.
* [3] Language Conditioned Imitation Learning over Unstructured Data. arXiv 2020 / RSS 2021
Decision: Accept (Poster)