Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors combine a large pre-trained CLIP vision-and-language model with a Transporter network in order to facilitate training a multi-task policy conditioned on natural language inputs with limited data. They compare against CLIP and Transporter baselines on both trained language-conditioned tasks and transferred languaged-conditioned tasks. They find that this method is able to generalize effectively to novel combinations of attributes (such as seperating the notions of 'blue' and 'cube', so that they can solve tasks involving blue cubes after having seen tasks with blue non-cubes and non-blue cubes). In tasks involving this kind of conceptual decomposition, they dramatically outperform other baselines. 
strengths_and_weaknesses: Strengths: 
I find this particular kind of of generalization across attributes extremely interesting, especially considering that they achieve it solely through end-to-end training. Additionally, the demonstration that large pre-trained networks for non-robotics language and vision tasks can be used to enable more efficient training training for robotics seems significant, as data and pre-trained models for these domains are much more accessible. I was also pleasantly suprised at the amount of training time used to achieve this level of result -- 6 GPU days is a decent amount but feasible for most labs. 

Weaknesses: 
It's not entirely clear how the authors get from the affordance prediction to the actual actions on the robot. 
summary_of_recommendation: This result is both immediately useful and offers up new and promising opportunities for future research. The use of natural language is both incredibly convenient for use and adding new tasks, and suprisingly data-efficient. The work additionally offers interesting new avenues for how large pre-trained networks from other domains could be used to enable more efficient robot learning. 
issues: It's not entirely clear how the authors are moving from the affordance prediction to actions on the actual robot. Since the labelled data all seems to be labelled images rather than action sequences, my best guess would be that the authors are using the affordances to select pre-programmed manipulations to take, and what parameters to call those manipulations with. I'd like to understand this better though, so a little more clarification would be nice.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a framework for learning language-conditioned, vision-based manipulation tasks. The high-level idea is to separate the spatial aspect of manipulation (e.g., where to pick and place) from the language conditions that specify the manipulation target. The goal is to generalize to new language commands by leveraging a pretrained vision-language model. The visual manipulation framework builds on the Transporter network (Zeng et al., 2020), which models multistep manipulation as a dense image prediction problem. The language condition is supplied through a pretrained CLIP vision-language embedding model (Radford et al., 2021). By conditioning the Transporter prediction on the CLIP embedding and training on a fixed set of demonstrations, the framework is shown to zero-shot generalize to follow new language commands (e.g., picking blocks of unseen colors). The method is evaluated on a set of simulated and real-world tabletop manipulation tasks.
strengths_and_weaknesses: I like the overall premise of the paper: large-scale vision-language models should be able to help visual-manipulation models to generalize better. I also like the overall execution of the paper: the proposed method is a simple extension of the Transporter model to incorporate language conditions and is evaluated extensively on both simulated and real-world manipulation tasks. However, a few doubts remain on the main thesis: aligning vision-language representation with visual-manipulation representations brings “the best of both worlds”. 

First of all, to what extent does the “spatial / geometric” aspect of manipulation benefit from the “semantic” aspect of manipulation? Intuitively, CLIP is an excellent medium for generalizing to unseen visual concepts and combination of visual concepts in *visual recognition tasks*. However, I struggle to understand what kind of synergies between vision-language embedding and manipulation can be exploited beyond, e.g., correctly recognizing a new object that’s not shown in the demonstration because the pretrained CLIP model has that object before. 

For example, if the Transporter model is never trained with pick-and-place an awkwardly-shaped or a slippery mug, correctly recognizing that it is a mug shouldn’t help with its manipulation. In other words, if pick-place tasks are mostly about understanding the geometric/dynamic aspect of the objects, why would a vision-language model help with its generalization? We may further extrapolate this argument: if we have a detection model that can correctly localize novel objects (e.g., 2D position & orientations) given language commands, to what extent do the language-conditioned Transporter network benefit from the CLIP embedding? 

Second, language commands contain information beyond object categories and attributes. For example, in the example given by the paper “pack all the blue and yellow boxes in the brown box”, to what extent does the relationship “in the” constitute a reusable linguistic concept beyond identifying a task? For example, would the model be able to generalize to “place a dice in the mug'' or “grasp the mug in the cabinet”? I understand that the paper does not claim generalization to novel relationships or non-object-related concepts. But it is important to discuss such limitations or even present negative results.

Minor:
- 
Line 165: what does it mean to “downsample” the goal embedding? Are there downsampling operations like strided 1-D convolution involved or is it just a linear projection?
summary_of_recommendation: I am overall neutral about the paper. I'm recommending weak reject right now but it's more like a "borderline". I will likely change my rating after seeing additional discussion / caveats about the main thesis of the paper and further experiments regarding the limitations.
issues: See "weaknesses"
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a language-conditioned modification of Transporter (Zeng’20). Pretrained CLIP (Radford’21) modules are used for the visual and the language stream with fine-tuning on language-annotated demonstrations. The proposed model succeeds on a range of tabletop spatial manipulation tasks in simulation and real world. The model outperforms chance performance (i.e. no language input) on most tasks. Using depth as an additional input somewhat improves performance. Multi-task training improves data efficiency, especially when trained on larger datasets. The model somewhat outperforms standard pretraining of the vision and language models separately (e.g. ImageNet and BERT models), because it leverages joint vision-language pretraining (see App. Tab 4).

strengths_and_weaknesses: 
Strengths:
- The paper presents a promising system that can perform spatial manipulation tasks from language and visual input.
- Several interesting experiments and ablations on this system are presented.
- The proposed method leverages large-scale internet datasets via using the pretrained CLIP model, which is a promising direction for robotic control.

Weaknesses:
- As the paper notices on l319, the base Transporter architecture is quite limited in terms of tasks it can perform and it’s unclear whether the finding of this paper will generalize to more general architectures.
- The paper claims that “CLIP is essential for few-shot learning in lieu of alternatives like ImageNet-trained ResNet50 [56] with BERT [35].” That does not follow from the comparison in the App. Tab 4. In fact, ImageNet ResNet + BERT outperforms CLIP in the few-shot regime on two of the 4 tasks. 
- In general, the experimental evaluation is highly confusing and lacks many baselines that one would want to see. For instance, the no pretraining baseline or the ImageNet ResNet + BERT baseline are the natural choice, but they are omitted in favor of chance performance, a minor architecture ablation, and an ablation that uses more data. More informative baselines are provided in the appendix. Perhaps just switching the baselines in the paper for the baselines in the appendix would solve this problem? 


summary_of_recommendation: 
The paper attacks an interesting problem and presents a system that has many of the right parts to attack that problem. I am leaning towards accepting the paper, but the presentation of the baselines needs to be significantly improved for that.

issues: 
See weaknesses
review_title: null

Meta-review: CLIPort combines CLIP (a joint language and image embedding) with TransporterNets (a method for performing data efficient BC on sparse action manipulation problems) to enable multi-task NLP conditioned policy learning. Multiple reviewers praised the techniques ability to generalize via CLIP pretraining and it’s leveraging of large-scale internet datasets, as well as the perceived data efficiency of the approach. The authors addressed the bulk of reviewer feedback, adding clarifications for limitations as well as additional real-world experiments and baselines. The paper achieves compelling real-world results and should be of interest to the CoRL community.
Decision: Accept (Poster)