Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Interesting paper, it uses language and demonstration conditioning to improve learning in an behaviour cloning set up for robot manipulation. The paper proposes:
1. a new architecture for training an eval conditioned on language and demonstrations together
2. some level of generalization with this
strength_and_weaknesses: Strengths:
- pretraining demonstration and language using contrastive loss is an interesting idea
- it's intuitive that conditioning on both improves performance, is a simple and elegant idea
- generalization results look decent
- the conclusion that language is worth 50 demos is useful, but this breaks down once you've 1000s of tasks

Weaknesses:
- tested on only sim, where data is very clean
- concerned that conditioning on one demo would make a policy rely on the spurious correlations of that demo : specific action situations, background, etc. 
- given several demos of the same task, how would you sample the one to condition on? how do you scale this for large datasets

clarity,_quality,_novelty_and_reproducibility: Clarity:
- very clear

Quality
- good quality

Novelty
- reasonably novel

Reproducibility
- high
summary_of_the_review: The paper proposes a method to condition on language and demo to solve robotic behaviour cloning. This is a decent set up in small-scale robot learning set ups. The idea is novel and simple. The claims of the paper are reasonably well supported within the data distribution used.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops an approach, Joint Demo-Language Task Conditioning (DeL-TaCo), for behavior cloning in a multi-task setting where the task is specified by both language and a demonstration. The approach includes a contrastive loss to guide the learning of the language and demonstration embeddings, which are then fed into a policy head. Results are conducted in simulation and show that virtual robot manipulation tasks can be learned by DeL-TaCo and that this approach generalizes better to novel tasks.
strength_and_weaknesses: Strengths:

The paper clearly describes a supervised learning approach to multi-task learning with tasks conditioned on both a natural language instruction and a demonstration of the task. 

The figures are relatively informative.

Pseudocode is provided.

The paper reports positive, empirical results on a relatively large dataset of multiple tasks.

Table 3 shows how much data is required to enable a demo-only policy to match the performance of the proposed method, DeL-TaCo, showing the proposed method is better up until reaching ~50 demonstrations for the dataset for the novel task.

A limitations and future work section is included.

Weaknesses:

This approach requires both language and instructions to function. As such, one would want to show superior performance to methods that require only one. For example, recent work by Wen et al. (2022) required only a single demonstration on a novel task:

Wen, B., Lian, W., Bekris, K. and Schaal, S., 2022. You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration. arXiv preprint arXiv:2201.12716.

On the language side, there are quite a few recent papers show strong performance conditioning on only language:

Sodhani, S., Zhang, A. and Pineau, J., 2021, July. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning (pp. 9767-9779). PMLR.

Silva, A., Moorman, N., Silva, W., Zaidi, Z., Gopalan, N. and Gombolay, M., 2021. LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation. IEEE Robotics and Automation Letters, 7(2), pp.1635-1642.

Nair, S., Mitchell, E., Chen, K., Savarese, S. and Finn, C., 2022, January. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning (pp. 1303-1315). PMLR.

Shridhar, M., Manuelli, L. and Fox, D., 2022, January. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning (pp. 894-906). PMLR.

This reviewer notes that the authors discussed CLIPort and the Nair et al. (2022) paper; however, these works were not benchmarked against. The paper does not present a clear argument against benchmarking against these approaches (other than the implicit argument that they only condition on language, which is actually an advantageous quality).

This paper is a relatively straightforward design of a neural network-based architecture where embeddings are learned for language instructions and demos (which are forced to be the same through a contrastive loss) and then fed into a policy head for supervised learning. It is unclear whether new theories or insights are gleaned (other than adding more input data can enhance performance by 5-9%).

More insight into why DistilBert was used could be helpful. What would other language embeddings look like?

An ablation study showing the efficacy of the contrastive loss and other components would be helpful.

clarity,_quality,_novelty_and_reproducibility: The paper is relatively clear, and the figures are quite nice.

The novelty of the paper is relatively minor. Prior work has explored conditioning on language and demonstrations, though, the combination of the two is more unique.

The paper does not appear to have code for reproducibility, though, additional details are provided in the appendix.
summary_of_the_review: This paper provides positive results for conditioning on language and demonstrations in multi-task learning for zero-shot generalization. However, the paper would be improved by more baselines and a more compelling case that the results are novel and significant.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper argues that solely using language or video demonstrations to learn robotic manipulation tasks is inadequate as it creates ambiguities in learning. To alleviate this confusion, the authors propose to train a multi-task policy that condition on *both* the video demonstrations as well as the language.

They work in the paradigm of imitation learning, wherein they train a single policy for all the robotic pick-and-place tasks.
strength_and_weaknesses: **Strengths**

1. The paper is mostly well-written, and the methodology section is easy to understand in one go.

2. Section 5.3 which compared how much was a language worth was a good discussion -- which drives the point that having language as input is indeed beneficial (as corroborated by several other works in the field)!


**Weaknesses + Questions:**

**Claims**
1. The authors claim the following in Sec 2.2

> "We argue that conditioning the policy on both a demonstration and language not only ameliorates the ambiguity issues with language-only and demonstration-only specifications but is much easier and more cost-effective for the end-user to provide."

I'm not sure what the authors mean by "more cost-effective for the end-user to provide" -- I agree that providing numerous demonstrations (at least 50 according to the experiments) has a high cost on the user, but having only language would have a lower cost (as done in [1])! 


**Methodology**

2. The rationale behind the Task Conditioning Architecture: How was the architecture selected? Was it based on previous works of [2], and [3]? 

    A simple variant (baseline) would be to concatenate the $z_{demo}$ and $z_{lang}$ and use that as input to the policy. Was this baseline tried? If so, what were the observations?

**Experiments**

3. Success Rate: I am not sure why this complicated way of computing the Success rate was adopted. Specifically, I'd like the authors to address the following questions:

   a) Why are the rollouts *during* the training being considered? What is the problem with computing the success rate after training (800-900k steps)?

   b) How does the current definition of success rate avoid *cherry-picking* when compared to (a)?

4. Training + Test Sets (a Scenario C): In all the experiments, the three objects in the scene are *visually distinct*. Having the same objects with a) the same colors, as well as b) different colors in the test set would be a crucial experiment to test the generalization of the method. I would like to see the results of such an experiment. Since it is only testing, I believe it will not take a lot of time.

5. Baseline Comparison: I would expect a comparison with [1] to be present in the paper. I understand that the code for [1] is unfortunately not open-sourced (which I don't hold against you, and blame the authors of [1]), but the method in [1] can be adapted to your codebase wherein you have a shared latent space -- which is essentially the crux of [1].


-------
**References**

[1] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. In RSS 2021.

[2] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In CoRL 2021.

[3] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
clarity,_quality,_novelty_and_reproducibility: **Writing**

6. How were the demonstrations collected? : On page 7 under the sub-section of *Data*, the authors mention that a "scripted policy" was used to collect demonstrations. What was this scripted policy -- a policy that uses inverse kinematics? Please mention the details of it.

**Section 4.1**

7. The variables $m$ and $n$ are introduced in the first paragraph and nowhere in the main paper are they defined. It is only in the appendix that their descriptions are given. Please define what they are for clarity.

8. Under the *Task Conditioning Architecture* sub-section, please briefly describe what *FiLM* does. One sentence should be sufficient -- otherwise, it makes the reader switch back and forth between the original paper and this.


**Visual Aesthetics (not considered for decision-making)**

9. I believe adding plots instead of the table would be much easier to read (except for Table 3). I say this after looking that the authors have used plots on the project webpage, which have better readability and also look better.

**Reproducibility**

10. The authors do mention the details of the model in the appendix including the hyperparameters they used. However, I would like to ask if the authors plan to open-source their code, if accepted to the conference?
summary_of_the_review: Even though the paper's contribution in methodology is significant enough for the community, due to the lack of extensive experiments and proper baselines as mentioned in the Weaknesses section, I would vote 5 : *marginally below the acceptance threshold*. However, my final decision is subject to change based on the authors' rebuttal and comments of other reviewers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Joint Demo-Language Task Conditioning, DeL-TaCo, as a new way of training and specifying goals for goal-conditioned task-learning agents. DeL-TaCo learns a demonstration encoder and uses a frozen, pre-trained language encoder to embed task demonstrations and language-specifications for simulated robots to perform pick-and-place tasks with various objects/colors/shapes. DeL-TaCo is compared to language-only and demonstration-only ablations, showing that both demonstrations and language are important to achieve the highest success rates, and two different encoders are compared, a pre-trained CLIP vs. a pre-trained DistilBERT + trained demonstration encoder.
strength_and_weaknesses: Strengths:
* The method is clearly described and intuitive, and is a natural next-step after prior work.
* Combination of demonstrations and language appears to provide significant improvements over either modality individually.
* The work is positioned clearly with respect to BC-Z.
* The authors developed a simulator/simulated task for their work.

Weaknesses:
* The proposed method feels like a solid first-step, but many natural follow-up questions are left unexplored (what if the language encoder is fine-tuned? Why not pass the CLIP embeddings through a small, fine-tuned MLP? How does a pre-trained vision module work for the demo encoder? etc)
* Training the demonstration encoder seems to require prior knowledge on the number of possible tasks, limiting the generality of DeL-TaCo  to situations with fixed numbers of tasks, despite the goal-encoding being quite general.
* Experimental reporting feels disjointed and incomplete. While the discussion seems to suggest that language is superior to demonstrations (e.g., saying that language is worth 50 demonstrations), in fact the demo-only method performs better than the language-only method in both experiments. There are also no upper-bound numbers for Table 2, so we can see that DeL-TaCo is better than demo/language-only, but we don't know how far it is from "perfect" performance.
clarity,_quality,_novelty_and_reproducibility: Clarity:
* The work is clearly explained and the method is easy to understand.
* The task description is more difficult to understand directly from the text, but the appendix helps to understand what objects were used, what "shape" means and how it is distinct from "object", etc.

Quality:
* The work is high quality and appears to have been conducted rigorously, though natural questions around training details or specifications appear to be left unanswered.

Novelty:
* The work has not compared thoroughly to related work, except through ablations of the proposed method.
* Prior work on language encoding for goal-conditioned agents is missing [1, 2]
* The overall method is a relatively straightforward combination of prior work without significant additional contributions, though this is not a reason to reject as the method is well-explained, well-motivated, and performs well.

[1] Sodhani, Shagun, Amy Zhang, and Joelle Pineau. "Multi-task reinforcement learning with context-based representations." International Conference on Machine Learning. PMLR, 2021.

[2] Silva, Andrew, et al. "LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation." IEEE Robotics and Automation Letters 7.2 (2021): 1635-1642.	
summary_of_the_review: The proposed method, DeL-TaCo, is explained clearly, is intuitive and straightforward, and appears to work well. The paper suffers from incomplete reporting in the results, a relative lack of novelty, and unanswered questions about training details or design decisions. With simple additional experiments around fine-tuned language encoders, clearer results for demonstration-vs-language goal-specification and performance upper-bounds, and position to recent related works, I would advocate for acceptance. As the paper reads currently, it does not seem publication-ready.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of learning multi-task policies from a combination of visual demonstrations and language utterances as complementary modalities. The paper proposes DeL-TaCo, a framework that conditions policies on language- and demonstration-based task embeddings that are aligned using contrastive learning. The framework is evaluated on a series of simulated robot manipulation tasks, with comparisons to single and multi-modality baselines. The results demonstrate the ability to learn multi-task manipulation policies that better generalize to novel tasks.

The paper was reviewed by four reviewers who agree that the idea of conditioning on both language and demonstrations as mutually beneficial sources of "supervision" is sensible and that the proposed framework is intuitive. The reviewers appreciate the analysis of the "value" of language input and find the paper to be very clear. The reviewers raised several concerns with the initial submission, notably the need for qualitative and quantitative comparisons to other single- and multi-modality baselines; additional experiments that explore greater test-time variations; and perceived inconsistencies between some of the claims made in the paper and the experimental results. It is clear from their response that the authors have put a significant amount of effort into responding to these concerns, which includes conducting several additional experiments as requested by the reviewers. The reviewers and AC appreciate these efforts, which have helped to clarify some of the initial reviewers' concerns initially raised, as reflected by two reviewers increasing their overall score.

This is an interesting and promising research direction. While the authors' response has certainly helped, the significance of the work is not yet fully clear. The paper would benefit from a more detailed analysis of the merits of language- and demonstration-based conditioning over their single modality counterparts (e.g., the additional experiments that investigate generalizability to test-time variation seem to suggest that DeL-TaCo is advantageous when language supervision is helpful, as measured by the language-only baseline, but can be inferior to a learning algorithm that only utilizes demonstrations). The authors are encouraged to consider a version of the paper that more thoroughly investigates the ability for the framework to take advantage of using language and demonstrations as complementary signals would provide a solid contribution to the robot learning community.
justification_for_why_not_higher_score: The authors clearly put a lot of effort into responding to the reviews, which has lead to at least two reviewers revisiting their score.

It is worth noting that while one reviewer recommended 'accept', they listed their confidence level as being limited. As a result, the AC places less weight on that persons' review.
justification_for_why_not_lower_score: As noted above, the AC would not object to the score being lowered in light of the reviewer who recommended 'accept' indicating their limited confidence in their decision. This, together with the fact that the AC personally knows (and trusts the opinion of) one of the reviewers make the AC less confident in the paper.
Decision: Accept: poster