Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control
Authors: Vivek Myers
Andre Wang He
Kuan Fang
Homer Rich Walke
Philippe Hansen-Estruch
Ching-An Cheng
Mihai Jalobeanu
Andrey Kolobov
Anca Dragan
Sergey Levine
Venue: CoRL 2023 Poster (2023)
Keywords: Instruction Following
Representation Learning
Manipulation

Abstract: Our goal is for robots to follow natural language instructions like ``put the towel next to the microwave.'' But getting large amounts of labeled data, i.e. data that contains demonstrations of tasks labeled with the language instruction, is prohibitive. In contrast, obtaining policies that respond to image goals is much easier, because any autonomous trial or demonstration can be labeled in hindsight with its final state as the goal. In this work, we contribute a method that taps into joint image- and goal- conditioned policies with language using only a small amount of language data. Prior work has made progress on this using vision-language models or by jointly training language-goal-conditioned policies, but so far neither method has scaled effectively to real-world robot tasks without significant human annotation. Our method achieves robust performance in the real world by learning an embedding from the labeled data that aligns language not to the goal image, but rather to the desired change between the start and goal images that the instruction corresponds to. We then train a policy on this embedding: the policy benefits from all the unlabeled data, but the aligned embedding provides an *interface* for language to steer the policy. We show instruction following across a variety of manipulation tasks in different scenes, with generalization to language instructions outside of the labeled data.


Reviews:
[{"review_id": "CP-ZgrC1X4d", "reviewer": "Reviewer_z7wQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes GRIF, an approach that leverages unlabeled trajectory data to train a language-conditioned policy. GRIF first train a shared task representation between language and task transition (i.e. pair of the initial state and the goal) to align both embeddings. Then they can train the goal-conditioned and language-conditioned policy together and leverage the unlabeled goal-conditioned data to improve the language-conditioned policy. This allows the robot to learn a language interface with few labeled language data. The experiments on real robots show that the learned policy outperforms baseline language-conditioned policy and the aligned representation has a better generalization in a retrieval task. ", "summary_of_recommendation": "The paper is well presented and the proposed method can potentially benefit training a policy with few labeled data. However, I’m leaning toward rejection, as discussed in the weaknesses and questions, it is unclear about the novelty, generalization, and how the language augmentation impacts the presented method. \n\n-----\n\nPost rebuttal update\n\nThe authors' responses addressed my questions. While there are limitations and shortcomings as pointed out by reviewer HoyX, it is a useful approach to leveraging visual goals to improve the limited language goal data and a well-executed system, I'm updating my recommendation to acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- As discussed in the weaknesses, the paper needs to address the questions about novelty, use of image encoder in the experiment, and the test for generalization\n- This paper claims that GRIF can be trained with fewer language annotations. The appendix shows that the totral number of trajectories is 54k and 7k of them contain language annotation. The appendix also mentioned that the language instructions are augmented using ChatGPT3.5 so this brings the language instruction to 35k. In this case, the number of language instructions is not as little as one may imagine. How does data augmentation impacts the performance of GRIF? And how sensitive the proposed performance is to the number of annotated trajectories? These experiments will be important to demonstrate the effectiveness of the proposed method.\n- (Minor) The tested tasks seem relatively simple, consisting only of \"put,\" \"place,\" and \"move\" instructions. If more complex tasks (like in other datasets, e.g. CALVIN, MetaWorld, etc) were included, such as \"slide the door to the left\", \"rotate the red block\", or “turn faucet off,” how will this impact the proposed method?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- The proposed method enables robots to get a language interface with little data. This is useful for training robots when the labeled data is limited.\n- This paper clearly presents the hypotheses of the proposed method and provides the experiments to verify them.\n- The paper presents the method and results clearly. It is easy to follow the paper.\n\nWeaknesses\n- The idea to use contrastive learning to leverage other modalities (where more unlabeled data is available) in robot control has been explored in some of the prior work. For example, Language-Image Value learning is a recent work that leverages video data and contrastive learning for learning a reward for robot control. The paper needs to discuss how the proposed method differs from the prior explorations to demonstrate the novelty.\n- While the GRIF outperforms all baselines, it is worth noting that the baseline methods employ ResNet18 as the image encoder whereas GRIF employs a modified CLIP model which uses ResNet50 as the image encoder. Some of the performance improvement may come from the bigger model capacity of ResNet50. It will be a fairer comparison if all presented methods can use the same ResNet50 encoder as in the modified CLIP.\n- The generalization evaluation is limited. They use a retrieval task (i.e. retrieval of transitions specified in natural language) to demonstrate that the learned representation can generalize to other language instructions. However, this is demonstrated in the experiments for the original CLIP, and it doesn’t show if this representation can help generalization in learning a language-conditioned policy. A better experiment will be having a test set where the language instructions are novel and measure the robot success rate of the novel instructions.\n\nMa, Y. J., Liang, W., Som, V., Kumar, V., Zhang, A., Bastani, O., & Jayaraman, D. (2023). LIV: Language-Image Representations and Rewards for Robotic Control. ICML 2023.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "BeSDYdZhOyP", "reviewer": "Reviewer_mB1L", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper describes a language understanding architecture for pick-and-place-like tasks that can be trained using only a relatively small amount of annotated data paired with a larger amount of unlabeled (self-play) data. Integral to the framework is the idea of learning encoders that align the task representation expressed in natural language $l$ with an encoding of the corresponding start-goal pair $(s_0,g)$. The paper learns this alignment using contrastive learning based on $(s_0,g,l)$ tuples from the labeled dataset. The unlabeled dataset (i.e., $(s_0,g)$, but no $l$) is used to fine-tune a pretrained vision-language model (CLIP). The method trains a downstream policy that is conditioned on the task embedding either jointly or with the embeddings frozen. The paper compares the method to several recent baselines on a series of table-top pick-and place tasks and ablates the different model components.", "summary_of_recommendation": "The paper considers an important problem and the topic of language understanding for robotics is of significant interest lately. The paper is missing qualitative and (ideally) quantitative comparisons to a highly relevant baseline. Some of the results are counter to what is expected, particularly those that involve ablations of the initial state. Further, the analysis is limited to only a handful of very similar instructions. Together, this makes it difficult to evaluate the significance of the contributions.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1) What is the nature of the acton space?\n2) As noted above, can the authors speak to the differences relative to CLIPort and (ideally) provide a quantitative comparison?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "STRENGTHS\n+ The ability to train a language understanding model with relatively few labeled datapoints and a larger set of unlabeled datapoints is of practical importance.\n+ The use of pre-trained vision-language models is sensible (and standard), though there are questions about the means by which they are adapted as noted below.\n+ The idea of coupling the initial and goal states together when learning the task embedding is sensible, particularly for tasks that can not be specified in terms of the goal state alone (i.e., as a means of learning a representation of the task as the \"change\" in the state).\n+ With the exception of a small number of grammatical errors, the paper is well written and well organized.\n\n\n\nWEAKNESSES\n- The overall method is only evaluated on 15 different commands that involve pick-and-place operations with a smaller number of objects (i.e., many of the instructions differ only in the pick and place object/location). Of these, the proposed model performs better than the baselines on 8 instructions. Given the small sample size, it is hard to tell whether these gains are significant. Note that under \"Robotics Focus\" below, there is no option for \"some hardware experiments but more necessary\" and so I selected \"sufficient demonstration on hardware\" over the other options, but I feel that more experiments are needed.\n- The paper neither cites nor compares to CLIPort, which has been shown to be capable of pick-and-place tasks very similar to those considered here, at least those that don't involve relative motion (i.e., Scenes A and B, which involve 11 out of 15 of the instructions). The paper should provide a quantitative comparison to CLIPort along with a discussion of their differences.\n- The main paper provides no details regarding how it adapts and fine-tunes CLIP to model changes in state. Given that the paper suggests that this is a contribution that significantly improves the generalization of the framework, and given the questions about the ablations below, it is important that the details be provided in the main paper. \n- Out of the 15 instructions, only 4 (perhaps only 2) (Scene C) require reasoning over the relative location of an object. It is not clear why explicitly representing the task in terms of the initial and final states is necessary. Interestingly, the performance gap between GRIF and the baselines (Fig. 3) is the smallest for these 4 instructions compared to those in Scenes A and B.\n- Related, it is not obvious why the performance of the \"No Start\" ablation is so poor given that the task doesn't require reasoning over the change between the start and goal states. Without more detail regarding the way in which CLIP was modified, one wonders if this is a result of these modifications (unless a vanilla CLIP was used for this ablation vs. the modified CLIP but without the initial state).\n- Unless I missed it, there is no discussion in the main paper or appendix of the robot's action space.\n- The contributions of the analysis of the learned task representations (Section 5.3) is limited. Further, as the method is using a pre-trained CLIP, the extent to which the language is \"new\" is unclear.\n- Referring to the fact that the model reasons over the initial and goal states as involving \"trajectories\" or involves understanding \"dynamics or motions\" is a bit of an over-sell. Start and goal states hardly constitute trajectories or dynamics.\n\n\n\nMINOR:\n* Lines 24-25: Stating that the model must ground intent in the current observation is an over-simplification. More generally, it requires grounding language in the context of a representation of the robot's action and state spaces, the later of which includes more than the current image.\n* Line 27: While neural network models require large amounts of data, the same is not true of traditional structured approaches that use learning, albeit at the expense of significant feature engineering.\n* Section 2: It is surprising that the paper makes no mention of LLM-based approaches to language understanding for robots.\n* Section 2: The paper is missing earlier work in imitation learning for language understanding [1,2,3]\n\nGRAMMAR:\n* Line 70: \"apply pre-trained VLMS for robotic control\" &rarr; \"**applying** pre-trained VLMS **to** robotic control\"\n* Line 159: \"capability\" &rarr; \"capabilities\"\n* Line 183: \"we proposed to learn explicitly aligned\"?\n\n\n\n\nREFERENCES\n\n[1] H. Mei, M. Bansal, and M. Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2016.\n\n[2] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. D. Reid, S. Gould, and A. van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.\n\n[3] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell. Speaker-follower models for vision-and-language naviga- tion. In Advances in Neural Information Processing Systems (NeurIPS), Dec. 2018.\n\n[4] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and Where Pathways for Robotic Manipulation. In Proceedings of the Conference on Robot Learning (CoRL), 2021.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "2mxzUhKnCP", "reviewer": "Reviewer_HoyX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper points out that the cost of obtaining goals expressed visually is much lower than the cost of obtaining language goals, but that language goals are often easier for human operators to provide. Accordingly, the authors propose to use a small amount of language instructions paired with a large amount of goals for learning manipulation policies. The method projects language and goals into a shared space, where the representations of each modality are aligned using a small amount of labelled data. The goals are additionally conditioned on a start state, meaning the goal representation encodes a difference from the start state rather than just a goal. Ablations show both the alignment and start state are important to performance. The authors demonstrate their method on hardware. ", "summary_of_recommendation": "Based on the paper's strengths, I'm recommending a strong accept. The weaknesses I described are fundamental shortcomings of any approach to the tasks tackled here, and so I think they are outside of the scope of what could be tackled in a rebuttal period, and they would hold of any other paper in the area as well. The method proposed seems like it is a useful method for addressing the state of the world as it is now (image goals being more available than text goals, but less usable). This may change in the future, but for the time being, the method given in the paper fills an existing gap. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. CLIP has been shown in the past to struggle with compositional generalization (https://arxiv.org/pdf/2212.10537.pdf, https://arxiv.org/pdf/2211.13854.pdf, i.a.). Right now, the paper does not explicitly test whether the CLIP-based model can perform equally well on instructions requiring a compositional analysis of the input. For example, are the instructions \"Place the pot on the towel\" and \"place the towel on the pot\" both handled successfully? More generally, I'm curious as to whether the explicit image alignment (and the inclusion of the initial state) helps in these cases. It seems like it should. \n\n2. What is the actual form of $\\pi$? Is it an MLP? How many layers, etc. etc. \n\n3. How does the approach scale w.r.t. language data? The paper uses 7k demonstrations, but what is the curve here, e.g. at 1k, 3k, 5k, etc.. What is the minimum number of language annotations you can get away with? \n\n4. On L109 the unlabelled dataset is $D_U$ but later on L116 it's $D_B$. Is this a typo? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n1. **Positive results with thorough ablations**: The results of the method are strong even against relevant, similarly powerful baselines. The authors ablate all of the components of their model and demonstrate large performance changes, motivating their approach. \n\n2. **Clear presentation**: The paper is clearly-written and the problem is well-motivated. The description of the system is generally easy to follow and the hypotheses tested are expressed explicitly. \n\n3. **Makes use of existing data**: As the authors point out, collecting language data is expensive and difficult, while image-based goals can be easily collected from hindsight relabelling or similar methods. \n\n## Weakenesses\n1. **Limited language**: The language used seems to be from fairly basic templates. This somewhat invalidates the motivation of the paper, which is that collecting NL data is expensive -- collecting data from templates is free. The expensive bit is collecting data from real people. While CLIP might generalize to more complex language, the results here do not convincingly show that the method can handle linguistic variation.\n\n2. **Short-horizon/simple tasks**: As far as I can tell, the tasks considered are relatively short-horizon (single pick/place). Adding the initial state $s_0$ here might have an outsized impact, as compared to a longer-horizon task where there might be a greater difference between states. The authors do address this somewhat in their limitations. \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a  method that explores joint image- and goal- conditioned policies with language using only a small amount of language data. All the reviewers find the proposed problem approach of practical importance. The authors have provided careful answers to the concerns of the reviewers with additional experiments, in particular in terms of novelties with some state of the art baselines. The submission receives one accept, one strong accept and one weal reject but the reviewer does not argue his recommendation.
Decision: Accept (Poster)