Venue: CoRL 2022 Workshop LangRob

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: The authors present SPRINT, a method for pretraining instruction-conditioned agents with offline RL. Assuming access to human-annotated trajectories, the authors additionally propose two methods for augmenting the offline dataset to increase diversity: relabelling language instructions with a pretrained LLM and cross-trajectory instruction chaining.

Strengths: 
- While components of this method exists in prior work (LLM augmentation, chaining instructions), the overall method seems novel and the experimental results look promising. 
- The evaluation set also seems to be carefully constructed, and the release of that benchmark would be useful for furthering research in this area. The experiments and relevant ablations also seem meaningful and strong.

Weaknesses / Feedback: 
- Would be nice to see further comparisons against prior work to place the SPRINT method in context, e.g. potential relevant prior work on languaged-based skill learning and reuse in ALFRED with "Skill Induction and Planning with Latent Language" (Sharma et al., 2021). Another interesting baseline would be templated instruction augmentation (synonym replacement) instead of LLM augmentation, to see if the LLM is necessary. 
- It looks like overall on EVAL_100 SPRINT completes less than two subtasks (Figure 5, left) -- it would strengthen the paper to have further discussion on the specific failure modes and challenges of further chaining. 
review_title: Review 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **Summary:**

This work presents Scalable Pre-training via Relabeling Language Instructions (SPRINT), a method for pretraining language-conditioned robot control policies. The main insight is to expand the initial language instruction task set via two main methods: (1) use an LLM to aggregate multiple short-horizon hindsight language labels into a single long-horizon language instruction and (2) use reward labeling to chain cross-trajectory experience by performing hindsight experience relabeling that also selectively changes the goal instruction by concatenating the two original instructions. SPRINT is tested in the simulated ALFRED environment in a zero-shot evaluation scenario and a finetuning scenario. They find that SPRINT outperforms the LCBC and AM baselines, is able to zero-shot to a variety of long-horizon instructions, and finetune in a new environment effectively.

**Strengths:**
- The method is a smart use of LLMs to impart semantic knowledge to generate more instructions without requiring any visual grounding
- The empirical results are strong, and very relevant to the topic of the workshop

**Weaknesses:**
- It’s unclear how this method scales to non-toy task robotics environments with a more realistic action space, where motor control learning often blurs the lines between language instructions
- Very simple baselines would help add more clarity into what the biggest benefits of SPRINT are (see below)
- Technically the method does not do instruction relabeling as much as instruction aggregation / instruction expansion; in-place instruction relabels are out of the scope of this work, as the paper notes would require grounding. So, perhaps reducing the claim of the method in the Abstract and Introduction would be accurate.

**Constructive Feedback:**
- While instruction-aggregation produces 2.5x additional pre-training task instructions, it’s not clear how many additional pre-training instructions the naive concatenation from Cross-Trajectory Chaining results in. Is performance directly correlated with # of unique pre-training tasks? If so, are there other baselines you could use, such as ablating the initial starting human label dataset or using data augmentation (ie. Mad Libs style synonym substitution) or using naive concatenation (ie. as you did for Cross-Trajectory Chaining), which could also produce additional pre-training tasks with no additional human effort?
- Since the instruction aggregation portion seems to be the most important part of the method based on 4.1.3, it would be helpful to understand why this, since one might expect transfer across experience to be especially important in long horizon environments. However, it seems that even without Cross-Trajectory Chaining, SPRINT performs quite well; some analysis of this counter-intuitive result would be helpful.
- In 3.3, it is mentioned that it is difficult for an LLM to summarize randomly paired low-level instructions, so naive concatenation is used there. Did you try a baseline where you also did naive concatenation instead of LLM aggregation?

review_title: Interesting Usage of LLMs for Instruction Relabeling

Decision: Accept (Spotlight)