Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a data-augmentation strategy to learn from demonstrations in an embodied agent setting. Two types of augmentation strategies are proposed - relabeling language instructions with a language model and changing together different tasks/trajectories to obtain a new task/trajectory. The proposed method claims to be able to learn a diverse set of skills with these augmentation strategies. The method outperforms two baselines on an evaluation set constructed by the authors based on the ALFRED dataset. 
strength_and_weaknesses: Pros
* Paper is generally easy to follow
* Interesting use of language models for instruction re-labeling and summarization

Cons
* Limited technical novelty
* The motivations for the paper are not clear 
* Details about experimental setup are vague/missing and the results do not look particularly convincing

clarity,_quality,_novelty_and_reproducibility: One of the biggest concerns I had about this paper is the evaluation setting. The authors refer to their methods as a ‘pre-training method’ and claim to be able to learn new tasks more efficiently by fine-tuning on them. I found this perspective quite unconventional. Taking the ALFRED setting as an example, one trains a goal conditioned policy on training demonstrations and expects it to generalize to new tasks, which is a supervised learning problem. The authors need to motivate the setting better and make it more meaningful. 

Why do the authors need to resort to offline RL approaches for this problem setting? Have you considered learning from the augmented demonstrations with imitation learning? If the proposed method is more advantageous, was the imitation learning (on the augmented data) baseline considered? 

The design choices in eq(4) and eq(5) need better motivation. What is the motivation for assigning non-zero rewards to only the final state of each sub-trajectory? 

The evaluation section needs better clarity for readers to understand the exact train/test setup.
The construction of EVAL_100 is vaguely described as ‘sampling sequences of 1 to 7 instructions’. I did not understand what this means (are they tasks? subgoals?). 

Do the baselines benefit from your data augmentation strategies? Or are they only trained on the original set of demonstrations? I would assume the comparison is unfair in the latter case. 

I found it unsatisfactory that the proposed method was not compared against state of the art methods on the ALFRED task, which makes it hard to contextualize this work. How well does the proposed method perform compared to these methods?

Figure 1 can be improved to make the approach more intuitive.

summary_of_the_review: Raising score to 5 post-rebuttal.

The paper suffers from lack of strong motivations for the problem setting considered and several aspects of the experimental results are lacking.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The proposed method is a offline language-based goal-conditioned reinforcement learning. It needs human annotator for labeling basic tasks with languages. It implements large languages models for relabeling and cross-trajectory chaining, in order to increase the diversity of the task set. Experiments are conducted to show the proposed method exceeds previous approaches in terms of task completion rate, more efficient finetuning and 0-shot generalization.
strength_and_weaknesses: > Strength

1. Generally well writen, structure of the article is well organized. Illustrative figures are intuitive and do a good job to help reader understand the main ideas. Important details are included in the appendix. Every claims of the authors are well-supported by their experiments
2. Experiments are well conducted on ALFRED household task benchmark. The results support the claimed points of the proposed. Comparisons are provided to show the superior performance of proposed methods. Ablation study is conducted to show all parts of the proposed method are useful.
3. This work has good motivation. It pretrains generalization agent with interpretable skills. The proposed method leverages easy-to-collect natural language instructions to generate unseen tasks by taking advantage of language semantics. The learned agents have skills that are semantically meaningful. 

> Weaknesses

1. The idea of generating unseen language-based tasks is not novel. Previous work has already used the idea of generating new language-based tasks. The paper IMAGINE used Construction Grammar Heuristic for this job while this paper uses large language model. Therefore, the technical innovation is limited. 
clarity,_quality,_novelty_and_reproducibility: The writing and figures are very clear, and the paper is of high quality. The proposed method is of mediocre novelty but the experiments are complete and well-conducted to show the advantages of proposed method. 
summary_of_the_review: It is a well-written paper with high completeness. The method leverages language models for efficient pretraining.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes two data augmentation schemes for instruction following training via offline RL:

(1) **same-trajectory instruction aggregation**: Given a trajectory with multiple sub-trajectories, aggregate adjacent sub-trajectories into a longer trajectory and relabel its natural language annotation with a summary of the individual instructions. This summary is generated via a large language model (LLM) prompted with few-shot examples.

(2) **cross-trajectory chaining**: concatenate two random trajectories, and their instructions via “and”. 

The proposed approach, SPRINT (Scalable Pre-training via Relabeling Language INsTructions), is evaluated on ALFRED benchmark, where authors create a set of 100 unseen
long-horizon evaluation instructions (EVAL 100), and a set of 20 evaluation commands that test the agent’s chaining capabilities (EVAL CHAIN). On these sets, SPRINT is shown to outperform imitation learning (L-BC) and offline RL (AM) baselines in zero-shot and finetuning setups.
strength_and_weaknesses: **Strength**: The proposed approach is simple and effective on authors’ test sets of ALFRED, with potential to apply to more instruction following tasks with long-horizon challenges.

**Weakness**: My main concern is that experiments are only done in one benchmark with test sets specifically created by authors that would intuitively benefit from such compositional data augmentations. Is it also possible to also provide results on standard ALFRED evaluation/test sets, which should be more than 100 selected task instances? 


Some other suggestions/questions:
- Maybe mention the domain (ALFRED) and learning method (offline RL, instead of more vague “policy pre-training”) in abstract? 
- Can benchmark setups be better explained? The first paragraph of Experiment sections talks about unseen task/language/environment, but then EVAL100 is said to be taken from the training set (why not eval set?), and EVALCHAIN unclear. A better explanation of task splits in original ALFRED and author created test sets is needed.
- Learning setups: Is comparison to baselines fair - do they have similar architecture (size), training steps, number of training language tokens? Are two data augmentation subschemes mixed 1:1 or other proportions? I tried to check some appendix but they are not clear, and maybe including them in main paper would be nice.
- The cross-trajectory scheme seems only applicable to offline RL, while the same-trajectory one also applies to imitation learning. Would be interesting how the latter alone might benefit IL, if authors want to claim usefulness beyond offline RL but for general policy pre-training.
clarity,_quality,_novelty_and_reproducibility: The method part is clear and easy to follow, but I have some clarification questions about experiment setups (see above). 

The method seems fairly novel and interesting to me, though I cannot confidently judge the novelty due to my lack of knowledge in related work.
summary_of_the_review: The approach is simple, interesting, and potentially useful for more instruction following tasks, but more evaluations and explanations of setups would help better justify it.

==

After rebuttal, authors addressed many clarification questions and I raise my score from 5 to 6 in light of these.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a scalable offline policy pre-training approach based on natural language instructions.

It enables automatic augmentation of pre-training data with large language models to relabel and chain across trajectories.

It empirically shows that the proposed approach outperforms prior pre-training approaches.
strength_and_weaknesses: ***Strength***

- It can populate pre-training data without additional human labeling. It only requires labeling sub-trajectories (maybe with crowd-sourcing), and the approach will combine the trajectories for new data.

- The evaluation results show its advantage over prior methods.

***Weakness***

**The weakness is mainly in cross-trajectory chaining.**

**1. The primary concern is the change of MDP**

The cross-trajectories chaining changes the original Markov Decision Process in the transition probability distribution P. More precisely, the value of chaining transition s_TA x a_TA x s_0B is changed.
For example, if sub-trajectory A ends in a conference room, and sub-trajectory B starts in a kitchen, then the last action of A warps the agent in the new MDP.
It encourages the agent to try warp when it wants to go to another room (after finishing a sub-trajectory).
It seems the reward of Q(s, a, z_B) is not designed to correct the problem of MDP change.

It might need explanations if the change of MDP is supposed not to have significant influences in general.

**2. The online change of Q**

As mentioned in the paper, the reward Q(s, a, z_B) changes during training and is updated online.
It requires the RL algorithm to accept online changes in rewards.
So, the paper may need more arguments on what RL algorithms are compatible (it says "any offline RL algorithm" in Section 4.1).

**3. The novelty**

- The instruction aggregation and using LLM for it might be novel. However, existing LLMs are used.

- The use of consecutive sub-trajectories seems straightforward.

- Cross-trajectory chaining has novelty, though many of its techniques are inspired by goal-conditioned RL approaches (e.g., offline goal chaining [Chebotar et al., 2021]), as mentioned in the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity:

This paper is clear to read.

Quality:

This paper may have some significant concerns.

Novelty:

This paper contains novelty.
Please see the last section.

Reproductivity:

The paper does not mention (anonymous) source codes.
summary_of_the_review: This paper proposes a scalable offline policy pre-training approach that outperforms conventional ones.
It combines sub-trajectories to augment data without extra human annotation.

However, it still has significant concerns, as mentioned earlier.
More explanations might help.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The proposed data-augmentation strategy with LMs to learn from demonstrations in an offline RL setup is interesting has been explore in some recent studies. The reviewers expressed concerns about limited novelty, comparison with right baselines like imitation learning methods, unclear motivation of the method. Some of these concerns remain even after the discussion.  



justification_for_why_not_higher_score: Not good enough. 
justification_for_why_not_lower_score: NA
Decision: Reject