Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper targets to solve real-world robotic manipulation tasks by leveraging a large demonstration dataset and a small amount of target task demonstrations.
The proposed method first pre-train a policy on the large dataset with multi-task offline RL, then fine-tune on the small demonstration data again with same objective.
During the fine-tuning phase, a small amount of data is sampled from the dataset used for pre-training to prevent overfitting.
The fine-tuning process is early stopped by heuristic validation rule.
In experimental results, several challenging robotic manipulation tasks are solved by the proposed method using proper design choices of network architecture and network size.
strength_and_weaknesses: ### Strength
* Provided results on robotic manipulation tasks can provide empirical knowledge on handling robot control tasks using offline dataset.

### Weakness
* Novelty
    - The problem setup of this paper is very relevant to fully offline meta-RL and the paper is stating "our approach is much more data-efficient and simple". However, any comparison or justification is not given in the paper, thereby misleading readers to be impressed as the problem setup is novel.
    - Given architectural choices are interesting and reasonable, but neither surprising nor significant. Moreover, in order to introduce the network design as a technical novel of this paper, the architecture should be applied and evaluated on more diverse domains.
    - The early stopping criteria is bizarre. The Q-value plot of passed example in Figure 3 is not monotonically increasing over the time. If there is an exact rule deciding whether it is "almost" monotonically increasing it is not described in the main text. If a human decide this, any of given results is reliable as there is no human-related study for this procedure.

* Comparison
    - As mentioned before, the comparison with offline meta-RL is missing. Also, any naive application of imitation learning method other than behavior cloning also deserves to be a baseline.

clarity,_quality,_novelty_and_reproducibility: ### Clarity, Quality
The paper is clearly written and easy to follow.

### Novelty
Many aspect of papers can be found in prior works, but it is not enough addressed.
summary_of_the_review: Although this paper provides interesting results on challenging manipulation tasks leveraging offline data,
the relationship between the proposed method and existing methods is not enough addressed.
Moreover, the proposed design choices are incremental but evaluated on a single domain.
Because the generality of the proposed method is not proven, it is difficult to be introduced to the community as a generic learning method for such a problem setup.
Thus, I vote to reject this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of letting robots learn to solve novel target tasks by leveraging diverse offline datasets in combination with only small amounts of task-specific data. To this end, the paper proposes a framework that uses multi-task offline reinforcement learning approaches for pre-training and then fine-tuning its policy on target tasks. The experimental results show that the proposed framework outperforms behavior cloning and offline reinforcement learning baselines including CQL and COG. I am leaning toward accepting this paper since it studies a promising research direction and presents a reasonable framework to address the problem with supporting experimental results.
strength_and_weaknesses: ## Paper strengths and contributions

**Novelty**
In my opinion, the idea of bridging offline reinforcement learning and behavior cloning to solve novel target tasks in a new domain with only a few demonstrations is intuitive and convincing. This paper presents an effective way to implement this idea.

**Clarity**
The overall writing is clear. The authors utilize figures well to illustrate the ideas.

**Related work**
The authors give a clear description of the related prior works from both the perspectives of offline reinforcement learning and behavior cloning.

**Experimental Results**
The experimental results show that the proposed framework outperforms behavior cloning and offline reinforcement learning baselines including CQL and COG.

## Paper weaknesses and questions

**Experimental results**
- It is mentioned in scenario two that COG does not outperform BC (joint), which is pretty interesting. Unfortunately, COG was only compared to BC but not PTR with this setup. It would be more informative to give a brief explanation or an intuition of why COG was at a disadvantage in that particular scenario.
- It is not entirely intuitive to me why the proposed framework outperforms BC baselines. It would be helpful if the authors give more intuitions.
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: I am leaning toward accepting this paper since it studies a promising research direction and presents a reasonable framework to address the problem with supporting experimental results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper experimented with using offline RL to pretrain and fine-tune policy and Q function. The results show that using offline RL to first pretrain the policy and Q function on a diverse offline dataset can improve the policy adaptation speed on new tasks. The paper discussed about several important design choices, such as network architectures to achieve better performance. Overall, the paper is more like an engineered work that combines some ideas from prior works and achieves fast policy learning on new tasks.
strength_and_weaknesses: **Strength**:

* The paper presents a few techniques, such as network architecture design and normalization trick, that are empirically shown to improve downstream task learning.

* The paper shows real-world robot experiments. This is a big plus as such experiments do require a significant amount of effort.

 * The paper compares the proposed pipeline with a decent number of baselines, including behavior cloning and especially prior works on first doing representation learning and then doing policy learning such as R3M.

**Weaknesses**:

* As the paper also says itself, the techniques presented in the paper are not new.

* Since the number of real-world tests is small (like 20?), I am not sure how one can report success rates with so many significant figures. Similarly, the paper tends to overclaim the method's effectiveness. For example, in page 5, the paper says that using "learned spatial embeddings" leads to `2x` improvement. However, if we actually look at table 10, we see that one case has 4/10 success rate and the other one has 7/10 success rate. With such a small number of tests, it is very confusing and not rigorous at all to make such claims (`2x` improvements). There are other similar claims in the paper. The authors should remove such claims.

* In table 2, I cannot find the 95% confidence interval.

* It's unclear how convincing the claim about generalizing to previously unseen domain is. From the video, I can only tell that the testing door is not that different from the training doors. Can authors add a figure showing all the training doors and the testing door? Since the claim is about generalization, it is important for readers to understand how much difference the training and testing distribution has. As a research paper, it is important to make each claim precise and accurate. While I enjoy reading the papers, especially seeing the real-world testings, reading such exaggerated claims only negatively affects my judgment.
clarity,_quality,_novelty_and_reproducibility: The paper writing is clear, except it tends to exaggerate the claims or does not define the scope of the claims properly.
summary_of_the_review: Overall I think the paper presents a decent amount of experiments and compare its proposed pipeline with some reasonable baselines. However, the paper writing needs to be improved to make the descriptions accurate.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework pre-training for robots (PTR) based on CQL that attempts to effectively learn new tasks by combining pre-training on existing robotic datasets with rapid fine-tuning on a new task, with as few as 10 demonstrations.
In particular, their method modifies CQL to include several crucial design choices that enable PTR to have strong generalization. These choices include using task indicators for policy and Q-function networks, using group normalization, and using mixed data of prior data and the target dataset.
In their experiments, they train the model on the Bridge dataset and finetune it with real-world demonstrations. Their method shows signiﬁcant improvement over prior RL-based pretraining and ﬁnetuning methods.

strength_and_weaknesses: ### **Strength**

* Generalization of offline reinforcement learning algorithms is an interesting topic. Although there is no new algorithm or framework in this paper, the exploration of useful combinations of existing technics still contributes to this area.

* The presentation of this paper is good. All details are clearly explained. 


### **Weaknesses**

* Missing baselines of offline meta-reinforcement learning [1][2]. Without comparing with existing methods, it is hard to position the empirical value of this paper.

* Sort of limited novelty. As the authors mentioned, “Although the individual components that constitute PTR are not especially innovative and are based closely on prior work, the combination of these components is novel.”Since there are already some existing novel meta-RL methods, properly comparing them could empirically improve the novelty of this paper.

---

[1] Mitchell, Eric, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. "Offline meta-reinforcement learning with advantage weighting." In International Conference on Machine Learning, pp. 7780-7791. PMLR, 2021.

[2] Xu, Mengdi, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. "Prompting decision transformer for few-shot policy generalization." In International Conference on Machine Learning, pp. 24631-24645. PMLR, 2022.


clarity,_quality,_novelty_and_reproducibility: ### **Quality**
The quality of this paper is high for proposing an empirical framework for offline RL.

### **Clarity**
The presentation is very clear.

### **Novelty**
The novelty could be improved by comparing it with existing meta-RL methods.

### **Reproducibility**
Code is not released but the technics used in the method are either well-known or easy to implement. So I think it is easy to reproduce the results. 


summary_of_the_review: I acknowledge the empirical contribution of this paper to the generalization problem of offline RL. However, without a fair comparison with state-of-the-art meta-RL algorithms, I suggest rejecting this paper for now.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of training real-world robotic manipulation by combining offline datasets and small task-specific datasets. The main contribution of this paper is the experimental work in real world settings, which was valued by all the reviewers. Unfortunately, there is a general concern about the limited technical novelty, and this is the main reason to reject the paper. 

As said before, the reviewers strongly value the empirical contribution of this paper and the amount of experiments performed. The reviewers also pointed out the lack of acknowledgement and proper contextualization of the work with respect to previous works. These contextualization aspects were addressed by the authors in their feedback and they also made changes in the paper accordingly. Additionally, some comments related to the need for clarifications on the results  were also addressed by the authors. However, after discussing this paper with the reviewers in a virtual meeting, and also with the Senior AC, we think the strong experimental contribution is not enough for ICLR, since the conference expects contributions with technical novelty. We strongly encourage the authors to submit the work to a robotics conference.

justification_for_why_not_higher_score: The main reason for rejecting the paper is the limited technical contribution.
Decision: Reject