Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes a method for robustifying a policy obtained via behavior cloning by collecting additional experiences in the environment and minimizing an optimal-transport-based (OT-based) trajectory matching reward between expert demonstrations and trajectories generated by the behavior policy (Algorithm 1 in Appendix C). Additionally, the paper proposes a particular adaptive scheme (Eq. (4)) for tuning the weight between the objectives for behavior cloning and for maximizing the OT-based returrn. Experiments on vision-based control tasks from DeepMind Control suite, OpenAI Robotics suite, and Meta-World suite, as well as real-world experriments on an xArm robot arm demonstrate that the proposed method ROT compares favorably to Discriminator Actor Critic (DAC) [7], even when DAC is augmented with the proposed prertreaining and regularization scheme, and furthermore ROT outperforms a state of the art OT-based Sinkhorn Imitation Learning (SIL) algorithm [12].
strengths_and_weaknesses: Strengths
- The paper provides a strong empirical evaluation and convincing results for the proposed ROT method
- The presentation is clear, the questions addressed by the experiments are explicit and easy to follow
- The method combines several ideas from prior works (as well explained in Sec. 5) but is sufficiently novel and may become a new baseline for BC+finetuning algorithms, plus further extensions are possible

Weaknesses
- A few important details about the algorithm can only be found in the appendix (e.g., what networks exactly are trained, feature preprocessing for OT, freezing of the feature processor, etc.). However this is OK due to the space limitations.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a novel imitation learning algorithm that adaptively combines offline behavior cloning with online trajectory-matching based rewards. The method is thoroughly evaluated and shown to outperform the baselines, including in experiments on real hardware. There are no concerns with regards to quality, significance, clarity, etc.
issues: None
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper attempts to accelerate the imitation with a few demonstrations. It adaptively combines offline behavior cloning and online trajectory-matching based rewards. Specifically, the paper contributes a method of adaptive regularization with soft Q-filtering, which regularizes the IRL policy to stay close to the BC pretrained policy. The paper shows in various tasks that the proposed method can accelerate the imitation while at the same time reach high success rate. 
strengths_and_weaknesses: Strengths:

•	The paper is well motivated in such that it proposes improving the efficiency of imitation learning both by decreasing the training time and cost of demonstrations. 

•	The experimental evaluation is extensive. The authors experimented on 20 simulation tasks and 14 real robot tasks and provides convincing results that the proposed method generally outperforms baselines in these tasks. 

•	The figures in the paper are well designed and composed. The ablation experiments are detailed and clear.

Weaknesses:

•	As the paper admits, on tasks such as Peg in box (hard) and Pressing a switch, ROT’s performance drops substantially. Also in some other tasks, the performance is not stable.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes to accelerate the imitation with a few demonstrations. The paper is well-written and overall technical sound. The literature review is thorough and experiment results are extensive and convincing.
issues: 1.	As mentioned in “Weaknesses”, the paper mentions that this performance drop might be due to the lack of visual features, but for fetch_push, fetch_reach, fetch_pick_and_place, the performance is also unstable and worse than other baselines. Since tasks like reach and push do not need much visual features, what could be reason for that?

2.	As mentioned in the paper, the expert policy is trained using DrQ-v2 for DeepMind Control tasks and OpenAI Robotics tasks. While in section 4.5, the proposed ROT is compared and outperforms DrQ-v2. Could you clarify what’s the difference between the expert policy DrQ-v2 and the one is compared?

3.	In figure4, the metaworld_hammer plot is missing the line for expert.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Summary: This paper proposes Regularized Optimal Transport, a new imitation learning algorithm that builds on top of optimal transport-based trajectory matching. The algorithm can learn from a few expert demonstrations and an online reinforcement learning phase. The algorithm is benchmarked in simulation and on real robots and achieved a high success rate.

strengths_and_weaknesses: ### Strengths

1. This is a great paper with solid work put in. It proposes an effective imitation learning algorithm that has strong performance. The proposed method works well on both simulated benchmarks and on real robots. On simulated benchmarks, it is several times faster in sample efficiency. And on real robotics tasks, the proposed method can achieve 90% success rate with 1 demonstration and 1 hour of online learning.
2. The authors provide a detailed analysis of the design choices and contribution of each component, providing insight of why soft Q-filtering and OT-based rewards are needed to stabilize the training.
3. Despite not in the main text, Figure 6 is very helpful to understand the intuition of the proposed method and why it is promising. 

### Weaknesses

1. The authors showed that the algorithm is somewhat robust to the out-of-distribution initial state. I am wondering how robust the proposed method is to other types of out-of-distribution states, such as adding a task-irrelevant distractor object in the scene during the second stage of training.
2. What action space does the real robot experiment use and does the proposed method work with other types of action spaces, e.g. impedance control, for contact-rich tasks, such as wiping a board?
3. What is the extent of initial condition deviation the proposed method can handle? When the deviation is too big, would there be a scheme where not initializing the policy with $\pi_{BC}$ be more beneficial?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Given the strong results and the thoroughness of the experiments, answers to scientific questions proposed in the experiment section, and the clarity in writing, I recommend a strong acceptance for this paper. I believe this is a great paper to have for the community.

issues: No major issues but hope to get clarification of the questions mentioned in the review.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper addresses jointly the issues of (1) sample inefficiency of inverse reinforcement learning (IRL) and (2) brittleness of behavior cloning (BC). To do so, this paper introduces a new imitation learning algorithm, "Regularized Optimal Transport" (ROT). The particular novelty of ROT as I understand it is to initialize a policy via BC, then refine that policy using IRL, with rewards computed using optimal transport (OT). A key insight seems to be to take care during the fine tuning step to (a) ensure that the fine tuning step doesn't take the refined policy "too far" away from the BC policy and (b) to control the degree of regularization with a simple heuristic based on the relative performance of the BC policy compared to the fine-tuned policy at a given iteration. The approach is shown to give good performance across a variety of tasks both in benchmark simulations and on a real robot.
strengths_and_weaknesses: # Strengths

- I found the paper to be very clear and well-written. I am not an expert in IRL, but despite that I had no trouble following the technical approach. I believe this paper would be very accessible to a reader.

- The overall idea is simple and intuitive; or at least, from the presentation it seems to follow quite naturally.

- The approach seems technically sound.

- The benchmark simulation + real robot experiments were well-presented and thorough overall, but see Weaknesses for a minor comment on this.

- The approach was demonstrated on a real robot.

# Weaknesses

- The present experimental evaluation compares the soft Q-filtering approach to hand-tuned regularization. However, given that [24] uses essentially the same idea, but with a hard assignment, it seems like the relevant experimental comparison would be to the hard assignment form of the regularization in eq (4) of this paper. It is mentioned briefly that on some experiments the soft assignment scheme improved speed of policy learning, though no such demonstration is presented (unless I missed something).

- The novelty with respect to prior work might be clarified a bit. In particular, as I understand it, the novelty is in both the use of the optimal transport reward function and the soft Q-filter regularization. In the context of prior work using similar regularization approaches (i.e. the hand-tuned approaches or hard assignment) then, I think this method differs in both the formulation of the reward *and* in the regularization. Is that correct? I wasn't 100% certain from the paper. I'll try to provide a little more detail in Issues.

## Update after rebuttal

All of my comments have been addressed and I have upgraded my rating from "weak accept" to "strong accept."
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I felt this was a very well-written and executed paper. The technical ideas and approach were clear and well-motivated. The main two technical contributions seem to be in the use of optimal transport-based rewards and the "soft Q-filter" regularization when fine-tuning models trained using behavior cloning. The experimental evaluation is thorough and convincing. My main suggestions to improve the work are minor: i.e. to add a comparison to the hard-assignment regularization approach (for completeness), and to perhaps clarify the contribution with respect to existing work.

## Update after rebuttal

I have upgraded my recommendation to "strong accept" as my comments have been addressed.
issues: Recapping from "Weaknesses" above, and giving a little more detail:

- The present experimental evaluation compares the soft Q-filtering approach to hand-tuned regularization. However, given that [24] uses essentially the same idea, but with a hard assignment, it seems like the relevant experimental comparison would be to the hard assignment form of the regularization in eq (4) of this paper. It is mentioned briefly that on some experiments the soft assignment scheme improved speed of policy learning, though no such demonstration is presented (unless I missed something).

An interesting addition to this paper would be an explicit comparison with the hard assignment approach. Given that (as I understand it) this paper is proposing both OT based rewards and an additional soft regularization, I'm not sure this is strictly "necessary" but such a comparison would be interesting and perhaps support the claimed improvements in speed versus the hard assignment approach.

- The novelty with respect to prior work might be clarified a bit. In particular, as I understand it, the novelty is in both the use of the optimal transport reward function and the soft Q-filter regularization. In the context of prior work using similar regularization approaches (i.e. the hand-tuned approaches or hard assignment) then, I think this method differs in both the formulation of the reward *and* in the regularization. Is that correct? I wasn't 100% certain from the paper. I'll try to provide a little more detail in Issues.

Here I can try to be more precise. In Section 3.2, the OT rewards are mentioned, but they don't appear in the actual optimization objective for the policy in eq (3). As I understand it, the Q function is building up an approximation of the value function with rewards computed via optimal transport. Is that correct? If so, it may be worth mentioning this under eq (3). If not, this should definitely be clarified.
review_title: null

Meta-review: metareview: This paper proposes a new imitation learning algorithm, Regularized Optimal Transport, which adaptively combines trajectory-matching rewards and behavior cloning. The paper demonstrates much faster learning with fewer demonstrations in both simulation and real-world robot learning tasks. 

All reviewers agree that the paper is well written, the approach is sound, the evaluations are thorough and the results are strong. There were some questions about the performance drop on several experiments, novelty, and comparison with the hard assignment approach. After the rebuttal and discussions, all of these concerns were addressed. Thus, we would like to recommend accepting this paper. 
confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)