Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper considers the problem of imitation learning from out-of-dynamics demonstrations. Since the demonstrators may have different dynamics than the imitator, these demonstrations may not be useful to the learner, if not detrimental. The paper follows existing work and uses a 2-stage procedure. It first learns a feasibility/transferability score for the demonstrations, and then uses an imitation learning algorithm on weighted data. The paper addresses a major limitation of f-MDP, the state-of-the-art approach, which is not able to deal with multimodal demonstrations. The proposed approach first clusters the demonstrations, then learns a policy for each cluster. Additionally, it uses a transferability metric that is easier to compute than f-MDP. The proposed approach is evaluated on various simulated tasks with different physics backends such as mujoco and pybullet.
strengths_and_weaknesses: ### Strengths
- **Motivation.** The research problem is well motivated. The challenges of the research problem, as well as the limitations of existing methods are presented clearly.
- **Organization.** The paper is in general well organized. The technical approach is easy to follow, and seems technically sound. 
- **Literature survey.** The authors did a thorough review of related literature. The authors clearly present the limitations of existing work, and how the proposed work is able to address these limitations.
- **Ablation studies.** The authors present a series of ablation studies verifying the importance of each component, as well as demonstrating how each hyper-parameter is chosen. 

### Weaknesses
- **Lack of real-robot results.** Although the proposed method is evaluated on different physics engines such as mujoco and bullet, it lacks experiments on real hardware. It seems like there are still considerable gaps between the simulation and real hardware experiments. First of all, the physics of real hardware could be very different from simulation. For example, in the videos on the website, the Franka robot and the box are often “sinking” into the table. Second, it is possible that demonstrations generated by RL policies in simulation differ from demonstrations that can be collected on the real hardware. Therefore, it would be nice if the authors can present results on real hardware, show that real-world demonstrations also have such multi-modal properties, and demonstrate that the proposed method outperforms baselines for such data. 
- **Comparison with other clustering algorithms.** It would be nice if the authors can compare their results with simpler clustering algorithms, such as K-means. Although in the appendix, the authors claimed that L2 distance can’t be used on trajectories of different lengths, it seems like trajectory segments with fixed length are actually used according to the appendix.
- **Loss function.** In Equation (8), it seems like $p_w$ is also a function of the discriminator $D$. How did the authors compute the gradient of $D$ in this case? If the authors did not consider the gradient of $D$ from $p_w$, the authors may want to note that in the paper.
- **Final policy.** It seems like $K$ policies are trained by the proposed algorithm. Is the reported results from the mixture of $\pi_k$ or the best one?
- **Source of multimodal demonstrations.** It seems like that in a lot of the experiment setup, the demonstrations are multimodal because they come from different dynamics. It would be nice if the authors can compare their results with a baseline where a policy is learned for each demonstrator (with transferability but without clustering). This can significantly strengthen the paper by verifying the importance of the clustering step. 
- **Data distribution for mujoco environments.** It seems that the composition of demonstrations is quite different among the mujoco environments. It is appreciated that the authors show results with different data composition for Hopper, but it would be nice if the authors show similar results for the other two environments. 
- **HalfCheetah.** It seems like for HalfCheetah, all demonstrations are feasible, since $\gamma_f$ and $\gamma_b$ from each demonstrator is smaller than the imitator. Is that true?
- **Ablation study with trans.** It is appreciated that the author presented the ablation study. It would be nice if the authors also compare their results with the one that uses clustering but not the transferability step. It can help validate the importance of the transferability step.
- **Common $\rho_0$.** It seems like the authors assume that all of the MDPs share the same initial state distribution, why is such assumption necessary? By the way, It seems like $\rho_0$ is actually defined in the MDP paragraph.
- **Feature extractor F.** Feature extractor $F$ first appears in Equation (1), but is not described until line 149. The authors are also advised to provide more details on the structure of $F$ that was used in the experiments.
- **Typo?** In Equation (5), should it be $(s_t^d, s_{t+1}^d)\sim\Xi_k$ under the first expectation? 
- **Transferable demonstrations.** In appendix C.3, the authors mentioned “equal number of transferable and non-transferable demonstrations”. How do the authors know which demonstrations are transferable, which are not?
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper addresses an interesting research problem that is well-motivated. The authors were able to clearly explain the contribution of the proposed work. Although the proposed method seems technically sound, the contribution seems a little incremental. The main weakness of the paper comes from the experiment section. I am concerned about the paper because no real robot experiments are presented, and some additional results are needed to support the claims. 

With that said, I am happy to increase my score if the authors present more experimental results during the rebuttal period. 
issues: See **Strengths and Weaknesses**.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a new Imitation Learning method designed to address action-space mismatch between demonstrator and learner. The action-space mismatch assumption manifests in the assumption that no expert actions are available, and also implies that the underlying dynamics of the demonstrations $p(s' | s, a_{\text{expert}})$ does not match the dynamics of the learner $p(s'|s, a_{\text{learner}})$. This is termed 'OOD-IL' (Out-of-dynamics IL). The proposed method uses contrastive learning and clustering to learn a set of discriminators and policies in a GAIL framework, which is designed to address potential multimodality in the training data. The resulting set of discriminators are used to form a joint distribution over state transitions that is intended to capture whether a particular state transition is feasible. This joint distribution is then used as the sampling distribution for performing a final stage of imitation learning in an 'OOD' (out-of-dynamics) environment.
strengths_and_weaknesses: ### Strengths
- The OOD-IL regmie is certainly of interest, as successful methods in this of relaxation of typical imitation learning would enable a much broader set of training data to potentially be used for imitation learning. 
- The proposed method appears to work better than prior work in this regime. 
- Ablations confirm the efficacy of the clustering component and the transfer component on one environment
- A sweep of the clustering hyperparameter (in the appendix) further confirms the efficacy of the clustering component

### Weaknesses
- W1 (main weakness): The main experimental flaw is that there seems to no evidence presented in support of the claim that the multimodality of the demonstrations is the main factor behind the performance difference between the proposed approach and other approaches. This is an important weakness because it is part of the paper's central claims. E.g. L48-50 "The multimodal demonstrations make it impossible to learn a deterministic policy from the f-MDP. Furthermore, f-MDP suffers from slow and inaccurate optimization difficulties due to a step-by-step optimization procedure"; L124-125 "This multimodal distribution of the demonstrations makes prior works ineffective. In this paper, we simultaneously remove the interference from the multimodal distribution and learn transferability to mitigate negative impact of non-transferable demonstrations." I expand on W1 below.

- W2: Discussion of prior methods that also perform clustering by contrastive learning are missing. I suppose the idea is natural, but the current paper would be strengthened by drawing connections to this line of prior work and discussing their significance to the current work. These papers look related [A, B, C].

- W3: I doubt the following claim. L181 claims that "we can embed our transferability into any imitation learning algorithm." More explanation is needed for why this is the case. For instance, how could one use this sampling distribution to perform BC? The expert actions at samples from $p_w$ are not necessarily available. Therefore, BC is a counterexample to refute this claim on L181. Either the claim needs to be removed or refined (inclusion of a refined explanation and/or evidence). The regime studied in the paper is particuarly one in which the expert actions for the robot are not available, so it seems to rule out any imitation learning algorithm that relies on these (i.e. we can only use state-based imitation learning algorithms). If that's the conclusion, it would be good to include in the appendix (and reference in the main text) an example objective function of a different imitation learning algorithm making use of $p_w$.

- W4: The limitations section needs work. It doesn't actually discuss limitations of the proposed method in the experimental settings investigated, which suggests that it 'solves' them.

**More details on W1**:
Because there are other differences between the proposed approach and prior works, a direct comparison between them does not necessarily identify the multimodality of the demonstrations as the source of the performance difference. Therfore, Figs 4, 5b, and 6b, -- the main analyis -- do not constitute conclusive evidence in support of the claim. Fig 7 contains an ablation of the clustering component, and the clustering component allows for more policies to be learned to fit the original (pre-transfer) data. However, even with the clustering component removed, because the policies are stochastic, they still have the capacity to fit multimodal trajectory distributions. Therefore, to me it seems there is no evidence presented in support of the main claim.

Here are my suggestions for experiments to run to collect this evidence: 
- (1) Re-run the ablations in Fig 7 and also include ablations of the proposed method for the other environments, except with deterministic policies. This will force each policy to be unimodal, and therefore *only* the clustering (inclusion of multiple policies) would allow for fitting of multimodal data. Comparing the results of the four variants: {with clustering, without clustering} X {with deterministic policy, with stochastic policy} is the key idea here.
- (2) Confirm that the performance benefits of fitting multi-modal data go away when the data is made unimodal. This could be done by modifying the Driving environment so that only unimodal demonstrations and task. Then, there should be little-to-no performance difference between the ablations.

#### Minor weaknesses
- MW1: S4.4 and Fig7 are unclear -- what is the environment used for these experiments? I'm guessing it's the driving environment (based on the similarities of the y-axes in Fig5b and Fig7), but this needs to be made unambiguous.
- MW2: L202 "Hooper" -> Hopper

### References
- [A] "Deep Robust Clustering by Contrastive Learning" https://arxiv.org/pdf/2008.03030.pdf
- [B] "Contrastive Clustering" https://ojs.aaai.org/index.php/AAAI/article/download/17037/16844
- [C] "Supporting Clustering with Contrastive Learning" https://arxiv.org/pdf/2103.12953.pdf

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The regime is interesting, and the method is interesting, yet the central claim is lacking evidence (see W1). While I think the work has potential for impact, this lacking evidence makes the paper technically lacking, and therefore I do not recommend acceptance of the paper in its current state.
issues: See the weaknesses section, in particular, W1. Based on my current understanding, following the protocol suggested in the experiments for W1 would be sufficient to address W1, if it indeed turns out to confirm that multimodality is the important key factor. However, it could turn out to be the case that these experiments do not show the multimodality to be important, in which case, I think the results would necessitate further explanation and the paper significantly reworked around a different hypothesis that is consistent with the experiments.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper focuses on recognizing multi-modalities in the learning from demonstrations, where the demonstrations come from different demonstrators, system dynamics, environments, etc. Building a demonstration clustering and transferability measuring procedures on top of the imitation learning allows the model to evaluate the similarity of the current and the demonstrated states. 
strengths_and_weaknesses: Pros: 

- The motivation of the paper is well explained, i.e. help distinguish demonstrations from different dynamics and therefore allows the imitator to imitate the demos sharing similar states transitions without interference. 
- The three parts of the model, i.e. clustering demos, measuring transfer ability, and training the imitation learning model with transferability are well presented.
- The paper has a proper ablation study and further Hyper-parameter tuning details in the appendix.


Cons:
- It is a pity that no real-robot experiment is conducted, which decreases my confidence of mine that this work can be easily deployed in a real system. Maybe the authors can share some knowledge or experience about the limitations of deploying it in a real system. 
- The experiment details of the third task is a little bit unclear to me, i.e., what is this "moving one box from one side to the other"? The task description and Fig.6 do not give me enough information about the experiment, and neither does the supplementary material. 


robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: In general, I think this paper reaches the minimum threshold of CORL, as it is well-written, especially in the model structure part. The reasons that I do not feel very confident to recommend this paper are (1) I do not have strong background knowledge of the methods used in this paper or related works and (2) there is no real robot experiment, which is not common in imitation learning. I am looking forward to getting some feedback from the authors regarding the reasons for not deploying current work in a real system and the scalability of the number of demonstrators.  
issues: - A question in my mind, what is the reason for not using a mixture model, e.g. GMM, to model this multi-modality demonstrations behavior?
- I like Figure 2 in the appendix, but I would like to know how the authors compute the transferability of the other 3 methods.
- As I said in the weakness section, what is the bottleneck of getting a real robot experiment? From my understanding, a task like "driving" should be able to be achieved by a robot in task space. In general, it should be easier for imitation learning to work with a real system than RL, especially because the authors presented a lot of real system settings in the introduction section. 
- Where did the demonstrations come from? From RL agent or some human input or hard-coded trajectories?
- The number of clusters, K, seems crucial in the current model. So, what is the relationship between K and the number of demonstrators? Imagine a practical usage case, like autonomous driving, with different cars, weather, and road conditions, even rules (driving on the left in the UK vs. on the right in the US), it seems that K will be very big. So what is the scalability of the current model?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposed a novel imitation learning method that utilizes multimodal demonstrations from different dynamics and action spaces.
The main contribution of the proposed method is to develop an algorithm that relaxes the assumptions in the general imitation learning and explicitly takes into account the multimodal nature of demonstrations, which has been overlooked in previous studies.
strengths_and_weaknesses: Strengths:
- Under a more general problem setting, this paper succeeded in imitating from the multimodal demonstrations without simlifying the problem.
- The proposed method is able to absorb differences not only in dynamics but also in the action space.
- The performance of the proposed method was properly verified by benchmarks, including ablation studies.

Weaknesses:
- When dealing with multimodality, it is inconvenient to design the number of clusters K to be specified in advance.
In particular, since the proposed method requires training pi_k for each cluster, it is not desirable to guarantee generality by making K too large without consideration, but too small K cannot capture the multimodality.
- I think the two-stage GAIL should eventually make the learning process highly unstable.
- The final pi to be acquired is obtained by GAIL using a subset of the multimodal dataset that is close to the imitator's dynamics, which should inherit multimodality.
In that case, depending on the probability model of pi, it may fail to capture the multimodality and underfitting may occur.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The problem setting that this paper challenges is very important and versatile from the viewpoint of practicality, and the proposed method steadily resolved the issues that arise in this, although its approach seems to be heuristically constructed.
The performance of the proposed method was properly verified, and the effectiveness of the method seems to be confirmed.
issues: I would like to see additional ablation studies for checking the effect of the number of clusters.
In particular, please clarify the limitation in terms of learning efficiency and computational cost.
In addition, we would like to know if the Interaction Steps on the horizontal axis of the learning curve in the paper include those required for learning pi_k.
If not, the authors should clarify the number of iteration steps required for learning pi_k, which reveals the decrease of learning efficiency in total.

From the benchmarks provided, it seems that the sub dataset extracted according to the transferability measurement is designed to avoid multimodality.
It would be better to evaluate the performance when it is multimodal, if possible.

Only for defining transferability measurement, it may not be necessary to cluster the dataset, and it seems sufficient to construct a metric space.
It would be good if the authors could add the necessity of clustering in this regard.
review_title: null

Meta-review: metareview: This paper studies an important problem of learning from multimodal demonstrations where multiple demonstrators have different dynamics than the imitator. This is relevant to the robotics community, though there were no real robotics experiments. The reviewers found the work to be original and well-motivated, easy to follow, and appreciated the inclusion of several ablation studies. The limitations of the approach were discussed but could be improved. Comparisons to different clustering methods and investigating the impact of the number of clusters would improve the paper. The authors are encouraged to address these concerns along with other concerns raised by the reviewers.

============

Thank you to the authors for addressing the authors concerns. I believe the paper makes enough contributions to warrant acceptance to the conference. One reviewer has a remaining concern of statements made within the paper as summarized here:

A deterministic policy can indeed model a multimodal trajectory distribution, as long as the dynamics distribution is stochastic, which can generally cause a policy to lead to different outcomes for repeated trials from the same initial state. 

Furthermore, even if the modelled policy is Gaussian, it cannot be seen as deterministic. A Gaussian distribution is never deterministic, except for the degenerate, 0-variance case. 

Please address these concerns in the text of the final paper.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)