Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to transfer policies between different MDPs based on the minimization of Gromov-Wasserstein distance. This distance provides a pseudo-reward that can be used to learn via RL the optimal policy in the target MDP given an optimal policy in the original MDP. The method is optimal if the MDPs can be mapped into each other through an isometry, but works also empirically in other cases.

main_review: Strengths:
- Good mathematical grounding
- Further exploration of an interesting alternative to map optimal policies to new embodiments (possible practical applications)
- Well written

Weaknesses:
- The main issue with this paper is the experimental evaluation. The presented results are just images of three cases. The images of the first case (Fig. 3) are hard to see. There is no numerical performance (success, reward, some degree of progress). This makes the experimental evaluation insufficient to understand the applicability of the presented approach. Add more experiments, numerical performance, different metrics…

summary_of_the_review: In summary, while the paper presents an interesting turn on previously presented ideas, and the mathematical foundation is well worked out, the experimental evaluation is insufficient to support conclusions about the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors focus on a more general cross domain imitation learning problem where only expert demonstrations from one domain is available. To solve such a problem, the authors use the Gromov-Wasserstein distance to align and compare states between tasks from different domains and propose a Gromov-Wasserstein Imitation Learning (GWIL).  They also show theoretically the possibilities and limitations of GWIL. 
main_review: Strengths:

1.	This paper introduces and addresses an important and general cross domain imitation learning problem. 
2.	Appling the Gromov-Wasserstein distance to align and compare two MDP domains provides insights to study cross domain imitation learning.
3.	The proposed GWIL is novel. The authors also well justified limitations theoretically.

Weakness:

1.	There are only 3 tasks shown in the experiment section. More experiment results are preferred to show the effectiveness of the proposed solution. 
2.	Experiment results are not well-visualized. It would be better to give a link showing the results in animation.

summary_of_the_review: In general, the paper addresses a general cross domain imitation learning problem where only expert demonstrations are available and proposes a novel GWIL. Though the experiment results are not visualized well, the work is highly likely to show new insights to researchers in imitation learning and domain adaptation domains.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A method is proposed for cross-domain imitation learning, without resorting to any form of correspondence.
This is done using a Gromov-Wasserstein distance between policies (in practice, Euclidiean distances on collected state-action pairs, 
within a given domain), which finds isometric transformations that best preserve distance measures, between the two domains.
Given an imitation domain and an expert domain with example trajectories, a pseudo-reward is computed 
based on the degree to which the distances from a state to its neighbors in the imitation domain,
are preserved in the expert domain. Given these pseudo-rewards, as computed for collected episodes,
SAC is used as an RL algorithm to optimize the policy. 
The paper contributes both a theoretical analysis and experiments with:  U-maze, pendulum-to-cart-and-pole, and half-cheetah-to-fallen-walker.

main_review: Strengths:
- novel idea, to the best of my knowledge, for a difficult problem;  will inspire future work on "learning by analogy"
- combination of theory, and empirical experiments;  I'm surprised by the extent the method works in practice

Weaknesses:
- with no learning curves presented, it is unclear if the cross-domain imitation learning actually provides a benefit,
  for non-trivial systems, in terms of learning time or performance, as compared to learning from scratch.  It would be
  beneficial to see these learning curves, and a wall-clock compute time comparison.
- the limitations could be better articulated. The scalability is unclear (although it is unreasonable to expect the first 
   iteration of an idea like this to scale right away). The isometry constraint is likely to be limiting in many settings, 
  as is the choice of the Euclidean distance metric in the (state,action) space.
- lack of an intuitive presentation of the Gromov-Wasserstein distance. I had to go elsewhere to obtain the intuition.
  The actual method used to compute the GW distance, on discretely sampled trajectories.

The notation used for the GW sets-of-state action-pairs is confusing, i.e., \tau,
because the GW-distance is invariant with respect to temporal ordering (to my understanding), whereas the notation
GW(tau, tau')seems to imply that the ordering needs to be preserved.  Perhaps introduce a different notation for
the data when the temporal information no longer needs to be preserved?
The connections of the trajectories / (s,a) data / occupancy measures needs to be better articulated for this reader.

Is there a simple figure that could depict the essence of eqns 6 and 7?

Figure 1 appears to simply show rotations.  If the goal is to show isometry for translations,
it would be better to scatter the spirals more irregularly throughout the domain of the figure.
Similarly, why not include a reflection, as stated in the caption?

In Figure 3, the agents position is largely invisible.  Most of the readers may simply think there
is an editing mistake and that the same figure was included 8 times.

In Figure 4, do the top and bottom row come from GW-corresponding state/actions?

re: adding time to (s,a) to preserve uniqueness
Wouldn't this cause problems, given that the GW-distance would now include time?


summary_of_the_review: The paper introduces a novel idea for imitation learning. 
It likely has many limitations, but the idea of find suitable imitation-based correspondences is one that is being
pursued on multiple fronts, and this is a new approach, with a mix of theory and some initial proof-of-concept examples.
The paper could do better at explaining core ideas, and still needs learning curves in order to understand the benefit of the
cross-domain transfer.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper frames cross-domain imitation learning as an optimal transport problem using the Gromov-Wasserstein distance. This problem is highly relevant to imitation learning settings where there is often substantial domain mismatch between action and state spaces, eg. a humanoid robot learning to walk from a human demonstrator. The paper introduces a reward function that can be optimised and proves that this is equivalent to minimising the Gromov-Wasserstein distance between state action occupancies of an agent and expert. Substantial discussion/ proofs are included to show that minimising the Gromov-Wasserstein distance is equivalent to recovering an optimal policy up to an isometry. This is both a blessing and a curse, as it allows for optimal policies to be recovered under extreme changes in domain or differences, but does mean that recovered policies could be entirely unsuitable due to isometry. 

The paper is well written and concisely written, although does get excessively mathy at times, when a figure could be more helpful. Experimental results corroborate the proofs and propositions, and highlight the value of the proposed approach.



main_review: Strengths:

This looks to be a strong contribution, attempting to solve an important problem. The paper is well-written and relatively easy to follow, and results and proofs are interesting.

Questions:

It's unclear how easy an objective the proxy reward is to maximise. I would appreciate more clarity around this (eg. can you show some reward curves across multiple seeds/ runs), and convergence. The Gromov Wasserstein distance is quite an expensive distance to compute, can you comment on the computational feasibility of optimising eq 7 and potential scaling issues? 

Repeated mention of seed dependencies and effects is also concerning, I would appreciate more commentary on this.

While I agree there are certainly settings where the Gromov Wasserstein distance makes sense from an imitation learning perspective, recovery up to an isometry can be prohibitive. eg. A human showing a drone how to take-off could result in a policy that lands drones, which is the opposite of what was demonstrated. I would value some discussion on these limitations - would it make more sense to optimise a different distance metric, or to use a different eg. non-euclidean kernel in settings like these? I suspect this is a non-trivial choice, that needs a substantial level of domain specific knowledge  - does this then run counter the original objectives of this work?

Minor:

Pg 1. Intro is well written, but it would be great if Fig 2 could be shown earlier to give some more intuition into the Gromov Wasserstein distance and the solution framing.

Pg. 2 typo "This takes us beyond limitation..."

In existing imitation learning literature, much is made around limitations around learning from non-expert demonstrations. I would be interested to hear how the proposed approach would cope with these?

Eq 1 is in dire need of a figure to explain this.

As I understand it, although all proofs are provided in finite action and state spaces the proposed approach is said to scale to continuous spaces as ultimately it is only reliant on a suitable kernel function that can be expressed for continuous spaces. Is this correct?

"We will see that in practice, running our method on different seeds enables to find an optimal policy in the agent’s domain". How do we know which seed produced the "right" policy?

Fig 3 needs improving - it is extremely hard to see the agent.

Fig 4/5. I'd love to see videos of these policies - did it actually learn to balance cartpole, or just to swing up? 





summary_of_the_review: I enjoyed reading this paper, and think it adds greatly to the conversation around cross-domain imitation learning. The proposed approach has a number of strengths and limitations which I would appreciate hearing more about, particularly when it comes to convergence speeds, repeatability and computational requirements, but also whether strengths/weaknesses of optimal recovery of a policy up to an isometry have just shifted the need for specification of a mapping between expert and agent into a different domain.

==== Post rebuttal comments ====
Thank you for engaging in the process, I still believe that this is a good paper.
review_title: null

Meta-review: All reviewers suggested acceptance of the paper based on that the paper addresses an important problem and presents and validates interesting ideas for approaching it. Therea are some concerns regarding limited experiments - I'd like to encourage the authors to make an effort to address these concerns and also a few others raised in the reviews in the final version of their paper. The authors already made several updates to their paper in that regard during the discussion phase so I believe that the paper would be an interesting conttribution to the conference and I am recommending acceptance of the paper.
Decision: Accept (Poster)