Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work introduces the use of Temporal Cycle-Consistency (TCC) for imitation learning. By training an encoder via TCC to embed similar states across different embodiments into similar latent variables, the authors can use proximity in latent space to a goal state as reward for imitation. The authors show this to great success in an artificial sim-to-sim 2D task and to decent success in a real-to-sim task.

To me, this paper is pretty water-tight. The method isn't groundbreaking because it's mostly using TCC, but the idea of using that as reward is novel enough to warrant publication. The experiments are all solid, the baselines are solid, and I'm honestly failing to see any major weaknesses. So I'm recommending acceptance.
strengths_and_weaknesses: ### Strengths:
- Great experiments. 2D sim2sim environment is a great illustration and quick turnaround cross-embodiment playground.
- Writing is great and clear. The background section is just right for my personal taste in terms of explaining the basics but not overexplaining anything.
- In the appendix there is a bonus dataset that should be put into the main body of the paper, X-REAL, that looks fun.
- Very specific about the implementation and training details (in main paper and appendix), great work! 
- Using 5 seeds for all RL experiments, good job!
- The smooth (learned) rewards in Fig. 6 and 7 are excellent.

### Weaknesses:

(None of these are major, just little things that I think might improve the paper)

1. Maybe also include or at least mention VIRL (https://arxiv.org/abs/1901.07186), since it's similar to TCN and needs fewer expert samples. Compared to your method, it takes an embedding of the entire sequence so far and compares it to the expert.
2. Fig.1 left side, bottom - what's that extra encoder at the bottom, and what's its output? How's it trained? Isn't that encoder part of the TCC training process? If that's the case, just remove it.
3. In line 123 you refer to Fig.2c but the task wasn't really introduced yet. So to the average reader, up until then, Fig.2c is just squiggly lines.
4. line 169 - you're using the mean embedding of all agents' last state as goal? Couldn't that potentially lead to an unreachable goal? If I look for example at the t-SNE embedding of the real videos, some of the final states are very far apart. Maybe as an analogy, when generating new images with a VAE by interpolating between two latents, when moving linearly through the state space, the results are usually unrealistic-looking, but when moving across a hypersphere between both samples, the results look much better (see here for example https://arxiv.org/abs/1609.04468). Maybe it'd be better to take the middle point on a hypersphere instead of the euclidean mean?
summary_of_recommendation: As mentioned above, the paper is very solid.
issues: Please see weakness section
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a method of learning cross domain representations.  These representations can be used as a reward signal to speed the training of RL policies.  
strengths_and_weaknesses: The paper is mostly clearly written and presents an interesting new method.

The performance using human data is impressive, both on the X-Real dataset and on the RLV tasks.
summary_of_recommendation: The paper presents a novel method for learning cross domain embeddings and reward signals and provides extensive experimentation to show that the proposed method is effective in a variety of environments.
issues: The paired data in RLV is in the expert domain, but is off policy. 

What is the difference between Env Reward in table 2 and Env Reward Only in Figure 5?

I don't see the training curves for the Drawer Opening task, despite Figure 7 including it.

The X-Real tasks would be better in the main paper, rather than the supplemental.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Paper proposes to use TCC (temporal cycle-consistency learning) for learning representations in cross-embodiment visual imitation learning tasks. The authors empirically demonstrate that the learned representations can deliver a simple reward function (the distance between the current state and goal representations) that is sufficient for RL. A cross-embodiment benchmark (X-MAGICAL) is also presented and the proposed approach attains superior performances on both the simulation and real-world data.
strengths_and_weaknesses: [Strengths]

[+] The paper is overall clear and well-written. I like how the authors approach their main results in an illustrative fashion and I believe this paper can also be easy to follow for readers without dense imitation learning background.

[+] The proposed method is well-motivated. I agree that cross-embodiment imitation learning can effectively be a representation learning problem and TCC seems to be a reasonable inductive bias to produce the needed representations. I'm a little bit surprised that the resulting representations can even provide a straightforward reward solution. 

[+] I particularly like the real-world experiments and the results are truly impressive.

[Weaknesses]

I don't have major concerns at this point but I do encourage the authors to address the following issues in a rebuttal:

[-] It is still unclear how the agent can benefit from cross-embodiment pretraining. Indeed I think the crucial question is: does the agent learn from other embodiments or just learn from visual pretraining. The following experiments may help clarify this question:

- Add a baseline that just leverages general-purpose self-supervised visual pretraining, e.g. MoCo, SimCLR.
- Try to learn from other embodiments only in the pretraining stage.

[-] Some papers on imitation learning from observations (somehow shares the goal of cross-embodiment imitation learning) should be cited:

[1] Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement

[2] Learning and Inferring Movement with Deep Generative Model

[3] Third-Person Imitation Learning

[4] Provably Efficient Imitation Learning from Observation Alone
summary_of_recommendation: Given the overall technical quality, I recommend acceptance but I'm open to other assessment if there is something I missed.
issues: I may consider further increasing my score if the following points can be addressed:

1. Citations to the aforementioned papers.

2. Extra results as requested above.
review_title: null

Meta-review: This paper proposed to learn cross domain representations as a reward signal to achieve RL policies, and present a self-supervised method to learn deep visual embeddings considering embodiment differences. 

I agree with reviewers that this paper is well organized and the idea is quite novel, the extensive experiments demonstrated the proposed method is effective in a variety of environments.

I believe this is a good work, I agree with reviewers on some minor points they mentioned, and suggest authors carefully address these comments to further improve this paper. 

More added: 
I would like to thank authors for your carefully and detailed responses to comments raised by reviewers, I really believe the revised version will be more better. Congratulations!
Decision: Accept (Oral)