Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed to learn skills correspondences between robots of different morphologies in an unsupervised way - without requiring paired data from the robots. They learn a skill translation model that maps skills from a source robot to a target robot. The translation model is learned by minimizing a loss composed of two components: (1) the likelihood of the translated skills under the target distribution estimated from target robot skills data, and (2) the likelihood of the target skills under the translated distribution estimated from the translated skills data.

The proposed approaches hinges on the assumption that robots with different morphologies follow similar high-level strategies when performing similar tasks. So, a sequence of tasks across the robots is used to learn the skills correspondences. 
main_review: Strengths:

The paper avoids the need to manually annotate datasets with low-level correspondences between robots, which requires expertise when dealing with robots with different morphologies. Instead they provide correspondences over skills, which is much easier to specify, even by non-robotics-expert users. 

Their experiments show superiority over baselines from the literature. 

Weaknesses:

Although the Baxter and Sawyer have different dynamics, their kinematics are not that different. Yes, the link dimensions are different but they have the same number of degrees. I would have loved to see experiments on robots with different number of degrees of freedom as that is more challenging. The argument that the choice of robots is dictated by the availability of the datasets is kind of weak given experiments are run in simulations. It can't be hard to generate a similar dataset for other robots. 

I would have loved to see an ablation study on the quality of skills across the robots. Surely, the data generation process needs to make sure there is some alignment of tasks across the robots. How much of this has an effect on the quality of the translated skills?

I would have also loved to see an analysis of the training time of the translation model in relation to the time it takes to learn the skill from scratch with the target robot.

One important aspect of transfer that is usually ignored, is the cost of transfer compared to learning from scratch. And the cost could be the time it takes to collect data to learn the translation model compared to learning the skills from scratch using the target robot. Of course, using data already existing avoids this but is still a problem for robot for which data doesn't exist. See [1] as an example, albeit in a different domain.

It's not clear if the translation model is evaluated on unseen skills or trajectories. My guess is no. How well does it generalize to unseen skills?

In Table 2, how come rewards in the source domain are higher than in the target for the HRL baseline? Especially between Bax-R and Bax-L. How different are these tasks when performed with either the right or left hands?

Moreover, why should we expect the translated trajectories obtain higher rewards in the target domain than in the source domain? Especially between left and right hands.

1. Makondo, Ndivhuwo, and Benjamin Rosman. "Towards improving incremental learning of manipulator kinematics with inter-robot knowledge transfer." 2019 Southern African Universities Power Engineering Conference/Robotics and Mechatronics/Pattern Recognition Association of South Africa (SAUPEC/RobMech/PRASA). IEEE, 2019.
summary_of_the_review: The paper proposes a novel idea of learning skills correspondences between robots with different morphologies in an unsupervised way, borrowing ideas from machine translation. The contributions are solid, with a few minor evaluation weaknesses as highlighted in the main review above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of transferring skills between morphologically different robots. This approach to learning skill correspondences is framed as a problem of matching distributions of sequences of skills across robots. The paper proposes an unsupervised objective, inspired by work in unsupervised machine translation, that makes the skill translation model learn to match the distribution of skill sequences. The proposed approach is experimentally evaluated on 3 transfer settings in a simulated robot environment.
main_review: 
The idea of being able to reproduce skill sequences between different robots based on videos or visual demonstrations is a very hard problem with potentially very impactful applications!

In this current work, I am not quite sure what is considered as morphologically different? Does it refer to different kinematics or robots with different DOFs?
Depending on this definition, the assumption: “different morphological robots use similar task strategies (in terms of sequences of skills) to solve similar tasks” might not always hold. Although the high-level of semantic skill sequences might be the same, the actual trajectories corresponding to those skills can be significantly different.

It would be useful to clearly emphasise the contributions of the approach, as many parts rely on TVI introduced in Shankar & Gupta (2020).

I am not sure how Eysenbach et al. (2019) and Sharma et al. (2020) are related to the presented approach. However, I think there are several older related works, which would help provide context and give motivation to the proposed approach, such as:
- Movement primitives:
  - Kober, J. and Peters, J. Learning motor primitives for robotics. ICRA, 2009.
  - Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., and Schaal, S. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 2013.
- Movement scaling/retargeting:
  - https://mediatum.ub.tum.de/doc/1192176/file.pdf
  - https://hal.archives-ouvertes.fr/hal-02398106/document
  - https://hal.archives-ouvertes.fr/file/index/docid/1054887/filename/Sakka2014.pdf
  - https://www.worldscientific.com/doi/abs/10.1142/9789814623353_0008

As well as work on imitation learning, learning from demonstrations and hierarchical RL, similarly as it is discussed in the related work of Shankar & Gupta (2020).

It would also be beneficial to show the performance of the proposed approach in comparison to simple skill transfer or learning from demonstration approaches as a baseline.

Text around the images in Fig2 is too small and difficult to interpret, maybe some alternative way of displaying it would aid visibility

summary_of_the_review: - Clarify contributions and assumptions
- More discussion on the limits of the approach w.r.t. robot morphologies.
- Update related work discussion
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an unsupervised learning method to identify skill correspondences for translating skills between robots with different morphology. The approach is inspired by unsupervised machine translation, and uses an unsupervised objective to learn a skill translation model to match the distributions across domains.
main_review: 
Strengths:

+ The idea of transferring skills between robots with different morphologies is interesting. 

+ Presentation of the technical approach is clear.

Weaknesses:

- What are the theoretical novelties? The novelty of the paper is not clearly discussed and justified in the paper. The idea of using forward and backward losses in the objective function is widely used to maintain consistency in domain adaptation and transfer learning. 

- The insight/premise that differently morphological robots follow similar strategies to address similar tasks seems questionable. For robot manipulation, this can be true. But for robot locomotion as an example, this assumption may not hold: robots can perform locomotion by walking, crawling, hopping, and wheeling, based on the morphology of the robot. How does the premise hold in general robotics tasks?

- The terms (skill, strategy, and mode) are confusing, and it is hard to imagine these terms in the context of a robotics application. Can the authors provide a concrete scenario of robots using a sequence of skills to complete a task? What are the examples of skills, strategies, and modes in that concrete scenario?

- What is the runtime (or computational cost) of the proposed approach and other compared methods? How many iterations are used for the algorithm to converge? The paper argues efficiency several times, but does not provide clear theoretical analysis or experimental support. 

- How to segment a sequence of observations into skill tuples? How the start and end points of each skill are determined when segmenting a long sequence?

- “Analysing qualitative skill translations” in the experimental section 4.1 has no experimental support. No experimental results are provided to justify the statements in this subsection.

- It is not convincing to argue the left arm and the right arm of a Baxter robot have a different morphology in the experiments. The Sawyer robot and the Baxter robot’s arms also have similar design and configurations. In the application of robot manipulation, suction cups, soft grippers and traditional mechanical grippers can be considered as morphologically different robots, or maybe robot arms with obvious differences, e.g., in terms of degrees of freedom. How the proposed approach can be generalized to transfer skills among these robots with obvious morphological differences?

summary_of_the_review: While the idea of transferring skills between robots with different morphology is interesting, the paper is preliminary and will need improvements on explaining its novelty, premise, and terminology; more convincing experiments using robots with obvious morphological differences are also expected.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper frames learning skill correspondences as a problem of matching distributions of sequences of skills across robots, and presents an unsupervised objective that is able to learn semantically meaningful correspondences between skills across diverse robot domains.

main_review: The paper is written clearly and is well structured. The contribution is presented in a clear way, the technical formulation is sound and claims are supported by experimental results.
The visual representations (diagrams, figures and videos) are clear and help understand the contribution. 
Implementation details are provided and comparisons with alternative methods allow to appreciate the benefits of the proposed method.
It could be argued that the reach skill is not complex enough to appreciate the accuracy of the proposed method in translating from one domain to another. This is partly captured by the results in table 2. Can you discuss these in more detail? In particular, can you explain the cases where the reward in the target domain is greater than that in the source domain?
How do you expect your method to transfer to real robots?

summary_of_the_review: The paper presents a method that is well formulated, sound and well motivated. The experimental results are convincing but require some further explanation. Details of implementation and experimental results are provided, so that it could be possible to reproduce the results.
review_title: null

Meta-review: The paper proposes an algorithm for unsupervised skill transfer between robots with different kinematics. Integral to the approach is the idea that while the robots differ, they may use similar strategies to perform similar tasks. Without access to paired data, the paper formulates the problem of learning correspondences between robots as one of matching skill distributions across robots. Drawing insights from work in machine translation, the paper proposes an unsupervised objective that encourages the model to learn to align the distribution over skill sequences. Experimental results demonstrate the ability to use learned skill correspondences to support transfer across different robots in different domains.

As several reviewers point out, the problem of learning to transfer skills across robots with different kinematic designs from video demonstrations raises a number of interesting challenges that are relevant to the robotics and learning communities. Among them, a fundamental contribution of the paper is the ability to learn skill correspondences in an unsupervised manner based on unlabeled demonstrations. The approach by which this is achieved (i.e., using distribution matching) is sensible and clearly described. While the reviewers agree on the significance of the research problem, they raise a few key concerns regarding the initial submission. Among them are questions about the nature and extent of the domain variations that the method can handle (e.g., between robots with different degrees-of-freedom); the significance of the contributions; and how this work is situated in the context of existing approaches to robot skill learning. Several reviewers question the definition of morphological variation and comment that these variations may violate the assumption that task strategies are similar across designs. The authors provided detailed feedback to each of the reviews, which helps to clarify several of these concerns. Unfortunately, several reviewers did not respond to multiple requests to update their reviews. The one who did decided to maintain their score.

The paper tackles an important problem in robot learning and the work has the potential to have significant impact on the way in which robots acquire new skills. The original submission together with the author responses suggest that there is are solid contributions here. The authors are strongly encouraged to revisit the discussion of the approach to more clearly convey the novelty of the approach and to consider experimental evaluations that better support these claims.
Decision: Reject