Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ** Paper Summary **

This paper proposed a novel framework to establish the correspondences across observations and actions, by using dynamic cycle-consistency. The paper focused on robotics applications, but the technical solutions are based on a pioneering method computer vision, generative adversarial networks (GANs) and cycle-consistency constraints, whose robustness has shown in many previous literatures. The proposed method was applied and attained the robustness on various tasks, such as cross-physics alignment, cross-modality alignment, cross-modality-andphysics alignment. 

** Paper Strength **
+ The paper is well organized and written.
+ Borrowing the concepts of GANs and cycle-consistency to solve the unpaired problems in cross-modal correspondence makes sense and good performance gains are expected.
+ Solving several simulation-based settings and real robot settings was nice.

** Paper Weakness **

I have no major comments on this paper, but have some minor ones.
- In Table 5, Ours (E) and Ours (J) have shown the performance gaps. It would be great if the reasons are properly mentioned.
- In Table 4, the authors compared the quantitative results with Cycle-GAN and INIT. It would be great if the thorough discussion for these experimental settings are provided.
- Computational complexity analysis is also required.
review_title: Review for "Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency"

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The paper provides an interesting (and to my knowledge, novel) approach for learning a mapping of actions and observations from one domain/character to another reasonably similar domain/character. This mainly allows the transfer of skills (i.e. policies) from one domain/character to another. Immediate use cases of this approach are in imitation learning and sim2real transfer. In order to learn these mappings, the authors use the idea of cycle-consistency in generative adversarial networks and adapt it to the imitation learning task. Importantly, this allows them to obviate the need for paired state samples across domains. The authors show that their method not only works across modalities (vision from real robot to states in simulated robot), but it also works to some extent on different character morphologies.

Reasons for score:
Overall, I vote for accepting this paper. To my knowledge, the method is novel and provides a viable and interesting approach for imitation learning and sim2real transfer. The experiments and the final quality are also high, however, the broader applicability to more challenging tasks and its limitations remain to be seen.

Cons:
- The chosen tasks are (understandably) simple, therefore the applicability of the methods to more challenging environments remain to be seen.
- The limitations of the method are not discussed well. I believe some commentary on the challenges of this method (e.g. the ease of training with GANs) would be useful. Also, more discussion on the use cases of the method and where it excels existing methods is missing.
- The code is not provided which hampers reproducibility. I strongly suggest providing the code if possible.
 
Question: 
- How does the method handle partial observability? In the robot arm example, a single image of the robot does not contain information about the velocity and angular velocity (of the joints). I'm confused as to how the model can actually infer these or work without knowing them.
- Can you spend more time explaining Figure 3 (c)? It seems strange that the "L1 error" increased when more data was available.
- Related to the last question: in this case, can you pre-train G the same as F and use cycle-consistency only for H and P? A brief explanation would suffice.


Fixes or suggestions:
- I will have to double check the conference style guides, but having tables and algorithms intertwined with the text in a single-column publication is not pleasing to the eyes. I suggest rearranging these elements.
- The plots in Figure 3 can be improved. At the very least, the distortions caused by resizing the images should be addressed.
review_title: Novel and interesting work

final_score: rating: 7: Good paper, accept
review_content: This paper presents a technique that leverages cycle-consistency to align data across domains given only samples of one-step trajectories (x_t, a_t, x_{t+1}) from each. The cycle-consistency is used over the space of actions across two domains: e.g., given data from X [in the form (x_t, a_t, x_{t+1})] and data from Y [in the form (y_t, u_t, y_{t+1})], the system should be able to discover an "alignment" between states across the two domains by enforcing that the action spaces be mappable to one another (either identical, as in the "cross-modality only" applications, or via cycle-consistency, as in the "joint model" applications). The authors go on to show that their approach, among other things, allows for strong alignment between real-world images and their corresponding underlying state.

Overall, the paper presents some solid results across a handful of interesting domains/applications. The paper is occasionally difficult to follow, particularly on the first read-through, in part because it is unclear precisely what form the data takes until Section 3, a problem not alleviated by a somewhat-unclear Figure 1 (see more detailed comments on this below). The introductory paragraph beginning with "In this paper" is quite hard to follow. Providing some more concrete examples in the introduction would be incredibly helpful for clarity. For example, mentioning that the input data to the learning algorithm takes the form of 3-element tuples (and their components), would be helpful.

The authors might also consider combining and aligning Figures 1 & 2 to help the reader follow along with the early pages of the paper. Perhaps also specifying the domain (i.e. functional domain: the space of the variables) on the figures themselves would help here. This lack of clarity in the beginning of the paper is likely its biggest weakness, as it is otherwise a good theoretical idea supported by strong results.

Smaller comments:
- Though it is clear why training _all_ of the networks jointly would lead to mode collapse (motivating the need to train F separately), it is not made particularly clear why the remainder of the system needs to be trained using the 'alternating' procedure employed throughout. A sentence or two explaining this decision would be helpful.
- Particularly in the figures, the authors should make clear that the `y` images are in fact renders corresponding to the predicted state. Without this "disclaimer" readers coming from the purely-image-driven cycle-consistency community may mistakingly believe that those images are the direct output of the CycleGAN baselines or the proposed approach. Modifying the green-backgound images in Fig. 1(a) to include "Render of y_t" (or similar) would help avoid such a confusion.
- Figure 1(b) is potentially misleading, since really the images are assumed to be identical between the two domains. Additionally the arrows on (b) and (c) in fig. 1 are also somewhat misleading, since it is not clear what the arrows represent. Consider adding more annotations on this figure describing in more depth what this figure is trying to communicate: namely the types of applications enabled by the approach, rather than the structure of the proposed approach itself.
- The Related Work section is quite thorough and helpful for providing context. Two other papers that the authors might have missed and consider citing that made progress in solving similar tasks are (1) "Adapting Deep Visuomotor Representations with Weak Pairwise Constraints" (Tzeng, Devin, et al, 2016) [which uses a task-specific loss for aligning sim/real robot arm images] and (2) "GeneSIS-RT: Generating Synthetic Images for training Secondary Real-world Tasks" (Stein & Roy, 2018) [which uses cycle-consistency for task-agnostic sim-to-real and then uses the translated data for quadcopter flight]. Both papers circumvent the need to collect paired sim and real images for real-world robotics applications.
review_title: A somewhat small yet solid contribution; paper clarity could improve

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: ##### Summary

This work is concerned with learning mappings between pairs of domains that may differ in representation (imagery vs. configuration state), physical parameters, morphology, or combinations thereof, relying only on unpaired training data from both domains.  The key idea is to leverage the (existing) concept of cycle consistency, incorporating dynamics to formulate a cycle loss that spans vastly different domains.  The cycle is closed by learning a domain translator $G$ that maps states (the paper calls them observations - this is unimportant as the system makes no such distinction, as either domain might represent a "state" or an "observation") from domain $x$ to domain $y$, and a forward model $F$ that operates in the $y$ domain.  In addition, functions $H$ and $P$ are learned that map actions $a$ and $u$ between their respective domains $x$ and $y$, in either direction.  The difference between $G(x_{t+1})$ and $F(G(x_t), u_t)$ is then the part of the overall loss function that closes the loop. $G$, $H$ and $P$ are trained adversarially using cycle consistency losses; $F$ can be trained by regression, exploiting training sequences in the $y$ domain.

The system is evaluated on four different tasks from OpenAI Gym, and is shown to be effective in diverse settings.

##### Strengths

* This work addresses an important, fundamental problem that arises in many applications, including state estimation and sim2real transfer.
* The paper makes a significant, nontrivial contribution to this problem, allowing hard cross-domain transfer problems to be solved dramatically better than before.
* The work is well related to prior literature.
* The claims are supported by the empirical results.
* The paper is well organized, readable and clear.
* The appendix provides a lot of additional detail important for reimplementing this method and replicating the results.

##### Weaknesses

* The implementation details, in particular the neural-network architectures, appear to be very specifically tailored to the scenarios of evaluation. Instead it would be helpful to provide insight into how a user of this method should go about designing the networks, given their application domains.
* For cross-modality alignment, paired training data are typically available.  How can the method take advantage of this? How would this impact the results, e.g. those of Table 2?  Some related results are given in Fig. 3(c) but it is unclear what exactly was done and what exactly the graph shows.
* An obvious and explicitly-stated application of this work is sim2real transfer, but no such experimental results were provided (I realize that it is not easy to set up such an experiment from which strong conclusions can be drawn).

##### Recommendation

Unless I am missing important similar work, the paper makes a strong contribution in an important area.  It should be accepted.  While I wrote more text under Weaknesses than under Strengths, the weaknesses are minor compared to the strengths.

##### Questions for the Rebuttal

* The results on cross-physics alignment (Table 1) seem to leave a lot of room for further improvement.  What is the limiting factor?
* Please comment on the listed weaknesses.

##### Details

* "RL score" = return?
* There are many little typos.

review_title: Strong contribution to a hard, important problem with many applications

Meta-review: The paper proposes a new solution for cross-domain correspondence in control, which combines GANs and cycle-consistency, and separates shifts in observation space and in action space. The paper targets unpaired data / simulations, and discovers alignment of state by enforcing that domains are mappable.

The paper was received well by reviewers, who pointed out several strengths: a strong contribution on a fundamental problem, and an interesting formulation; a well written and well positioned paper; This compensates minor weaknesses, in particular the fact that transfer has been tested between two different simulated environments. 

The reviewers unanimously suggested acceptance, the AC concurs.
Decision: Accept (Oral)