Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper proposes a learning approach for zero-shot imitation learning in an RL setting across domains with different embodiments and viewpoint mismatch. The proposed approach involves two steps, alignment and adaptation. In contrast to previous work, the alignment between domains, represented as MDPs, is learned from unpaired, unaligned samples from both domains. The paper presents a theoretical formulation of the cross-domain imitation learning problem, and presents an algorithm for training alignment and adaptation from data. 

I think the paper is well written and theoretically well-founded. The authors provide experiments comparing to many previous works in cross-domain imitation learning and show that their approach outperforms previous approaches. 

I only have minor comments.

1. For imitation between reacher2 and reacher3, there are multiple correspondences between the two domains due to the redundancy in the 3-link robot (and in general with n-link robots). How does GAMA deal with these redundancies? For example, an n-link robot is able to perform the same task with different configurations (elbow up, elbow down, etc.,) and all these will correspond to one configuration of a 2-link robot. 

2. In practise, can 'alignment complexity' and 'adaptation complexity' help identify if transfer with GAMA between two domains is not beneficial? Results in Figure 5 only shows cases for GAMA in which the two metrics are good, resulting in good transfer. However, I am interested in knowing if there could be cases in which the two complexity metrics tell beforehand that transfer will not be beneficial. 

3. In Figure 5's caption, Adaptation complexity is on Left and Alignment Complexity in the Middle. However, the text refers to Alignment complexity on the Left and Adaptation complexity in the Middle.

4. A reference is missing in Appendix B.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes Generative Adversarial MDP Alignment (GAMA) for imitation learning. Given a set of paired MDPs, GAMA learns a state mapping f and an action mapping g such that one MDP can be reduced to another. For a new test MDP pair (x,y) where expert demonstrations are available for y, GAMA can use f to map a state of x to a state of y, mimic the expert behavior, then use g to map the expert action back to an action in x. The reduction is theoretically motivated, the optimization is based on adversarial learning and finally, experiments on common Gym environments show that GAMA can achieve effective transfer.

Pros
- The writing is great and easy to follow
- The method is theoretically motivated
- Experiments prove effective

Cons
- The proposed method may not work well for complicated environments

(1) In the discussion after Def.4, given an alignment task set D_{x,y}, how do we know whether a common (w.r.t. all MDP pairs) reduction exists? In other words, how do we know that the MDP pairs are from the same equivalent class (joint alignable) in practice?

(2) The alignment needs to learn a lot of components: state mapping f, action mapping g, domain dynamics P^x. The domain dynamics can be difficult to learn for complicated environments, which may jeopardize the learning of f and g as a result because they depend on the accuracy of the learned dynamics. The experiment only uses the hidden representation for the image as input. Such lower-dimensional representation is not always available.

(3) Experiment:
- What happens for the UMA in Fig.4 top-right?
- Table 1 only has the results of three alignment tasks. How about the rest?
- What are the error bars in Table 1 (and also Fig.5)? Based on how many runs?

Minors:
- It is more common to use "state-action pair" instead of "state, action pair".
- Sec.6.1, task task exemplify
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary
This paper proposes a method to perform alignment between demonstrations in different domains in order to adapt a policy from expert domain to the test domain. They also include a formalism that studies alignments of different MDPs.

Strengths
1) The paper presents a formal framework to study many recent methods in cross-domain/cross-view imitation learning with a common lens. 
2) The solution proposed by the authors might have a big practical advantage for cross-domain imitation learning settings. They claim that they do not need to learn the policy using RL in the new domain as they learn the action mapping between the two domains to retrieve the corresponding action in new domain based on the action in expert domain. This is a very interesting result.
3) The solution is quite modular with 3 important parts: i) state mapping network, ii) action mapping network, iii) dynamics model. 

Weakness
The biggest weakness of the paper is in the evaluation section. 
1) Issues with baselines:
i) There seems to be some problem with UMA. In Figure 4, it cannot figure out the alignment with self domain. Why is that the case? Where is the cartpole in UMA row for pen-cartpole alignment ? Given such poor performance during alignment is it fair to compare to UMA in the subsequent section?
ii) Why are the other methods not able to align well when there is no domain shift? 
iii) How is dynamic time warping actually used to calculate the state correspondences for training IfO and IF? Is it calculated on top of the states?
iv) How are the policies trained for the transferability task? Which RL algorithm? "Baselines fail to learn the writing task as an inaccurate proxy reward function harms performance." 
2 aspects are involved here:
1) initializing from alignment network
2) using rewards from the state representation to train the policy.
Do the algorithms still fail if the true reward is used with the alignment initialization? 

Reward scaling is known to be important for RL algorithms[1]. May be features from these algorithms have to be scaled to scale the rewards for the RL algorithms to work.

2) Issues with environments:
i) The environments considered for domain transfer seem quite trivial for the alignment between them to be a big problem. For example, the authors present the task R2W: reacher2-tp that has a "third person" state space with a 180 camera angle offset. I am not sure how this rotation changes the state for the policy that uses states as input. One of the states of reacher is vector between goal and state which should still be informative enough to solve the task. Presumably the rotation creates a different view for the image based experiments. There also the transformation is quite easy for the spatial autoencoder given that it has access to coordinates through the spatial softmax layer. 

ii) It is unclear what the R2W task entails or how difficult it is. 

"The transfer task is writing letters as fast as possible. The transfer task differs from the alignment tasks in two key aspects: the end effector must draw a straight line from a letter’s vertex to vertex and minimally slow down at the vertices. "

It seems there is a term that penalizes the policy from doing things quickly. Some questions regarding this task:
1) How is this task specified? 
2) Is agent rewarded for slowing down too?  Does the goal of the reacher and the state representation corresponding to the vector between end effector and goal get updated once it reached a particular point? Does the agent have to write one particular letter? Why do the other baselines not work at all on this task?  

iii) Is it possible to run experiments on environments presented in Invariant Features paper so that the importance of CDIL can be assessed better.

4) "HIGH-LEVEL COMPARISON TO BASELINES" section is misleading. While the different baselines (TCN, IfO, TPIL) were developed in the context of view mismatch that does not mean the algorithm cannot be applied for embodiment mismatch. For IF also the same idea can be applied for viewpoint mismatch. TCN also proposed single-view version which can be used to learn representations without paired data. Hence, it is not right to say TCN can't work with unpaired alignment data. It also needs to be stated in this section that CDIL requires access to actions while some baselines like TCN do not need actions that can be used for training.  

Questions
1) How were the alignment videos generated?

Decision
This paper has the potential to be an important paper in this field. But at this point needs further empirical evaluation with stronger baselines and known benchmark environments.

Minor Comments:
1)  "boltmzman machine reconstruction error" - Boltzmann
 
References
[1] "Deep Reinforcement Learning That Matters." Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger
review_title: Official Blind Review #2

Meta-review: The authors propose a novel approach for imitation learning in settings where demonstrations are unaligned with the task (e.g., differ in terms of state and action space). The proposed approach consists of alignment and adaptation steps and theoretical insights are provided on whether given MDPs can be aligned. Reviewers were positive about the ideas presented in the paper, and several requests for clarification were well addressed by the authors during the rebuttal phase. Key evaluation issues remained unresolved. In particular, it was unclear to what degree performance differences were purely caused by issues in alignment, and reviewers did not see sufficient evidence to support claims about performance on the full cross domain learning setting.
Decision: Reject