Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes an algorithm for imitation of expert demonstrations, in situations where the imitator is acting under a different environment (different dynamics, for instance) than the one used to collect expert demonstrations. The algorithm builds on GAIL with the following modifications – the discriminator is made dynamics-invariant by adding a domain-adversarial loss, and the policy is made to condition on a dynamics context. A separate dynamics posterior network is trained (either supervised or unsupervised) to predict this context at test-time. 


I have the following concerns about the paper:

1.	Lack of novelty: 
         a.	Learning a dynamics-invariant discriminator with the gradient-reversal-layer was proposed in Stadie et. al (2017). How is this approach different? In particular, what is the new element in Figure 1. and complete Section 3.4?  
         b.	Learning a posterior network to predict context codes, and conditioning the policy on those was explored in papers such DIAYN and InfoGAIL. 

2.	The proposed algorithm is reliant on the possibility of being able to sample from a distribution of environments (with varying dynamics), and then collect many trajectories in that environment (Line 6-7 in Algorithm 1). This is a severe requirement for real-word scenarios, and somewhat antithetic to the robotics learning motivation given by the authors in the introduction. Moreover, Figure 7 seems to imply that the method doesn’t generalize well to unseen environments, if enough environments can’t be sampled during training time.


Minor comment:

Figure 8. Friction value should not go from -3 to 3. Also, this single result inspires no confidence in the benefit or general applicability of the VAE-based context prediction.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary:
The submission considers the problem of imitation learning when the dynamics of the expert are not known to the agent and the dynamics of the agent may change frequently. It is however assumed that the agent has access to a parameterized simulator that can simulate the expert dynamics. The parameters for the simulator are not known but are assumed to be drawn from a known distribution.
The proposed method is based on GAIL but uses several modifications:
- A contextual policy is trained that also takes the dynamics-parameters as additional input. At each iteration, a new environment is sampled for performing the policy-rollout.
- A "posterior" prediction network is trained to maximize the likelihood of the parameters that were used for the different roll-outs. This network is used for the test case, where the true dynamics of the agent are not known.
- The discriminator might use features of the state-action input that correlate with the corresponding dynamics. Classifying based on such features may be undesirable because the discriminator might no longer produce useful rewards. In order to address this problem, an additional head is added to the discriminator that outputs a prediction of the dynamic parameters. The prediction error is trained by backpropagation, however by flipping the sign of the gradient at the last shared layer, the features of the discriminator are optimized to be unsuited for predicting the dynamic parameters (the technique is known as Gradient Reversal Layer).
- A VAE-based method for learning latent dynamic parameters is proposed, by training a conditional VAE to reconstruct the next state, where the current state and action are provided as context to the encoder and decoder.

Contribution:
One of the strong-points of the submission is the fact that it features several different, orthogonal contributions. I also think that the considered problem setting is relatively interesting. However, also when considering real applications such as robotics, I am not convinced that explicitly modeling the dynamic changes is necessary. Some existing imitation learning methods focus on a setting where the dynamics of the expert may differ from the agent, but the dynamics of agents do not change. This setting does not require dynamic-contextual policies and seems to be applicable to typical robot applications.

Soundness:
The different components of the proposed methods seem reasonable to me. They do not come with (and arguably do not require) new derivations but seem rather like pragmatic solutions for the encountered problem.
The optimization problem (Eq.2) seems to be formulated slightly inaccurate, because the last term should in my opinion not depend on theta. If I understand correctly, the policy should not maximize the likelihood of the dynamics posterior.
The contrastive regularization loss needs to be better motivated. A high KL in the first term may not necessary be bad, for example, if the confidence in the prediction of (s_0, a_0, s'_0) is lower compared to (s_1, a_1, s'_1). If a similar regularizer has been used in prior work, such work should be referenced. Otherwise, it needs to be motivated.

Presentation/Clarity:
The presentation of the work is arguably the main weakness of the paper.
The submission does not seem polished. It contains a large amount of typos. Figure 2 is confusing and adds little compared to the text description. Also the structure could be improved. For example, the submission introduces the posterior loss and outlines the algorithm before describing the individual components.
The paper uses some techniques such as conditional VAEs [1] or contextual policy search [2]
are used but not described / referenced.

Evaluation:
I like that the different aspects of the proposed method are also evaluated individually. The ablations with respect to the adaptability and GRL are crucial. The evaluation of the performance could be improved. PPO and UP-True use the true reward function, so the only real competitor is a naive GAIL-baseline that uses randomized dynamics during training. I'm not aware of prior work that considers the exact same setting as the manuscript. However, one of the main arguments for inverse reinforcement learning is the claimed generalizability of a reward function as opposed to a policy. I see that learning a new policy after each change in the dynamics may be too costly in some settings. However, comparisons to methods such as AIRL that aim to learn reward functions that are robust to changes in the dynamics would be highly interesting.

[1] Sohn Kihyuk, Honglak Lee, and Xinchen Yan. “Learning Structured Output Representation using Deep Conditional Generative Models.” Advances in Neural Information Processing Systems. 2015.
[2] Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters.  A survey on policy search forrobotics.Foundations and Trends in Robotics, 2(1–2):328–373, 2013.

Comparison: How about IRL, e.g. AIRL?
What about state-only GAIL?

Typos:
Equations are not properly integrated into the sentences (missing punctuations)
"domains, as oppose to one domain."
"Inspired by to GANs"
"and generates a environment"
Figure 5a (legend): "dyanmics"
"that can generalized across"
Algorithmbox: "A environment", "and Generate environment"
"is achieved through 1) allowing"
"the policy is mainly concerned with the end-effector of the latent parameters"


Question: 
What are the network architectures?

According to line 10 of the algorithmbox only the current trajectory is used for updating the dynamics posterior. Why?
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper describes an approach that combines domain adversarial neural network with generative adversarial imitation learning. In the setup, each environment is dependent on some latent context variable (e.g. physics parameters) through which latent variable dependent policy and latent variable independent discriminators are learned. 

I don't think the exact same idea has appeared in existing literature. The authors justifies its connections and differences between third person imitation learning, but it seems that the proposed method bear some similarities to the NeurIPS19 papers below. 

https://arxiv.org/pdf/1909.09314.pdf
https://drive.google.com/file/d/1urPE7J5tT8dzoBHSFvZKwNLsQieU706o/view

The following papers also assumed that GAIL-like methods in a meta learning setup, where environments depend on context. Perhaps the difference here is that the discriminator is also trained with a gradient reversal layer, so it encourages the discriminator to not use redundant state information. Also in this paper the source domain only contains demos from one env, which might highlight the importance of the gradient reversal layer.


Questions:

In "dynamics learning", it seems that the inference network learns is mostly the context variable c, wonder if it is better to use terms like "latent variable inference" to avoid confusion.

What does the standard deviation mean in Figure 6? It seems a lot of them are even larger than the mean.

There is little explanation to the VAE-ADAIL experiments -- is it safe to assume most of the experiments require certain knowledge of the latent variable in order to be successful? Maybe some of the arguments about VAE can go to the appendix.

Why would in some cases UP-True performs much worse than ADAIL? In Ant it is even worse than PPO expert.

How would you adapt to unseen environments in ADAIL-pred? I don't see explanations in the text about how this is done. Essentially, how are samples obtained from the environment in order to perform posterior inference?

review_title: Official Blind Review #2

Meta-review: This paper extends adversarial imitation learning to an adaptive setting where environment dynamics change frequently. The authors propose a novel approach with pragmatic design choices to address the challenges that arise in this setting. Several questions and requests for clarification were addressed during the reviewing phase. The paper remains borderline after the rebuttal. Remaining concerns include the size of the algorithmic or conceptual contribution of the paper.
Decision: Reject