Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of imitation learning from observations only (no access to expert actions), in the setting where expert and agent are in different MDPs. More precisely, transition dynamics are different. The proposed approach consists in instantiating two agents, an advisor and a learner, such that the advisor is trained for its next state distribution to be close to the one of the expert, while the learner is trained so its state-action occupancy matches the one of the advisor.

The reason they instantiate this framework is that otherwise, next states may be unreachable by the agents hence the trained discriminator (if directly optimizing a GAIL loss between agent and expert occupancies) may perfectly classify and learning may then be challenging. Instead, in their setup the agent learns through a proxy living in the same MDP - the advisor.

Contributions:
- They introduce an algorithm aiming to mitigate the issue of agent and expert having dynamics mismatch
- Their demonstrate improved robustness to such mismatch in continuous control environments

main_review: $\textbf{Strengths}$:
- Empirically, they demonstrate their algorithm is more robust than baselines in the mismatch setting. 
- The approach is quite innovative
- The problem they tackle is very interesting, as in realistic scenarios, their will usually be such dynamics mismatch.

$\textbf{Weaknesses}$:
- I found the paper slightly hard to parse, i.e. I had a hard time understanding for instance why we could not directly leverage the advisor policy, and why the learner policy trained so its occupancy would match the advisor's would be better suited than the advisor's. It would help if this could be further clarified, maybe with an intuitive visual example showing the expert, the advisor and the learner policies?

summary_of_the_review: I find the paper in this current form to be borderline due to the fact that it is hard to parse and to understand why the framework is instantiated in the way it is. It is still an interesting contributions, and empirical results are promising. Upon such clarifications to be made (along with potentially visual examples providing intuitions), I would be considering raising my score to a weak accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles imitation learning from observations under model mismatch.  This is a realistic consideration when tackling more complex or real environments where methods such as IL (and its variants) may see dramatic improvements over more conventional RL-based approaches (to the point of being the only feasible approach).  Model mismatch makes existing ILO methods unstable (in theory, at the very least).  To ameliorate this, the paper introduces an _advisor_ policy, which learns from the expert demonstrations but under the constraints implicitly imposed by the capacity of the learner or the learners dynamics.  The advisor is then used to train the learner.  The method is empirically validated on some standard benchmarks and appears to perform reasonably well.   

main_review: ### Summary++:
The setup for this paper is super interesting.  The domain is a realistic one: there is some limited data collected by simply watching an expert interact with the environment, but, where the learner is different in some critical way.  The question is then how do we leverage the observations of the expert to expedite learning an agent.  If solved generally (see 5 + 6 below) this would be a brilliant addition to the literature. 

Sections two and four are _very_ well written.  Section one motivates the problem well, but doesn’t provide much detail on what content will be provided in the paper.  Section three is not well written, does not clearly explain the method, and is instead a bit of a drawn-out narrative that is hard to follow.  The empirical validation presented is lacking.  Finally, there is only a very short supplement that only provides hyperparameter / architectural settings and no further insight.  

The crux of my critique is that I believe the empirical validation, to justify a method with this many moving parts, is lacking; and that the clumsy exposition of the methodology obfuscates why each component is necessary, and how they work in conjunction to create a performant policy.  While I believe there is a good method and a good conference submission in this line of work, it is not currently in that state.  

### Major comments:
1 - My main complaint with this work is the lack of clarity in how the advisor actually works and what it provides.  The advisor is this magic box that encapsulates the expert policy _and_ the constraints/limitations of the learner.  The advisor is defined in learner space (as that is the only space we tractably have access to) and so how is the policy learned by the learner different to that of the advisor?  Why can’t the learner just imitate the advisor exactly (using, for instance, DAgger)?  If the learner can imitate the advisor exactly, then what is the point of the advisor?  Why not just learn the learner by whatever method you use to learn the advisor?  It is then also poorly explained how the learner is subsequently recovered from the expert demonstrations and the advisor policy.  Eq (2) is pointed to, but without much more detail as to the intuition that underlies this step.  

2 - The method itself is also _very_ complicated.  As far as I can tell, there are at least four trained elements: the RED network, a reward network, an advisor policy and a learner policy.  That is a lot of stuff to specify, train, tune etc.  There is also no indication of how robust to hyperparameter/architecture settings the performance is.  i.e. If I happen to use a RED network with too-lower capacity, how is performance affected and how easy is it to diagnose that that is where the problem is?  I understand that all things RL are inherently a bit magic-boxey and can be sensitive, but even still, this method seems very convoluted/involved and is not sufficiently empirically validated to convince me that this complexity is warranted.

3 - Following on from 1 & 2, is the advisor and/or RED even strictly required?  It seems, and I may have gotten it wrong here, that you are essentially pre-training some policies to match the expert as well as it can, and then refining the policies in places where the expert can’t be matched.  I can just about believe that RED provides a tractable method for learning a policy that keeps as close to the experts state evolution as possible.  The advisor policy then steers the learner towards those actions?  Why not just weight the ILO loss by the prediction from RED?  Or just initialize the learner to match the state distribution where it can, and then refine based on reward?  Or something more akin to THOR [Sun, Bagnel & Boots, 2018]?  If the authors can comment on and clarify this.  

4 - I will say upfront that this may be a little unfair as a critique, so take this with a pinch of salt.  The absolute performance of the method (and even the baselines) is disappointing.  Although I don’t know this for sure, I am relatively confident that something like regular SAC applied directly to the environment would achieve better performance in fewer environment interactions.  Therefore, why are we even bothering to use the expert demonstrations if regular RL in the learner could achieve competitive performance?  I was desperately hoping that this method would leverage the “imperfect” demonstrations to obtain a learner policy with dramatically fewer environment interactions, even if the absolute/final performance was slightly lower than the best-possible performance.  As I mentioned above, there is a TON of internal mechanics in this method, and while I believe there is an algorithm that can do ILO in this mismatched domain, I am as of yet unconvinced that the AILO method presented is _the_ method for doing it -- both in terms of absolute performance and technical overhead.  

4.1 - The results in this paper have not convinced me that this method successfully, expediently and robustly extracts information from demonstrations from a slightly different environment to expedite learning a policy.  

4.2 - I think the following “baselines” are missing:  
- SAC/RL on the learner (also to provide the true upper bound on performance of the learner).
- DAgger on the learner using the misspecified expert policy. 
- Behavioural cloning (BC) on just state-actions from the expert.  
- Initializing a learner to the BC learned-policy and then performing RL to refine that policy. 

Some of these are not practically applicable methods in real-life given the domain, but are, in my opinion, required to elucidate the performance of the method (bearing in mind that baselines are also there to help the _reader understand_ the performance and limitations of the method, as opposed to simply whether or not it works).  

5 - I would also have liked to have seen the method applied to a smaller-scale toy example.  You even motivate the problem with a simple gridworld example -- why wasn’t the method applied to such a gridworld?  A simple example would help you pull apart the method and verify that all the components are working as intended, and would also allow instructive and intuitive visuals, graphs, diagrams, heatmaps etc to be generated to further help the reader _understand_ the method and its components, as opposed to simply posting a grid if graphs showing that it achieves half-way reasonable performance.  

6 - I think there may also be a missed opportunity exploring (at least in principle) connecting IL and sim-to-real domains.  Sim-to-real seems to me to be a domain where this method (with some tweaking) could have some utility, as that is entirely derived from mismatch between two domains.  The advisor seems in-part to be a more flexible representation of an expert policy (learned in sim) that accommodates for the imperfections of the learner (learned at test-time in real).  There may be nothing in this, and it is just a suggestion, but the setups are so similar that _any_ comparison is conspicuous by absence -- even just commenting and discounting.  

### Minor comments:

A - I would like to see a more specific explanation of the method earlier in the paper.  The advisor is loosely mooted early on, but is only really introduced with any specificity at the bottom of page 3.

B - Only proper nouns should be capitalized (“Reinforcement Learning” -> “reinforcement learning”;  “infinite horizon Markov Decision Process” -> “infinite horizon Markov decision process”).

C - Figure 1 should be improved to improve its clarity.  I have no idea what the “squiggly” lines represent.  Are these lines something we will learn?  The second part of that figure also offers no helpful content as it is too high-level.  I would also move it up to be a banner figure at the top of page two.  This would help frame your approach more succinctly and earlier on.  I would also consider (even more simply) pseudocoding the algorithm and placing it alongside the diagram on page two to drive home the method early. 

D - Figure 2 could easily be compressed to save space if required.

E - The algorithm block is totally unhelpful.  It should either be made more precise using full mathematical notation, or, made more intuitive using pure pseudocode and high-level objects.

F - Releasing code for this is ultra important because I do not believe I could reproduce the results in this work from the prose and limited supplement alone.

summary_of_the_review: This work is interesting and shows promise, and if it were to work, would be a useful addition to the literature.  However, I am unconvinced by the method itself.  The method is quite involved, with lots of moving parts.  Furthermore, the presentation of the core of the method (the advisor) is remarkably poor, considering that it is _the_ contribution and that the rest of the paper is so well written.  The empirical validation is also a little lacklustre, both in terms of range and actual performance.  I also have more subjective issues with aspects of the paper (see comments above) that, while it would be unfair to reject solely on those issues, dissuade me from giving the authors the benefit of the doubt here.  Therefore, I recommend that this paper is not accepted for inclusion in this review cycle;  but note that with some reworking this paper could make a good submission to a good conference in the near future.  

Good luck.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new algorithm for solving LfO task where the dynamics are different in the expert environment and the learner environment. The algorithm first trains an intermediate policy in the learner environment that mimics the expert's state trajectory and then the learner can just mimic this intermediate policy in the learner environment. The authors further show in the experiment that the proposed method outperforms previous LfO algorithms in several locomotion tasks. 
main_review: The idea of using RED for estimating the joint density for consecutive states for imitation learning from observation is interesting, and the overall structure of the paper is clear and easy to follow. 

Eq. (3) seems confusing: $J(\pi_a)$ is a function of $\pi_a$, but on the RHS of the equation it is taking a maximum over $\pi_a$. I am not sure what are taking maximum over since $\pi_a$ should be fixed in this situation as a input of $J(\pi_a)$. Also, the last equation does not hold if it is taking $\max$, it only hold if it is taking $\arg\max$. I do understand what the authors are trying to show here but this may need some revision. Similarly to Eq. (4).

Also the usage of the intermediate policy still lacks some intuition: if the advisor policy is able to recover the expert state trajectory, then optimizing the expected return s.t. the state distributions are the same should solve the problem. If the advisor policy is not able to fully recover the expert state trajectory, how would we expect the learner policy to recover the state distribution of the expert policy since the learner policy is matching the state-action distribution of the advisor policy (Eq. above Eq. (2))? For example, how is the learning path (the squiggly line) in Fig. 1 achieved by the proposed objectives?

The setting itself is not new: learning from expert and environments with different dynamics has also been proposed in many papers with different names: policy transfer, policy adaptation, domain transfer, sim2real, etc.

The main major issue is the experiment: the current baselines are weak since they are just LfO methods but do not specifically deal with dynamics mismatch. Even in this situation the proposed method is outperformed or evenly matched by the baselines. This is not the main issue, the main issue is the lack of comparison with baselines under the same setting as the proposed method, such as [1]. (Possibly, plus works in the above papers (policy transfer, policy adaptation, domain transfer, sim2real, ...) that work in the same setting.) Only comparing with baselines that do not specifically deal with dynamics mismatch does not seem very fair, and thus the experiment is not thorough enough for such an empirical paper. 

[1] Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. arXiv
preprint arXiv:2002.11879, 2020.
summary_of_the_review: Some mathematical representations still need some improvement. The experiment is not convincing enough. I would recommend a weak reject for the paper.
review_title: null

Meta-review: The submitted paper considers the very interesting problem of imitation learning from observations under transition model disparity. The reviewers recommended 2x weak accept and 1x weak reject for the paper. Main concerns about the paper regarded clarity of the presentation, complicatedness of the proposed method, and experimental validation. During the discussion phase, the authors addressed some of the comments and provided an update of the paper providing additional details. While some of the reviewers' concerns still stand, I think the addressed problem is very relevant and the proposed method can be (with clarifications and improvements of the presentation) be interesting to parts of the community. Hence I am recommending acceptance of the paper. Nevertheless, I strongly urge the authors to carefully revise their paper, and taking the reviewers' concerns carefully into account when preparing the camera ready version of the paper.
Decision: Accept (Poster)