Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces MobILE, an approach for training agents to imitate from expert states only, i.e., when actions are unavailable. In particular, the approach utilizes an exploration bonus to more efficiently explore within the environment. The approach is compared to standard imitation learning approaches within OpenAI gym tasks. 

main_review: Clarity: Overall, I found the paper to be clearly motivated and mostly well-written, though I found section 2.1 to be somewhat confusing.

Originality: This approach introduces using exploration with an imitation from observation approach. Adversarial methods have been used for training both Imitation Learning (IL) and Imitation from Observation (IfO). The main contribution is using an additional exploration bonus to train an IfO method to tradeoff exploration and imitation.

Quality: I would have liked to see more ablations on each individual contribution of the paper. For example, the original GAIL-style training compared to the IPM method used in the paper (without bonus) and GAIL with the exploration bonus compared to MobILE. If this would not be straightforward then other exploration methods could perhaps also be considered. 

In general, I am unsure if this work is a contribution to the IfO or IL community, particularly because the forward-dynamics model is learned online and hence does not seem specific to training IfO methods. It would be great if the authors could comment on how this approach is a contribution to IfO.

Significance: The reason I am leaning towards rejecting the paper is that I am unsure of the significance of the approach. It seems the main contribution is to utilize a better exploration approach to imitation from observation than random exploration. This could be useful, as exploration has not been well-studied in the imitation from observation literature. However, the work is not compared to any other exploration methods. Since the exploration bonus is learned from data captured online, other intrinsic exploration bonuses used for RL, such as curiosity [3] or RND [4] could have been evaluated. The bonus used in the paper is in fact quite similar to that used in [3] and so it would be good to at least include it in the related work section. Additionally, it is unclear how much of a contribution using ensembles for the bonus is given the paper mentions having more than two ensembles did not lead to any further gains. 

Another problem is that the experiments are somewhat weak. The approach is only compared to a single IfO approach, BCO, but there have been several advances since this work was introduced. While the paper discusses other such approaches such as FAIL [5] and ILPO [6], the argument is that these approaches only work in environments with discrete actions, but MobILE also works with discrete actions and so it should be reasonable to compare in these settings.

Comments:

- The adversarial setting (i.e. the purpose of discriminators) should be more clearly and formally explained. 
- Could the authors further discuss why a forward dynamics model is better defined than an inverse dynamics model? If the dynamics are stochastic for example then $\Vert f(s,a) - s’ \Vert$ $f$ will just learn to predict the expected $s’$ rather than a deterministic example. But this is also the case in $\Vert I(s,s’) - a \Vert$ if there is not a unique $a$. I.e. it will just compute the expected $a$ rather than a deterministic one. Practically, I do not imagine many environments where this would be much of an issue. 
- Why is BCO constant in the figures? It also has an online stage where the inverse dynamics model is being learned and getting increasingly better experiences to train the bc policy. How were the dynamics models trained? 
- I was surprised to see that the approach performs better than GAIL, since that method has ground-truth actions. Could the authors discuss this result in more detail?
- Was it necessary to scale/bound the outputs of the exploration bonus? 
- The graphs should include tick marks on the plots (rather than only color) for better accessibility

[1] Generative Adversarial Imitation Learning. Ho et al.

[2] Generative Adversarial Imitation Learning from Observation. Torabi et al.

[3] Curiosity-driven Exploration by Self-supervised Prediction. Pathak et al.

[4] Exploration by Random Network Distillation. Burda et al.

[5] Provably Efficient Imitation Learning from Observation Alone. Sun et al. 

[6] Imitating Latent Policies from Observation. Edwards et al.
limitations_and_societal_impact: The authors discuss some potential societal impacts, particularly that it is import to ensure the quality of expert policies.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies a model-based approach to imitation learning from observation (ILFO). The main idea is to learn a forward model as well as ‘strategic’ exploration which is based on the concept of optimism in face of uncertainty. The proposed algorithm, MobILE, combines (i) model-based RL (forward dynamics learning), (ii) exploration based on uncertainty (maximum disagreement of model), and (iii) imitation learning. More specifically, models and the novelty-bonus function learned from online interaction with the environment can be plugged in to optimize the IPM imitation learning objective, where the bonus term incorporated into the discriminator would diminish the discriminator’s power for novel states. This algorithm features an ability to to trade off exploration and exploitation(imitation), and achieves good results on standard gym continuous control benchmarks (as good as imitation learning with action data available) despite the challenging ILFO setting.
main_review: Overall, I think this work is a well-written paper with solid theoretical contribution and strong empirical result. The main contribution is to present a novel combination of model-based imitation learning and exploration (uncertainty and optimism), and to show theoretical optimality bound as well as empirical validation. Given its novelty and contribution, I am slightly leaning towards recommending acceptance at this point.

Significance: The proposed approach seems to be designed with principle and makes sense. The MobILE framework has some good advantages compared to existing work such as FAIL and ILPO, such as it is compatible with continuous control and does not require much cost function tuning. Actually the practical algorithm is quite a straightforward combination of model/exploration/GAIL component, but some theoretical insights provided would be a novel and significant result within the ILFO setup. Although I was unable to examine and validate all theoretical results and proofs, the theoretical results on optimality bound seems significant and important. It is interesting to see that the difficulty of ILFO over standard IL is formalized through the exponential gap, which is also a good theoretical contribution. 

Regarding the exploration bonus $b$, don’t you need to have a weight (either as a hyperparameter or a learnable variable) to balance the impact of exploration? It is obvious that such exploration terms can improve the algorithm’s performance (as shown in Section 6.2), but how important are they and to what extent do they contribute to the imitation learning?

Regarding experiments, one concern I have is that a direct comparison to sensible baselines (GAIfO [64]) is lacking. Although it is reported that GAIL outperfoms GAIfO in all tasks (which is very expected), it would have been much better to have a direct comparison between MobILE and GAIfO or other baseline. For example, is exploration mostly accountable for the performance improvement under the same “learning from observation” setting? Without optimism (i.e. just model-based GAIfO) would it still work better or comparably to those LfO approaches?

A minor question) why does BC-O perform much better than BC in CartPole? Is it just a mistake? In L293, it is said that BC outperforms BC-O in all benchmark tasks.


Writing/Clarity:
- Motivation of the problem is well-written, and discussion of related work looks comprehensive and informative.
- With its theoretical contribution in mind, the paper is overall a bit difficult to follow. As a reviewer and for readers who do not have much expertise on imitation learning and theoretical RL, I wish there were a more kind background or preliminaries section (around section 2. Setting) to review the GAIL and IPM setup for the sake of self-containedness and more clarity.
- Naming of the problem: I think LfO is a more common, popular, established abbreviation to denote learning from observation. In that regard, why don’t authors use the term ILfO instead of ILFO? I also suggest the authors to refer GAIFO as “GAIfO” as per the original paper [64].
- L107: It would be good to write down the optimization objective that includes $c$ (the cost function) for better clarity and how it connects to the IPM objective (L117). Given that $V^\pi$ is the expected total cost under $c$, it’d be good to have a more formal and clear definition.


limitations_and_societal_impact: Section 7 discusses limitation of the work a little bit, but more detailed or comprehensive discussion would be also worth considering. There is concerns or issues regarding societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies in the setting of Imitation Learning from Observations (ILFO).
The contribution is proposing a framework for policy learning with uncertainty estimates by dynamics model learning, which leads to better balanced exploration and imitation.

main_review: 
The paper proposes a framework for imitation learning from observations, MobLE incorporating policy learning with exploration bonus by learned dynamics model. The framework of MobILE seems reasonable for the motivation and general one, however, there seem to be some concerns on the discussion of the motivation of this study, and explanation of the method and experiments.

## Pros
* Proposing a unified framework for ILFO with a combination of policy learning, exploration bonus measured with a learned model ensemble. The combination is novel in my understanding and potentially applicable to other algorithm choices.
* Giving a mathematical explanation for why the proposed exploration strategy is more efficient than random (or no) exploration.

## Concerns
### Discussion on the applicability of the work
The key idea of MobILE is dependent on exploration bonus with disagreement between learned dynamics model ensemble. I have questions about how this bonus is generalizable to other settings (e.g. input) than the ones shown in the experiments.

The experiments in the paper are performed on “state” input, and the learned policies are tested on the same robot (MDP) as the expert. For me, the assumption of no access to the actions of an expert is tricky and overly simplified in this case, since we have access to actions of the learned policy at the same time. Giving the possible situations where the assumption holds will make the paper more convincing.

Moreover, in the experiments in Section 6, the bonus is measured with L2 norm of state space, however, in the practical applications, the estimate seems not straightforward. For example, learning from video experts is a promising application, since videos of performing a task are easier to obtain. But how can we measure the bonuses if we use videos as the experts without actions? There is some literature on imitation learning from video experts [1-4], and is it possible to incorporate MobiLE framework with these visual methods?

### Comparison with other ILFO methods
In the experiments, the proposed method is compared against BC, BC-O, and GAIL. In these methods, only BC-O is ILFO method except for the proposed one. However, there are some previous methods in ILFO with different characteristics as mentioned in Section 1.1. What is the empirical difference between the proposed framework and the previous one (e.g. estimating inverse dynamics)? Adding comparison with other ILFO methods will convince the reader how exploration in the MobLE framework is promising.

### Number of the models in the experiment and the formulation of uncertainty (L165)
In L165, the uncertainty is defined with "sup" of L2 norm between two different models from possible pairs in the ensemble (termed as “maximum disagreement” in L219). However, the experiments shown in the paper are performed only with two dynamics models (the uncertainty is purely measured with discrepancy of next state predictions between two models). So, there is a mismatch between the formulation and experiment. L276-277 mentions that using more than two models does not improve performance much, and I suppose these results should be included in Appendix D for justification of the formulation of taking “sup” of mismatches between pairs of models.

## Additional Comments
### Model learning for optimizing the policy
In the proposed algorithm, the dynamics model learning is used only for calculating the exploration bonus, which I feel is a bit costly for its computation, especially in settings with high-dimensional states. Is it possible to use them to boost policy learning as well, like [5-6]?

### Minor correction
* L298: BC-0 -> BC-O


[1] K. Schmeckpeper, et al. Reinforcement Learning with Videos: Combining Offline Observations with Interaction. https://arxiv.org/abs/2011.06507

[2] A. Edwards, et al. Imitating Latent Policies from Observation. https://arxiv.org/abs/1805.07914

[3] K. Schmeckpeper, et al. Learning Predictive Models From Observation and Interaction. https://arxiv.org/abs/1912.12773

[4] M. Chang, et al. Semantic Visual Navigation by Watching YouTube Videos. https://arxiv.org/abs/2006.10034

[5] S. Sutton, et al. (1991). ​​Dyna, an integrated architecture for learning, planning, and reacting.

[6] T. Kurutach, et al. Model-Ensemble Trust-Region Policy Optimization. 
https://arxiv.org/abs/1802.10592

limitations_and_societal_impact: This paper tackles sample inefficiency in ILFO settings, which is a key problem in the previous studies. However, as mentioned in the main review, the paper does not discuss well on its applicability, which potentially could be the limitation of the proposed method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates the problem of imitation learning from observation alone (ILFO). The authors leverage the idea of optimism in the face of uncertainty and propose a novel model-based framework for ILFO that can handle both discrete and continuous settings. The authors provide a regret bound of the framework and show the exponential gap between IL and ILFO and the benefit of uncertainty-based exploration. The authors further corroborate the theoretical and algorithmic results with experiments in simulation. 
main_review: Strengths:

Although the idea of optimism in the face of uncertainty is not new, its application in ILFO seems noval and inspiring. The framework that the authors develop is general enough to work under both discrete and continuous settings, and it does not require the existence of an inverse dynamics model which is required by some of the previous works. In terms of theoretical findings, the authors show an exponential gap between IL and ILFO in the regret bound with respect to the number of actions, which is interesting. In addition, the authors theoretically justify the benefits of the confidence-based exploration. Furthermore, the experimental in simulation results look encouraging. 

Comments and questions:

I appreciated that the authors are explicit about the realizability assumption on cost and transition functions. However it may be strong in practical applications. Can the current analysis be extended to lift the assumption in a straightforward way? 

In Theorem 3, the rate applies to the best policy among the T polices. However, in Algorithm 1 the last policy is returned. Is it guaranteed that Algorithm 1 will always converge?

How’s the dependence on horizon H of the proposed algorithm compared to other existing ILFO algorithms including FAIL?
In Line 6 of Algorithm, does it mean that N expert states are required in each of the T iterations? 

The paper is easy to follow in general. However, I think there’s still room for improvement. 
- I did not find where regret (mentioned in e.g. Page 2 Line 47) is formally defined or motivated in the paper before it appears implicitly late in the paper -- Theorem 3.
- I don’t quite get the definition of Information Gain in Eq. (3). Does it assume how the dynamics model is learned? As the alg can be arbitrary, can I choose an algorithm that does not learn the dynamics so that \sigma_t does not decrease?
- Although Section 4.2 is one of the major contributions of the paper, it is a bit short and dense. For example, Line 254 - 256, a three-line sentence, is a bit hard to parse. Maybe to make room for it, some discussions on the different examples can be moved to the appendix. 
- In Page 3 Line 103, it seems that “query the expert” is a stronger assumption than what actually is required: access to expert state trajectories

limitations_and_societal_impact: yes
review_title: null

Meta-review: I appreciate the authors for detailed rebuttal and additional experimental results, and the reviewers for engaging in detailed feedback (especially reviewers axQZ and eB6p for active discussions). The novelty and impact were better communicated after the discussions. I like that the paper studies a generic problem of ILFO from model-based perspectives, discusses its fundamental differences from IL theoretically, and proposes better strategic exploration through models as a practical solution. One concern is some missing references/baselines for model-based GAIL/GAIFO-like algorithms, e.g. MAIL [Baram et al. 2016], but MobLIE ablation study without optimism appears to cover similar cases.
Decision: Accept (Poster)