Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposed Stabilized Likelihood-based Imitation Learning (SLIL) which iteratively estimates the expert state distribution by using Denoising Continuous Normalizing Flow (DCNF) and maximizes the policy learning objective in Eq. (3) that matches expert policy and state distribution. Compared to other baselines such as BC, DRIL, GAIL, SLIL was shown to find out better expert state distribution (in terms of seeking multiple modes, Earth Mover's Distance (EMD) while achieving the empirical return close to that of the expert (which is widely used in the imitation learning literature). The strength of using DCNF instead of using Continuous Normalizing Flow (CNF) is well desribed and supported by the experiments evaluating the test log likelihood in Figure 9. Authors also analyze SLIL's hyperparameter sensitivity that supports the claim on the SLIL's stability. 

main_review: Regarding the strength of this work, the contribution of this work over given baselines is clear, and I'm satisfied with the detailed explanations and experiments, e.g., comparison with LIL, the justification of using DCNF rather than CNF, how DCNF is trained algorithmically, quantitative tests using EMD and test log likelihoods, etc. 

Regarding the weakness of this work, I think some recent relevant works are missed, e.g., Dadashi et al., "Primal Wasserstein Imitation Learning", Kim et al., "Imitation with Neural Density Models", and some references therein. Although these papers didn't aim to solve the mode collapsing directly, their algorithms are much more sample-efficient than GAIL in terms of the number of expert trajectories and the number of environment interactions (i.e., GAIL is no more a SOTA algorithm as stated in this work), and both are relevant to this work in the sense that expert support estimation is used. Primal Wasserstein Imitation Learning (PWIL) shows their algorithm works efficiently even when we have a small number of expert trajectories (1 or 11 trajectories), whereas this work presented empirical results assuming more than 4 expert trajectories (more than 80 expert trajectories for Humanoid). In "Imitation with Neural Density Models", energy-based model is used to estimate expert support estimation, which are based on maximmum likelihood estimation similar to this work. 
summary_of_the_review: Overall, writing is clear, and the empirical results are shown to outperform GAIL and some other baselines. However, some SOTA algorithms (which I believe is relevant to this work) are missed, and it is unclear that the mode collapsing issue also happens for those relevant works or not. Although I agree that there is an algorithmic novelty for this work, I believe more elaborated results should be given with the aforementioned references.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach to the problem of state-action based imitation learning, in which an agent aims to solve a specific task given state-action expert trajectories. Their objective SLIL combines (or interpolates between) generative modeling and behavioral cloning in order to avoid mode collapse of the agent policy (common issue of adversarial approaches to IL like GAIL) and to mitigate distributional shift which BC typically suffers from. 

Contributions:
- An extension of continuous normalizing flows, coined denoising continuous normalising flows (DCNF), which is claimed to alleviate the manifold hypothesis, and is trained via maximum likelihood estimation.
- A strong imitation learning algorithm consisting in estimating the expert's state distribution, and learning a policy that recovers the expert policy and that has an occupancy distribution close to the distribution induced by generative model learned via the DCNF.

main_review: $\textbf{Strengths}$
- The approach is very well motivated, aiming to mitigate issues of both BC (distributional shift) and GAIL-like approaches (mode collapse and instability of minimax optimization)
- The algorithm performs very well on all benchmark environments, which are themselves distinct enough. 
- The ablations illustrate the robustness of the method to hyperparameter variations. 

$\textbf{Weaknesses}$
- In the description of the proposed approach to generative modeling, coined denoising continuous normalizing flows, there are a few points I do not understand, and where their may be some incorrect statements:
  - It is claimed $P_{E,\sigma}(\tilde{s}|s) = P_E(s)\mathcal{N}(\tilde{s}| s, \sigma^2I)$. I think the RHS is actually equal to the joint distribution  $P_{E,\sigma}(\tilde{s}, s)$ instead?
  - The statement that follows about the $0$-noise limit is also incorrect I thinkj, their should be some sort of marginalization in order to have an accurate proof?
  - In the algorithm, I think the noise scale in the Gaussians should be $\sigma_i^2$ instead of $\sigma_j^2$?


summary_of_the_review: I believe the paper proposes an interesting approach to imitation learning, however the technical issues described in the weaknesses section makes me currently tend towards a weak reject. If the authors were to successfully clarify these, I would consider raising my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a straightforward non-adversarially-trained imitation learning method. The goal is to avoid mode collapse and training instability that can occur in some adversarially-trained imitaiton learning methods, while still addressing the distribution shift problem that basic Behavior Cloning is subject to. 

The paper:
- Motivates a two-stage learning procedure by starting with a likelihood-based state-action distribution matching objective that would require bi-level optimization
- Presents a lower-bound of the objective in which optimization can occur serially of two components: MLE of the expert's vistation distribution, and then policy optimization over two terms, one of which is the LL of visited states under the the modelled visitation distribution, and the other is the behavior cloning loss.
- Presents a proof that it is a lower bound. 
- Proposes a concrete approach to performing this optimization problem by using a denoising conditional normalizing flow for modeling the expert's visitation distribution, where the CNF seems to be motivated because of its ability to model densities of complex distributions, and the denoising is motivated to enable stable training in situations in which the data lie in a lower-dimensional manifold. 
- Performs experiments on 10 mujoco environments that illustrate that the proposed approach achieves better mode coverage than some comparable methods, that the denoising is empirically effective for generalization, that the algorithm is more robust than GAIL to variation in learning rate, among other hyperparmaeter variations.
main_review: Main strength
--
This paper's main strength is that it proposes a straightforward method that both conceptually and empirically satisfies important desiderata of imitation learning methods (1) address the cascading error problem of BC-based methods (2) address the instability (mode collapse) of adversarially-trained IL methods (3) avoid dependence on an interactive demonstrator (i.e. DAGGER).

Main weakness
--
The paper's main weakness is some imprecision in a few places

### Imprecision
There are some issues (a,b,c) with the proof. (a) The proof refers to component $\enclose{circle}{2}$ in Eq 1, yet Eq. 1 has no denoted components. Is this reference correct? If so, then Eq 1. should denote the second component with an underbrace. If not, should it instead be Eq. 2 or Eq. 3? (b) The proof states "with sufficient expert data $\mathcal D_E$, $\enclose{circle}{2}$ can be expressed with $\sum_{s \in \mathcal D_\pi} \frac{P_E(s)}{P_\pi(s)} \log P_\pi(s)$. I don't see how this is true, because $\enclose{circle}{2}$ in Eq (3) is $\sum_{s \in \mathcal D_\pi} \log P_E(s)$, i.e. the state distribution is the state distribution of the policy. Again, is the reference to $\enclose{circle}{2}$ correct? These ambiguities / errors prevent me from being able to completely follow the proof. (c) It's not obvious from the definition of $g(\pi, P_\pi)$ that $\sum_{s \in \mathcal D_{\pi^*}} \log P_{\pi^*}(s) \geq \max_\pi \sum_{s \in \mathcal D_\pi} \log P_E(s)$ for two reasons (c.i) because the sums are not normalized by the the dataset size, it would be easy to invalidate the inequality by changing the dataset sizes (c.ii) What if $\pi$ was such that $\mathcal D_\pi$ only contains $s$ at the mode of $P_{\pi^*}(s)$? Wouldn't the RHS be at least as large as the LHS, in that case?

S4.1: $P_E$ is undefined (it's mentioned that it's the expert's state distribution, but a formal definition is needed). Is it more correctly denoted $P_{\pi_E}$, to "typecheck" with the subscripting of $P$ with a policy (i.e. $P_\pi$) to denote the state visitation distribution induced by policy $\pi$?

### Other issues
- The motivation of the gating parameter ($\lambda$) seems to come from dataset size imbalance (between $\mathcal D_E$ and $\mathcal D_{\pi_{\theta_i}}$). Why can't this dataset balance be addressed by simply normalizing the losses by the size of the datasets? Can the authors comment? If this is insufficient, the paper needs to discuss why.
- Missing related work: like the submitted paper, [A] also performs likelihood-based imitation learning with a normalizing flow for expert distribution matching, is not adversarially trained, and uses perturbation of the expert trajectories with gaussian noise. Discussion of the differences is important to contextualize the current paper's contributions, e.g. [A] used the distribution for planning rather than to learn a final policy
- The description of the experiment corresponding to Figs 11, 12, and 13 is lacking. It's not clear what environment(s) the experiment are run on.
- The introduction needs more justification for why DCNF is used in favor of other likelihood-based deep generative modeling approaches. Is continuity necessary? Is denoising necessary? Section 3 talks about the latter (C#2), perhaps incorporate some of this motivation into the introduction.

[A] Rhinehart N, McAllister R, Levine S. Deep imitative models for flexible inference, planning, and control. arXiv preprint arXiv:1810.06544. 2018 Oct 15.

summary_of_the_review: Despite some important issues with method presentation and justification, I think the paper presents a solid method and solid evidence of its utility. The paper can definitely be improved, but as it stands now, I think it's a good contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper present a novel method for imitation learning aiming to stabilize training and mode coverage compared to existing methods. The proposed method (SLIL) extends behavioral cloning by an additional objective of maximizing a state-only reward that is defined by a density estimate (trained once beforehand) of the expert's state distribution.
For density estimation SLIL uses a continuous normalizing flow where the expert states are corrupted by Gaussian noise, where the noise-level is decreased during training. The noise is introduced to tackle the "Manifold Hypothesis Challenge", that is, the inability of normalizing flows to accurate approximate distributions that lie on a lower-dimensional manifold. 

The main contributions are:
1. A novel imitation learning method that combines the BC loss with a RL loss for reaching expert states
2. A slightly modified approach to tackle the manifold hypothesis challenge (Kim et al. [2020] used Gaussian noise where the noise-level was sampled from a uniform distribution, independently for each data set, rather than using the same noise-level for all data and annealing it during optimization)
main_review: Strong Points
* The empirical results are good: SLIL achieves good results, no matter whether few demonstrations or many demonstrations are provided.
* The presentation is good, however, I have some suggestions below.

Weak Points
* Technical correctness (1): The main claim (Theorem 4.1) that SLIL tightly bounds the likelihood objective (LIL) is not supported. The SLIL objective differs by substituting the term $E_{p_\text{E}(s)}\big[ \log p_{\pi}(s) \big]$ with $E_{p_{\pi}(s)}\big[ \log p_{\text{E}}(s) \big]$, that is, switching the expert' and agent's state distribution. The proof has several issues: First of all, it starts with circular reasoning, by assuming that the optimal policy of the SLIL objective, $\pi^\star$,  matches the expert's state distribution. It then derives the bound (which is only valid for $\pi^\star$) that uses the equality $\max_x  f(x) + \max_x g(x) \ge \max_x \big[ f(x) + g(x) \big]$ which is not at all tight in general.
* Technical correctness (2): The SLIL objective is further modified by introducing a hyper-parameter that disables the RL objective ever $\lambda$ iterations, which effectively changes the objective being optimized without any theoretical justification.

* Novelty (1): SLIL is a hybrid between BC and Energy-Based Imitation Learning (EBIL) [2], a reference certainly missing. Like SLIL, State-only EBIL starts by estimating the expert's state distribution and uses it as a fixed reward function for imitation learning. As the SLIL loss is basically an interpolation between two prior imitation learning losses the contribution seems minor, in particular given that the objective of SLIL is not well motivated (Technical correctness (1)). It is also worth noting that AIRL uses noise contrastive estimation to estimate the expert policy and uses this density estimate directly as reward for MaxEnt-RL, which is similar to the 2nd term of the SLIL objective. The differences are thus that AIRL ues MaxEnt-RL instead of RL, $r=\log p_E(a|s)$ instead of $r=\log p_E(s)$, and NCE with an EBM for density estimation rather than maximum likelihood with normalizing flows. 

* Novelty (2): The "Denoising continuous normalizing flow" seems to be a minor variant of Softflow [Kim et al. 2020]. Furthermore, this minor contribution is not evaluated at all (neither Softflow, nor any other methods for handling the manifold hypothesis challenge were compared). 

* Evaluation:  The improved stability compared to GAIL was not demonstrated. Please add learning curves for the data of Fig. 10 to the Appendix (choosing a suitable number of demonstrations (e..g. 5) for each experiment).

Additional Comments
* It is not clear why the hyper-parameter lambda was introduced in favor of a fixed coefficient that trades off the two objectives, which would at least maintain a fixed objective function. 
* Much of 4.2 seems to be related work. I think it would make more sense to put it before the "Method" section to make the contributions more clear. In exchange, the problem formulation of LIL could be put under method, since it is introduced to "derive" the SLIL objective.
* Algorithmbox 1 is almost useless, because it mainly shows that we have an iterative training loop, but doesn't really show what happens at each iteration.
* End of page 5. The definition of $P_{E,\sigma}(\tilde{s}|s)$ looks like  $P_{E,\sigma}(\tilde{s},s)$ to me.

References:
[1] Liu, R., Gao, J., Zhang, J., Meng, D., & Lin, Z. (2021). Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. arXiv preprint arXiv:2101.11517.

[2] Liu, M., He, T., Xu, M., & Zhang, W. (2020). Energy-based imitation learning. arXiv preprint arXiv:2004.09395.
summary_of_the_review: The contributions seem rather incremental and the claims are not well-supported.
I still gave a relatively high recommendation, because the results are fine and the overall ideas of 1) tackling covariance shift of BC by using an RL objective to match the expert state distribution, and 2) estimating the expert state distribution once instead of iteratively based on NCE with adapted noise to improve stability, are reasonable.
review_title: null

Meta-review: The paper proposes the use of a state distribution estimation objective with a classic behavioral cloning objective for imitation learning. 
The submission also proposes the use of a continuous normalizing flow training technique coined "denoising normalizing flow" to learn the state distribution. The authors experimentally validate their method on several MuJoCo continuous control benchmarks.
The theorem 4.1 does validate the fact that this proposed objective is can be maximized by the target policy.
However, the technical contributions (proposal of new objective and the denoising normalizing flow method) are marginal compared to previous work (e.g., SoftFlow or Energy-Based Imitation Learning). 
The empirical validation is lacking more extensive comparison with PWIL or NDI, which are more recent methods attempting to address the challenges described in the submission. 
I'm recommending this paper for rejecting for this conference.
Decision: Reject