Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a decoupled policy approach to state-only imitation learning, wherein the policy is decomposed into a next-state prediction network, which aims to predict the desired next state given the current state under the expert policy, and an inverse dynamics module, which given the current state and the desired next state, predicts the action. The utility of this split is that it allows for additional supervision during policy optimization, as the state transition network can be trained to match the observed expert transitions, and the inverse dynamics module can be trained on the transitions in the trajectories obtained during policy optimization. These supervised losses are placed on top of a standard policy gradient method, as gradients to both components of the decoupled policy can also be obtained through a standard likelihood-ratio policy gradient applied to a reward function corresponding to likelihood of the executed state-transitions being drawn from the expert policy. This reward function is obtained by jointly training a discriminator to classify sample transitions from the learned policy from those from the expert data.

main_review: Overall, this presents a novel approach to state-only imitation learning, which, to the best of my knowledge seems to be different from prior work primarily due to the fact that it combines state-only adversarial imitation via policy gradient (as in GAIfO) with a decoupled policy structure which allows additional supervision to each component separately. Intuitively this makes sense, and in the experiments the authors show that the approach can achieve better performance than baselines. I also appreciated the comparisons with GAIfO applied to the same decoupled network policy to show that the improved performance comes from the additional supervision, and not just the network architecture. I also appreciated the system diagrams showing both the overall framework and the regularization schemes.

The paper however also has some weaknesses in its clarity and experimental evaluation.

First, I found the organization of the paper to be confusing. The authors spend a large portion of the paper discussing the fact that the solution to the SOIL optimization problem may not be unique. While I agree with this analysis, I do not agree that this lack of uniqueness should pose problems for optimization -- gradient based optimization works well even if there might be a family of equally performing policies. The notion of "optimality" stays the same, as the authors aren't arguing that the objective of SOIL should be reconsidered. Instead, the purpose of the paper is to argue for an parameterization of the policy which takes into account the non-uniqueness of the solution. 
For this reason, section 3.1 felt superfluous: The end result (equation 4) is a fairly direct conclusion from the problem statement of SOIL.

In contrast, sections 3.3 and 3.4 felt lacking in detail. In the discussion of the multi-step optimization, it was unclear why certain gradients were dropped -- the reasoning "we already have an accurate input at each time step and for each training step, we only update the first one" was not clear. Furthermore, in equation 15, it is unclear how the Q function is estimated, and also how the integral over $s'$ is computed in practice. The details in the appendix also do not provide this information. In 3.4, it is unclear what it means to "directly optimize $\mathcal{L}^\pi_{\phi,\psi}$ instead of iterative training." Also the authors state in the appendix that pre-training does not help much, and is can sometimes hurt performance, and indeed, only a few environments use a pretraining step. This should be made clear in the body of the paper, as it seems to be an important factor in performance.

The algorithm seems to be designed for deterministic environments: this is an assumption in the theorems, and the MSE-based regularization schemes do not take into account stochasticity in the dynamics or policy. While the approach seems to be effective on NGSIM environment, which is stochastic, it is evident from the appendix that neither form of regularization was used. The authors should be more transparent about this limitation of their work in the body of the paper.

Finally, it is unclear what procedure was used to tune hyperparameters for the experiments. The hyperparameters vary quite a bit from task to task, with different forms of regularization used, different forms of the reward function, different amounts of pre-training, and different amounts of gradient penalty. This suggests that the performance of the proposed method is quite sensitive to hyperparameter selection. Indeed, this may also be a flaw of prior work in this domain (as GANs are notoriously difficult to tune), but such limitations should also be more transparently discussed in the body of the paper.

Minor comments:
	- Proof of proposition 1 is not rigorous -- it should either be made rigorous, or the body of the paper be revised to say "proof sketch is in the appendix"
	- Tables should bold all entries that could statistically be the best -- for example, in Table 2, InvertedDoublePendulum, GAIfO and DPO (w PG) have overlapping confidence intervals, so bolding only DPO (w PG) is misleading.
	- In the toy experiment, it is unclear what the effect of the added action ambiguity achieves. It would be good to also include experiments comparing to k=1 to show if empirically the baseline algorithms suffer from environments with action ambiguity.

limitations_and_societal_impact: The discussion of limitations and negative societal impacts are minimal and inadequate. The limitations of an approach go beyond error bounds, but also into more practical considerations. I would encourage the authors to include a broader discussion, as I state earlier. Furthermore, while the checklist says that section 6 includes a discussion of potential negative societal impact, I do not see any discussion to this effect there.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the state-only imitation learning problem by decomposing it into two components: a state-transition predictor and an inverse dynamics model. Specifically, it argues that the problem is difficult because there are potentially multiple policies that could match the same state trajectory of the expert. 

Instead, it proposes to learn what they call a hyper-policy, which is a family of policies that lead to the same distribution over state trajectories. Then, after recovering the optimal hyper-policy, it learns an inverse dynamics model to recover the actions, and this model can be learned by interacting with the environment.

However, to actually achieve good performance in practice, they propose two regularizations, i.e., a multi-step prediction loss and a cycle training loss. The parameters of the hyper-policy and the inverse dynamics model are also trained with a policy gradient objective where the reward is based on a discriminator (which is trained to predict whether the transition (s, s’) is from the expert). 
main_review: Originality: 
- This paper decomposes the state-only imitation learning problem into learning a state predictor and an inverse dynamics model. As noted by the related work, this decomposition has previously been studied in the context of imitation learning [16] and reinforcement learning [3]. 

- Notably, the QSS work also shows that their method can be adapted to the state-only imitation learning setting. It would be good to clarify the differences and to include a comparison in the experiments if possible.

Quality:
- The method is compared to a couple of prior methods. Comparatively, their full method demonstrates better performance. However, it seems like the reward from the discriminator is what enables good performance, more so than the decoupled structure.

- The evaluation is on Mujoco continuous control tasks and a simulated driving task. These have relatively low-dimensional state-action spaces and relatively deterministic dynamics, which makes learning a forward dynamics model, required by the cycle training, easy. I’m curious how well this would scale to higher-dimensional observations however.

Clarity:
- The paper is overall well-written and organized, and provides a good overview of the existing works in the literature in the related work section and through Table 1.

Significance:
- Imitation learning from observations is an important problem setting with real-world applications (as shown through the driving experiment).

- Since the results are quite close and the differences less discernible in many of the Mujoco tasks, evaluation on a set of harder tasks would be more convincing.


==========================

Thanks for the detailed response, and for the additional comparison to QSS. 
limitations_and_societal_impact: Yes. See the main review for other limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a method for state-only imitation learning (SOIL) that decouples learning the expert state-transition distribution from the actual policy needed to reach desired states. In particular, given a dataset of expert state sequences, the method trains two models: one that predicts next state given the current state, and another one that learns inverse dynamics, i.e. which action is needed to reach the next state from the current state. Where the state prediction can be directly learned from the state-only expert data, environment interactions are collected using a sampling policy to learn the inverse dynamics model. Additional techniques, such as multi-step optimization and cycle training are used to improve prediction accuracy of the state-transition model. In addition, training is improved by introducing a discriminator with a GAN-like reward that can provide intermediate reward signals to the policy. The method is evaluated on several MuJoCo continuous control environments, where it is shown to outperform other SOIL baselines, and on a real-world human driving dataset where it can learn effective driving policies just from human driving trajectories. 
main_review: Strengths
- State-only imitation is important for initializing robotic policies without access to ground truth actions, especially when imitating non-robotic agents like humans.
- The paper is well-written, and straightforward to understand and follow. Mathematical derivations are coherent.
- The method is shown to outperform multiple SOIL baselines and work with a real-world dataset.

Criticism
- One of the motivations for the presented work, which is discussed in Section 2, is that there is often not a one-to-one correspondence between expert state transitions and the underlying policy. However, as described later in the paper, both state-transition function and the inverse dynamics model are deterministic, which prevents them from representing a multi-modal policy. It would be interesting to see a discussion how the ambiguity of the underlying expert policy can be represented within this framework.
- To better understand scalability of the presented approach, it would be interesting to see evaluations on higher-dimensional problems, e.g. using visual inputs.
limitations_and_societal_impact: The authors discussed limitations of their work. The proposed approach does not lead to any obvious negative societal impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new method for state-only imitation learning, based around explicitly representing the transition from one state to the next (based on state-only expert demonstrations) and an inverse dynamics model providing feasible actions to produce that transition (based on rollouts from a sampling policy). Theoretical analysis attempts to show that this reduces policy ambiguity. Empirical results show improved performance relative to baselines, with the most striking results in a gridworld and on real-world driving data: on standard MuJoCo continuous control tasks, the mean of the method tends to be higher, but the confidence intervals overlap with baselines.
main_review: There are many data sources that only provide observations. To exploit such data sources, we need to be able to learn from data without actions (the focus of this paper) and from partially observable data (not the focus). The basic idea of exploiting the structure of the problem by separately learning to predict the expert state transitions and to predict actions that lead to them is plausible. Unfortunately, the paper is confusingly written, key theoretical details are either incorrect or stated too informally to verify, and the empirical results while promising are not fully convincing. Given these deficiencies, I do not think this manuscript is suitable for presentation at NeurIPS.

# Issues with derivation

The proof for Proposition 1 (appendix B) is very informal. While the overall claim seems approximately correct, it is missing some key assumptions (which you would likely have noticed had the proof been written rigorously). In general one may not be able to recover the policy from the state-action OM, since the OM $\rho(s,a)$ may be zero for all actions $a$ transitioning out of a particular state $s$. For example, this can occur if $s$ is not reachable from the starting state of the MDP. Granted you may not care about this unindentifiability (although it does matter in the transfer case), but it does mean there is policy ambiguity even in the state-action case, and there is no one-to-one correspondence. Additionally, the “proof” that there is no one-to-one correspondence between $\rho(s,s')$ and $\pi(a\mid s)$ seems to be missing a step – simply writing down the marginal formula does not establish this. Indeed, there exist MDPs where the transition dynamics are invertible, so that we can establish $(s,a)$ from $(s,s')$.

Definition 1 is also rather imprecise. A literal reading implies that a hyper-policy is any set of policies where all policies have the same state transition OM. This would include, of course, the empty set and all singleton sets of policies. In this case the following claim that “there is a one-to-one correspondence between the hyper-policy $\Omega$ and the state transition OM $\rho_{\Omega}(s,s')$ is clearly false. For any $\Omega$ with size $|\Omega| > 2$, any subset of it is still a hyper-policy and has the same OM. I think you need the hyper-policy to be the set of *all* policies with this OM for this claim to go through.

Proposition 2 is stated without proof, and I do not believe it to be true due to the issue discussed with the Definition, although the intuition behind it seems plausible so I imagine this can be fixed.

# Experimental results

I appreciated the gridworld experiment -- although simple, this does provide a striking qualitative illustration of the difference between the methods. However, it would help to have some discussion as to why your method performs better in this case. In particular, I suspect the GAIfO failure mode was due to mode collapse, which often happens with GAIL (and GANs more generally). Should we expect your algorithm to systematically avoid these problems, or will similar problems occur given enough task complexity?

Table 2: It appears the confidence interval for DPO overlaps with the confidence interval for at least one other method in every column. Although the means being >= other methods in 5/6 cases is certainly promising, it would be more convincing if the difference was statistically significant. Can you run more seeds to reduce variance?

# Minor points

Correctness:

- “Furthermore, if the environment and the expert policy are both deterministic (which is usually the case in real-world scenarios such as robotics)” – this seems to be claiming that human demonstrations are deterministic. While humans may be deterministic in a philosophical sense, unless you are observing all relevant state (how much sleep the human had, how caffeinated they are, etc) I very much doubt this will be true in practice.

- “we also induce a similar error bound for rollout with a state-to-action policy as BCO [22] to show the advantage of the decoupled structure”. How is the decoupled structure an advantage when the error bound is similar (not stricly less)?

Clarity:

- Line 26: “state-to-action policy” – don't all policies mapping states to (distributions) over actions?

- Line 30: “state transition” is unclear. Do you mean visitation distribution? State sequence?

- Line 89-90: I did not understand any of this: “Therefore, a state-to-action mapping function may be too implicit for matching the state sequence, which could cause training instability and lead to sub-optimal policies. In that case, we must find a one-to-one corresponding solution to solve SOIL explicitly and efficiently.”

- Line 99: “and $\Gamma$ is its space” – what does this mean? Clarity is always important, but is absolutely essential when stating definitions and theorems.

- Would be helpful to introduce BCO somewhere as it's referred to repeatedly in the experiments. The only place it is defined seems to be in a table in related work.

# Update after rebuttal

I would like to thank the authors for a detailed response to my and others reviews. The propositions are now much more clearly stated and although I did not have time to check every detail of the derivation for Proposition 1 it appears to now be sound. In light of this, I am increasing my score.

Thank for you running the additional experiments. In particular it is helpful to see from Figure 3 that BCO learns the correct behavior (albeit slowly) when there is an invertible transition dynamics. I may be willing to increase my score further depending on the confidence intervals (see question below).
limitations_and_societal_impact: There is no discussion of social impact, and little discussion of limitations -- there is a brief discussion of future work in the conclusion.
review_title: null

Meta-review: The paper presents a method for state-only-imitation learning. The key ideas are: 
(a) Predicting future expert states from the current state. 
(b) Using the inverse model to transition from current to the predicted state. 
(c) Use of cycle consistency loss to improve the training of expert state prediction. 
(d) Error bound on tackling the challenge of compounding error. 
(e) Use of discriminator training to further improve performance. 

Two reviewers score the paper as 6, one as 7, and the other with a score of 5 to reject the paper. The considerations are: 

- Reviewers asked for a comparison to QSS and other baselines that the authors provided. 
- Authors clarified the proof of their propositions. 
- Authors clarified concerns about modeling stochastic distributions. 
- Reviewers were split on the clarity of the writing, that authors promised to improve. 

At the end of the rebuttal, M4Kk believes that updates both in terms of the writing and the experiments are substantial and therefore require another round of review. Considering the fact that the authors had addressed a majority of the concerns and the rebuttal incorporated all that would go in the revised version, I was initially inclined to recommend the paper for acceptance. 

However, studying the paper in more detail revealed an issue with the lack of comparison with prior work that I mentioned in my comment to the authors. After considering the author's response my opinion is: (details in the discussion and the thread below): 

The main premise of the paper is that state-only IL is hard because there are multiple policies consistent with a sequence of expert states. The paper proposes decoupling the learning into predicting expert states (i.e., what to do) and using the inverse model to follow the predicted expert states. However, as reviewer DMhm remarks, the most gain can be attributed to the use of discriminator rewards. The authors acknowledge this in the rebuttal, but also mention that additional supervision signals can be used due to the de-coupled policy structure which can improve performance/sample efficiency. The paper can thus be seen as proposing a set of ideas that encourage the imitator to follow the expert demonstrations better than a naive inverse model. In this view, 

- Comparison with an improved version of inverse models that uses cycle consistency [1] is critical. It is because both cycle consistency in training of inverse model and the proposed method of using discriminator rewards (possible due to de-coupling) are both encouraging state-matching, albeit in different ways. 

Further, if the proposed method outperforms [1], then the paper should be re-written to highlight that PG fine-tuning is critical (which experimentally leads to maximum gain, Figure 5) and decoupling allows this. The way the paper is written, it's hard to understand why de-coupling. 

In light of missing comparison and that paper would have gone a substantial re-write to incorporate suggestions from all the reviewers, I am siding by reviewer M4Kk, "revisions are substantial enough that I think the paper would benefit from another review cycle, especially given the general lack of polish in the initial submission."

[1] Zero-shot visual imitation, ICLR 2018.

Decision: Reject