Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a simple method to robustify policies that are trained from a small number of demonstrations using supervised learning (i.e., behavior cloning). The key idea is to synthesize recovery trajectories that are not included in the training set. This is done by generating possible short-horizon trajectories that end at the demonstrated state using a learned backward dynamics model $p(s_t, a_t| s_{t+1})$. The trajectories are then used as additional data to train the behavior cloning policy. The backward dynamics models are encouraged to generate diverse trajectories in order to enrich the set of out-of-distribution states that the policy can handle. Both the policy and the backward dynamics model are trained from a small number of demonstrations. The method is shown to achieve better robustness compared to naive behavior cloning method and a competitive offline IL baseline (VINS) on two simulated environments. 
strengths_and_weaknesses: Overall I enjoyed reading the paper. The method is conceptually simple and easy to apply on top of existing offline imitation learning methods. The empirical results also seem to support the main claims of the paper. 

My main concern about the paper is the scope and the details of the experimental evaluation. 

- The evaluation tasks are rather simple both in terms of the nature of the tasks (reaching / pushing & 2D nav) and the types of dynamics involved. Hence it is unclear to me whether the method is still effective for more complex tasks such as Franka Kitchen and dexterous manipulation (Fu et al., 2020). Given the ease of implementation and the fact that these environments have expert data available, I’d strongly encourage the authors to stress test the proposed method on these harder tasks. 
- How is the forward dynamics model baseline implemented? Specifically, how does the policy make use of the rollout generated by the forward dynamics model?
- How does the accuracy of the backward dynamics model influence the final robustness performance? I’d suggest a simple ablation study that injects different levels of noise to the learned backward dynamics model. 
- According to Table C.1, even BC can achieve near perfect performance in both Fetch and Maze tasks with only 5-20 demonstrations. Unless I’m missing something, this seem to directly contradict with the results in prior works (e.g., in D4RL) that require a few orders of magnitude more data to achieve reasonable performance. I hope the authors can explain this discrepancy or point out my misunderstanding. 

questions: Please refer to the main comments listed above.
limitations: The paper does not mention any limitation of the proposed method. Like mentioned in the comments above, I hope the authors could stress test the method and discuss the limitations of the method both conceptually and empirically.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work introduces a novel method that incorporates a backward model into the offline imitation learning process to improve the model robustness against covariate shifts. The generated demonstrations from the backward model would train the policy to recover from the error states and back to the main distribution that it is familiar with. The motivation is clear and the idea of using offline data augmentation is a promising direction for improving the robustness of imitation learning algorithms.
strengths_and_weaknesses: Strength:
1. The paper is well-written. The problem setting and technical terms are clearly introduced.
2. The idea of learning a backward model to generate more offline demonstrations is novel. Most prior imitation learning works only focus on the method part, I believe a good way of augmenting the training data is also an important direction to improve the robustness of offline imitation learning.
3. The idea of decomposing the backward model into two functions is a good way to model the different features of the action generation and state transitions. To handle the multi-model distribution of action generation, a conditional Gaussian probabilistic model is used, which could further generate diverse trajectories by sampling the noise. I found these architectural designs promising and reasonable.
4. The experiment results in both robot and maze environments show the promising advantages of the proposed approach.

Concerns:
1. One assumption is missing: this method assumes that the state transition in the environment is bidirectional. While in many real-world scenarios, the forwarding trajectory is not reversible which might make the generated traces impracticable in the actual environment. The proposed method might struggle when the environment has many bottleneck state transitions that are not reversible. But I do agree this is out of the main focus of this work.
2. It seems like the experiment tasks are more like some proof of concept. I'm a little bit concerned about the performance in high-dimensional and complex tasks. For example, what if the observation is an image?
3. I do agree generating more trajectories from given demonstrations could improve the robustness of the learned policy. But it is still unclear to me why the model could generate novel initial states and train the policy to recover from it. In the Fetch experiments, it seems like  BMIL solved many cases with unseen initial states. But these states shouldn’t be generated by the backward model since it is unseen in the given demonstrations. Then, why the policy model could handle it so well?
questions: 1. Why train the backward model together with the policy model? As an offline method, I thought the easiest way is to first train the backward model alone and then train the policy alone. I hope the authors could give more details and motivation about the training pipeline.
2. What’s the observation used in the Fetch experiment? I assume it is the proprioception / joint angles of the Fetch arm. Do you give the target pose of the object? Is the proposed method limited to low-dimensional observations? Any discussion about potential high-dim observation inputs like images would be great.
3. How does the method perform if the given demonstration is sub-optimal? Let’s say the demonstration is collected by a human expert and there are imperfect motions. Will it be a challenge for this method?
limitations: 1. Bi-directional state transition assumption.
2. Low-dimensional observation space.
3. Might be hard to handle sub-optimal demonstrations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper seeks to improve the robustness of traditional behavior cloning, by aggregating backward traces from the demonstration to nearby states into the dataset. They do so by learning a factored backwards dynamics model, and rolling out this dynamics model from demonstration states to retrieve backward traces. By training a subsequent learner policy on original demonstrations and backward traces that lead the agent back to the demonstrated states, they improve the robustness of the learned policy on 4 different domains. 

strengths_and_weaknesses: ### Strengths

** Clarity: **

1. Overall, the paper is written very well. The problem tackled is well motivated, the approach is described in detail and most design choices are well justified. The contribution is well contextualized in the context of prior work, and the experimental results are also described in sufficient detail. Two particular ways in which the paper is written that should be appreciated are - 

    a. The paper does not exaggerate their claims beyond what their experimental results show. 

    b. The work also pre-emptively answers several questions that naturally arise during the discussion of their work (such as the accuracy of the backward dynamics model, and the effect it has on robustness of the learnt policy).

** Quality: **

2. Overall, the paper is also technically sound. 

    a. The paper honestly describes their claims (more robustly solving a set of tasks that current IL algorithms can currently solve), and their claims are supported by the experimental results they present. 

    b. The choices of methods are well justified, and for the most part, the paper provides intuition about the functioning of their method (such as the dynamics of accuracy of the backwards model in training, and its relation to how robustly the policy learns). 

    c. I also appreciate that the paper is very upfront about which specific problem setting they tackle, and when their approach would, and would not, be useful.

### Neutral

** Originality **
1. The originality of the paper is fair. The paper adopts a very similar approach to prior work [41], but applied to the imitation learning domain rather than the reinforcement learning domain as in [41]. This suggests the approach itself is not entirely novel, and only differs in some of the lower-level implementation details. This; however, is acceptable - the paper is upfront about this similarity to prior work, but clearly discusses why solving a similar problem in the imitation learning domain is important, why current methods that do so are limited (such as DAgger being expensive to run, etc.). As a result, one must give credit to the paper for identifying the applicability of this approach to the imitation learning setting. As a result, the overall originality of the paper is fair. 

### Weaknesses

** Significance **
1. While the experimental results presented are technically sound, they are presented in somewhat limited settings. 

    a. The tasks that the paper evaluates their approach on are quite simplistic tasks, such as solving a maze or controlling the Fetch end-effector to reach / push / place objects. In these tasks, “robustness” of the policy is a nice property, but not crucial to success or necessary for safety. The paper could benefit from evaluating their approach against baseline approaches on a more diverse set of tasks that are inherently harder to solve. Examples of such tasks could be more complex manipulation tasks, tasks in the joint space, continuous navigation tasks, and more interestingly, tasks where dynamics matter as well, such as continuous control of cars or legged agents. While I appreciate the authors mention this is a plan in future work, the impact of the paper in its current form is a tad underwhelming, and could be greatly improved by such evaluation on diverse tasks. 

2. Applicability of approach. The paper could also benefit from both discussing, and experimenting on, a broader settings (in addition to other tasks as mentioned above). The paper currently states such settings are not considered, but it is unclear why. Some such settings include - 

    a. For some reason, the paper only explores robustness with respect to spatial dimensions (such as joint coordinates, and not velocities, as mentioned in Lines 143-144). It is unclear why they restrict their discussion to this setting. Being able to recover to arbitrary states, and not just spatial dimensions, would be a big improvement to the paper. Also, several interesting questions then arise - does one need to follow a different action selection strategy for such dimensions? Does one need to train the backwards dynamics model any differently for such cases? Are backwards dynamics models as well conditioned when joint velocities or other non-linear dynamics are concerned? 

    b. Another interesting setting is when there are additional demonstrations available. Quantifying how much the robustness improves from the proposed approach with respect to number of additional demonstrations / some measure of coverage of state / action space of the demonstrations would be interesting to see, and useful in assessing the impact of the paper.

    c. Another interesting area (but one that is likely better suited for future work) is when there is also access to computationally expensive interactive experts and demonstrations. Could the proposed approach be combined with interactive feedback to verify or improve robustness of the learner?

### Note to authors
I strongly urge the authors to consider evaluating their approach on the above suggested domains / settings, as I believe it would contribute to making their paper much stronger. Currently, my biggest reservation with the paper is these evaluations, but should these be improved, I think this could make for a strong paper. 

questions: In addition to the primary in-line questions mentioned in the above section, here are a few other questions I have - 
1. In Line 148-149, the paper mentions irrecoverable states. The reviewer agrees that the assumption of existence of recovery actions is reasonable outside of such irrecoverable states. However, does this preclude the use of the approach in domains where irrecoverable states exist? For example, could a backwards model be queried at states nearby irrecoverable ones, and be made to be more robust near such states? 
2. How do the authors balance the backwards model B being accurate w.r.t the forward policy, with its ability to generate diverse new traces? (as mentioned in lines 163-164). 
3. How reasonable are the assumptions of Lines 179-180, of “”If the reverse time model B is accurate and the previous action generator BA gives sufficiently diverse actions”? 
4. In Line 195, is the definition of $a’$ meant to be $a’ \sim \mathcal{N} ( \mu, \sigma), \mu = [ a - k, a + k]$, rather than the specified equation? 
5. In line 191, why is it necessary for the action-selection strategy to cover the expert actions? Is this to maximize robustness? What happens to robustness if the action selection distribution and the expert actions only intersect? 
6. In Line 198, the paper states “As we do not wish to deviate too much from the demonstration data, we perturb only the first action when $s’ \in \mathcal{D}_{demo}$”. Why is this desirable? Why don’t we want to stray far away from the demonstration data? 
7. In Lines 203-205, the paper states “In addition to using relatively short rollouts that limits compounding errors, we hypothesize that this is these traces do not necessarily need to be accurate in order to benefit the policy. It may be that having the general correct direction contained in the traces is sufficient for the policy to eventually reach states in the demonstration data”. By this logic, isn’t it sufficient to have an arbitrary action selection strategy, rather than one that relies on the backwards dynamics? 
8. How are the forward dynamics models trained in Line 298 used? 

limitations: For limitations, please view the section on weaknesses above.
review_title: null

Meta-review: metareview: The paper proposes training a backward model to teach agents to recover from drifting off the optimal state trajectories provided by a limited number of demonstrations. All reviewers have voted to (weakly) accept. 
confidence: Less certain
Decision: Accept