Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper applied Online Imitation Learning to offline RL. The central idea is, at each iteration of training, the proposed approach will sample the data points from the neighboring policies based on the experience picking. The intuition is built based on the observation/analysis of the discrepancy between demo policy and initialized imitating policy.  I like how authors scrutinized their observations with theoretical analysis. The neighboring policy picking targets at solving the "dilemma" of quantity-quality. However, the main technical contribution is not super very in-depth, authors tackle the KL-divergence constraints through the assumption that each trajectory is sampled by a single policy, which I am not sure whether it is very practical as claimed by authors (yield good bound and performance as illustrated in experimental studies).
main_review: Strength:
- Paper is very well-written and easy to follow. The authors moved almost all proof to the appendix and made the paper accessible.
- The central ideas are motivated by real-world observations and theoretical analysis

Weakness:
- The experimental evaluation could be done with more complex datasets
- The technical depth: The evaluation of the two policies is based on relaxing the trajectories sampled by a single policy, not sure whether the assumption is validated.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed and interesting algorithm to do imitation learning with an automatic curriculum learning technique. In particular an automatic filtering of better trajectories under the current policy  and a selection of the best ones in that set.
main_review: 


Originality:  The idea is original, even though it sounds like BAIL for trajectories. But the math has been done well and it is well supported.

Quality: Claims are supported. I believe some sections / paragraphs would benefit for more clarity. Also it would greatly benefit from more extensive experimentation. Adding more environments.

Clarity: The submission is well organized but incomplete. The notation seems porr and writing need clarity, The figures would benefit from extra information some

Significance: The results are better than SOTA. Event thought in D4RL is not compara with CQL and BAIL, and more environments would benefit the claims.

Notes:
Confused by notation in line 91, unclear how the sum of samples belonged to the BC policy corresponds to a well formed policy.
Line 95, notation is not clear, D was defined as having trajectories but here it is used as having policies.
Line 97, re-used D for indication of tasks.
Line 159, Clarity "the close policy".
Line 181. "kinds of policies" grammar.
Line 195. Poor notation. "The target Dt" .
Figure 5 ( show comparison) needs to improve the graph, the seeds graphs as points are confusing.
Would benefit from adding D4RL don't compare with BAIL and CQL [ or even model based offline rl)

limitations_and_societal_impact: Its seems not readily applicable, but the authors could have written about the benefits of doing Offline RL in the sense of the impact of training. Or a computational analysis could have interesting, in the sense of cost of training, CO2 impact, cost, etc.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The method proposes an algorithm for Offline Reinforcement Learning. The work starts from the idea that behavioral cloning struggles at learning from data coming from several policies but, on the contrary, easily learns a behavior policy if initialized with a “close” policy.
A curriculum is thus built for behavioral cloning to efficiently learn the “best policy shown in the dataset”. For this, a behavioral cloning is trained on a sequence of trajectories defined the following way: the next trajectory selected for training is the most probable according to current policy. The trajectories-dataset is then filtered by saying that the return should be above a threshold given by a running average of the trajectories seen so far.

main_review:  
I appreciate the simplicity of the method, although some explanations could, in my opinion, be simplified. I have questions regarding the experiments although the results are quite convincing.

Remarks:
- 3.1: It would have been more informative to take the ⅓ (resp. ⅔ ) policy at ⅓ of the final performance rather than ⅓ of the training steps. The initializations are quite close in terms of return so I find it quite normal that such a good inits lead to great final performance.

Derivation of the algorithm:

- I find the derivation of the algorithm a bit confusing. First, there are some mistakes in the mathematical notations that make reading harder (e.g. l.213, no consistency between $(s_t, a_t)$ and $(s, a)$) or in the Eq.7 where there are confusion between $\pi_i$ and $\pi$, as well as between $\hat{\pi}_i$ and  $\hat{\pi}$.

- Proposition 1) is not rigorous. “Corresponds to” does not mean anything mathematically. The proof in B.2 uses implications rather than equivalences. So you proved that a trajectory satisfying the "KL condition" also satisfies the “probability condition”. However you are using the converse property.

- However, the final way of defining a “close policy” is simple and makes sense. I would advise either cutting off the KL based argumentation and go directly with the heuristic or refine the mathematical derivations.

- Even though the return filter is a very simple idea, I very much appreciate the fact that it brings a natural termination condition to the algorithm. I would like to see an ablation without the return filter at least in the appendix.

Experiments
- I argue that results of Table 1 should not appear as is. Indeed, when introducing a new dataset, it is unfair for baselines to use “default hyperparameters”. If you introduce a new dataset, you need to sweep the hyperparameters of the baselines at least as much as the ones of your own algorithm (or at the very least the learning rate...). You also need to specify how hyperparameter selection was performed. For your information, [1] discussed how to select HPs in offline RL and [2] discussed other ways to perform HP selection for imitation.

- Otherwise, the results are quite well presented and convincing.
Baselines in appendix E.2 are very natural and I would recommend adding it to the core of the paper if you have the space to do so. Can you however explain the weird x-axis with algorithms not all starting at step 0?

-Ablation E.3 is also very insightful. It is very helpful to assist potential users in finding good hyperparameters.

It is overall a good paper. I would be ready to increase my score if authors clarify the algorithm mathematical derivation and either solve the experimental issues I mentioned or debunk my arguments.

[1] Paine, Tom Le, et al. "Hyperparameter selection for offline reinforcement learning." arXiv preprint arXiv:2007.09055 (2020).
[2] Hussenot, Leonard, et al. "Hyperparameter Selection for Imitation Learning." arXiv preprint arXiv:2105.12034 (2021).




-----------    UPDATE AFTER AUTHOR RESPONSE  -------------

Thanks for the very detailed answers!

Q1: Thanks for adding this additional datapoint. I am glad to see that this 1/3 performance point also confirms your hypothesis.

Q3: It is a shame we cannot update the pdf, but I am happy the authors will now present the KL derivations as a relaxation rather than an equivalence. This will make the paper more rigorous and will not diminish the impact of its results.

Q4: Thank you for adding this ablation. It looks confusing in the authors response (typo in the reported numbers?) but the figure that was shared is clear and supporting the importance of the return filter.

Q5: Thanks for the additional efforts tuning the baselines. Please include these new results in the main paper.

Q7: I would recommend aligning the beginning of the training rather than the end... otherwise the x-axis is just.... wrong.

Given the additional work, including results and clarifications, provided by the authors, I increased my score to 7.

limitations_and_societal_impact: No direct societal impact.
review_title: null

Meta-review: This paper generated an interesting discussion between the reviewers and the authors. Especially, the authors made a very good job at addressing reviewers' concerns. They ran new experiments, finetuned the baselines and conducted a thorough ablation study. The paper is now considered as presenting a SotA algorithm for Offline imitation learning.
Decision: Accept (Poster)