Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a technique for combining "explicit" and "implicit" priors for reinforcement learning with demonstrations. Implicit priors refer to priors that store the knowledge from demonstration in a neural network of some kind (such as a flow-based generative model), while explicit priors store demonstrations as a database that can be queried when learning a new task. The overall method has several steps: 

1. Given a dataset of task-agnostic (TA) demonstrations, cluster them using k-means (where the last state of each trajectory is used as the clustering feature). If the given datasets are already divided into separate tasks, this step can be skipped. 
2. Learn a flow-based prior for each of the cluster from the previous step, and learn a flow-based prior for task-specific trajectories. The number of task-specific trajectories is much less than the total number of trajectories. Often, there's only one task-specific trajectory (one-shot learning). 
3. Learn weights to combine the numerous flow-based models into a single model. These weights are learned by minimizing a loss on  the task-specific dataset, and I have some questions about this step below. 
4. Given a new task, train a policy \pi(z|s) that controls the prior model, which in turn controls the environment. This is similar to prior work (PARROT). 
5. You are also allowed to access a dataset of expert trajectories on the current task. This dataset is accessed via a retrieval procedure wherein you search for a state that is most similar to the current state, and retrieve the next state. The next state is then passed as an additional input to the flow-based models, allowing the flow-based model to act as a sort of inverse dynamics model -- given state and next state, the flow based model predicts an action. This step is crucial for good performance from what I can tell. 

The authors evaluate their method on simulated robot manipulation tasks. 

Currently, my rating for the paper is somewhat low as I am not able to make sense of the results, and I am hoping the authors are able to answer the questions I asked here so that I can revise my rating accordingly. 

Post-rebuttal: I have updated my rating to a borderline accept following the author response, as the authors have addressed a most of my concerns. I would encourage the authors to release their code if they can, as it would make the paper reproducible, and would spur future research in this area. One of my concern was that the results seemed almost too good to be true, and this concern can also be put to rest with a code release. 
strengths_and_weaknesses: Strengths
- The paper tackles an important problem: how to best make use of task-agnostic and task-specific demonstrations to speed up reinforcement learning of new tasks. 
- The paper obtains some impressive results on an interesting suite of tasks. 
- The paper compares against a number of baselines, competing methods, and ablations. 

Weaknesses
- The overall method is somewhat complex, with a number of moving pieces, and some of the design choices (such as the retrieval system) seem a bit ad-hoc. 
- The experimental results are a bit puzzling, and I will add more details on this in the Questions section below. 
- Most of the analysis / ablations are actually in the Appendix. Overall, the authors spend too much space explaining the method (which could have been shortened, since we don't need all the details on how flow-based models are trained, we can refer to prior work for that), and too little space analyzing the experimental results. 

questions: - Authors use the terms “explicit priors” and “implicit priors” frequently, but these terms are never clearly defined (and they are not standard terms, as far as I know). From my reading, it seems like implicit priors refers to a method that trains a neural network using provided demonstrations, while explicit priors store all demonstrations in memory and can be later accessed using a nearest neighbor style technique. Can you clarify what exactly these two terms mean, and add these definitions towards the beginning of the paper? 
- The authors train several different flows on the task-agnostic and task-specific datasets, and then combine these flows using some coefficients, and these coefficients are learned by minimizing the log-likelihood of the “combined flow” on task-specific trajectories. However, given that one of the flow models was trained on task-specific trajectories alone, this subsequent optimization might result in the coefficient for the task-specific flow model to receive most of the weight. Am I missing something here? Can authors share what coefficients were assigned to the task-specific flow model in their experiments? 
- I understand the motivation behind the push-forward term in the retrieval objective function (Equation 6), but I did not completely follow the definition of the indicator function \delta. Can the authors add another equation that precisely defines when \delta = 1? Is it when s_next = s? 
- In Figure 3, why does CEIP have a much higher reward right at the start of training (when steps = 0)? Why do other flow-based methods (such as PARROT) start with a much lower return? In particular, I would expect PARROT-TS to have a good initial performance, given that it’s trained on data for the task at hand (and task is relatively simple). Also, since CEIP trains on both TA and TS data, it would make sense to compare against a version of PARROT that uses both TA and TS. 
- In Figure 4, the learning curve for CEIP for about half the tasks are essentially flat, with maximum performance achieved without any finetuning. Can the authors explain what is happening here? Is imitation learning alone sufficient for solving this task? The results seem very puzzling to me.

limitations: There is almost no discussion on limitations in the paper (the authors only briefly mention the computational cost of training many different flow models), which is disappointing. The lack of discussion around limitations, combined with results that seem almost too good to be true is a bit concerning.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This submission proposes a skill learning algorithm for offline RL that can leverage task-specific demonstrations as implicit and explicit priors. The skill policy consists of a mixture of normalizing flows (trained on task-agnostic demonstrations) where the mixture weights are fine-tuned on task-specific demonstrations. The flows resemble an inverse model, in that they are conditioned on both the current and next state. The next state forms the explicit prior, as it is queried from the task-specific dataset via nearest-neighbor lookup for each state encountered during training. The explicit prior enables very effective few-shot generalization (i.e., utilizing the task-specific demonstrations without with random latent vectors, as this is what I suppose corresponds to the first data point in the learning curves), and learning a high-level policy to predict latent state flow inputs can further improve performance.
strengths_and_weaknesses: The paper is very well-written and easy to follow. Clarity in the experimental section could be improved by stating what the abbreviations in ablation studies mean, and the ablations in the appendix are a bit hard to follow. Overall, the idea presented is well-motivated, novel and concerns a topic of high interest. The technique mainly builds upon PARROT and includes tricks for fine-tuning on task-specific demonstrations.

The experiments clearly show that the proposed method is effective in the settings considered; in fact, there is no setting in which few-shot generalization does not already work much better than all baselines considered. Overall, this suggests a potential limitation that doesn't seem to be acknowledged explicitly in the paper: how much does the method rely on precise task-specific demonstrations being available? What would happen if the environment changes slightly, e.g., the microwave in the kitchen environment would be moved 10cm to the left? I'd be happy to raise my score if this limitation is properly addressed, as I think it would help greatly in judging the method's applicability and impact.
questions: - How would simply replaying a task-specific demonstration perform on the benchmarks? Does it already provide optimal reward and trajectories? I assume that fine-tuning the normalizing flow mixture requires ground truth actions of the demonstrations, so the return of the demonstrations themselves could be added as a baseline/topline.
- How much leverage does the high-level policy actually have over the actions? The flow policy is invertible, but conditioning on current and next state might, depending on the environment, provide all required information to compute an action already. Some analysis on this aspect would be appreciated.
limitations: The paper is not clear on the quality of demonstrations that are expected for the method to work effectively.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method CEIP that leverages explicit and implicit priors from the demonstration for reinforcement learning. On top of PARROT [45] which uses normalizing flow prior to task agnostic dataset to generate action, CEIP learns coefficients to combine flows to generate one task-specific flow parallel and use (state, predicted next state) pairs as the input of the flow instead of the state. As a result, CEIP achieves better performance than its baselines, PARROT, FIST, and SKiLD. 
strengths_and_weaknesses: **[Strength]**

1. The paper is well-written and easy to understand. 

2. Especially, the detailed description of baselines, environments and other experimental detail make the reader have a better understanding of with the attached codebase.

3. The proposed method clearly outperforms other baselines, PARROT, SKiLD, and FIST in all experiments and the authors also present the contribution of explicit prior by presenting Figure 5(c).


**[Weakness]**

1. It seems that there is no difference between task-agnostic flow (f_1 to f_n) and task-specific flow f_{i+1}. The ratio and size of the task-agnostic dataset and task-specific dataset are needed. Moreover, I recommend presenting the coefficient of f_{i+1} in those cases.

2. Another concern is that a detailed description difference between CEIP and PARROT-TS-EX-forward is missing. According to Line 220, PARROT-TS only uses task-specific data while CEIP-TS uses both task-specific and task-agnostic data. Is the dataset size a key factor of the difference or the Few-Shot Adaptation?


3. To present the effectiveness of the explicit prior, showing the results of TA-EX, TA-EA-forward in PARROT are better than TS-EX and TS-EX-forward (Figure 5(c)).


4. It is good to compare using CEIP with the ground-truth task label (oracle) for not using k-means clustering.

questions: 1. In Line 202, us -> use

2. In Figure 5 (b) and Figure 9, it is hard to differentiate between solid and dotted lines. It is better to use different markers for the same category of EX instead of two lines.

3. In Lines 164-167, “most useful” is a vague term, it would be better to use other expressions such as making the best performance, etc.

limitations: The authors addressed the limitations and potential negative societal impact of their work. As mentioned in the paper, I am concerned about the training time. I recommend parallelizing the training procedure.
review_title: null

Meta-review: metareview: All three reviewers have elected to accept the paper, with accept ratings of 5,6,7. 

The reviews were thorough and demonstrated an understanding of the paper, and the authors have addressed many of the suggested edits. I like that the paper tackles the combination of parametric vs. non-parametric learning. One weakness of the paper, from a reproducibility POV (and also mentioned by the authors in limitations), is that there are a lot of moving pieces in the system (RL, non-parametric dataset lookup, one flow per task + 1 additional one for distilling them). It would seem quite annoying to implement correctly, if starting from scratch (but this is just an aesthetic feedback). 

Despite the authors saying that the paper is "not too good to be true", I still find the stark contrast between baselines and the proposed method a bit hard to believe. I believe the code (if released) by the authors would reproduce the stated results in the paper, but what I am more skeptical of is that the baselines couldn't be tuned to perform much better. This is important for this specific paper, given the complexity of the method: a practitioner would want to know whether there is a simpler way to implement the improvements proposed here. For example, authors mention "The key of our strong results are due to our combination of 1-layer flows with explicit prior, which are missing in the baselines. SKiLD and FIST have an LSTM-VAE architecture, which is too heavy with few task-specific trajectories compared to 1-layer flows; PARROT includes neither explicit prior nor flow combination."

This which makes me wonder whether there isn't some simpler way to implement this, i.e. k-NN retrieval paired with contrastive embeddings + small networks for behavior cloning.

A minor nit: The explicit / implicit priors terminology was also confusing to me, as I typically think of this as "amortized inference + retrieval" or "parametric learning + non-parametric learning". 

Recommendation: accept.


confidence: Less certain
Decision: Accept