Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the problem of KL-regularized reinforcement learning for locomotion tasks, where the KL regularization penalizes deviations from expert demonstrations.

This work makes the observation that fitting expert demonstrations to a certain class of conditional neural density models (e.g. neural networks that output the parameters of a Gaussian distribution), results in policies whose variance collapses for states that are different enough from the expert data. 

The paper argues that this collapse causes instabilities in learning KL-regularized policies: for Gaussian policies, the KL penalty grows quadratically to infinity as the expert policy variance goes to 0. To address this issue, the paper proposes to instead compute the KL penalty by fitting expert demonstrations to models that do not suffer from this collapse, in particular Gaussian Process regression models. Gaussian Process models result in variance that increases (depending on the choice of kernel) for states that are far enough from the expert data.

The paper provides some analysis on why the variance collapse affects the RL optimization process, experiments that  aim to show an empirical relationship between expert policy variance, KL penalty and magnitude of the policy gradients, and a comparison between various methods that use KL regularization with an expert policy.

main_review: As summarized above, this paper presents a method for KL regularized RL, in which the KL penalty computation uses a behavioural policy prior fit from expert data. The proposed method uses Gaussian Process models for fitting the expert data. The paper is clearly written, proposes a simple approach and provides empirical evidence of its usefulness. 

It is notable that, in the experiments presented in the paper, the resulting learned policies produce visually smooth behaviours when regularized with a smooth prior (even if the policy class is not smooth). The paper analyses the problem of variance collapse in KL regularized RL, showing the relationship between variance collapse (which is empirically observed) with optimization instabilities due to exploding gradients.

Some things to improve:

The paper limits its comparison to one between conditional Gaussian models with ReLU networks and GP models with smooth stationary kernels. While this is enough for the main point of the paper (addressing the variance collapse), it doesn't necessarily mean that parametric models will have variance that will invariably collapse, particularly in the case where modelling uncertainty is introduced in the parametric model fitting (e.g. the comparison with ensembles). The paper does not need to answer how parametric models could be made to work in this setting, but the main text should be explicit about what is specifically being compared.

An useful comparison that is missing in this story is the effect of expert dataset size on the performance of the algorithms. Is it always better to use GP regression for fitting the prior as the expert dataset size changes? 

What kind of ensemble was used for the experiments (bootstrap, different random seeds/initialization)? Since the paper introduces a comparison with ensembles, another useful comparison of the method would be against an ensemble model, as you let the size of the ensemble grow.  In the context of the discussion about ensembles, it would be helpful to include a discussion about other models that could be applied in this setting (e.g. bayesian neural networks). and their limitations.

A comment on the robotics experiments: in general, human expert demonstrations are not readily available for robot tasks. Collecting human expert data is still expensive and Perhaps the authors meant that there exists publicly available datasets for learning from demonstrations for robot manipulation tasks.

limitations_and_societal_impact: Section 4.1, which the authors point to as a description of the model limitations at the end of the paper, only mentions the cubic complexity of fitting GP models with exact inference. Further, this limitation is presented as not really being a limitation due to existing algorithms based on conjugate gradient solvers for exact inference with GPs. The paper, however, fails to mention other important limitations, in particular ones affecting the current implementation: how well will the method work with non-smooth data? with high dimensional input data such as images? How can the model be extended to multimodal expert behaviours? While the paper does not need to provide solutions to these issues, the discussion of limitations would be more helpful if it pointed out problems that are not easily addressable.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors identify a problem with KL-based reinforcement learning from expert demonstrations. Poor out-of-sample variance predictions of NN expert policies ends up unreasonably penalizing RL exploration outside the expert data distribution, potentially causing numerical instability. To remedy this they propose to instead use non-parametric expert policies based on a scalable GP approach.

main_review: This seems like a solid incremental improvement. They thoroughly analyze an important problem in previous work and propose a fix. RL from expert demonstrations is a relevant niche with real-world applications. The paper is very well-written and the experimental results appear to improve on prior state-of-the-art.

Some comments:

- L43: "without ad-hoc design choices" - Like what? this is both needlessly vague and ends up sounding a bit objectionable.

- L178-185: I don't believe it is the limited capacity from its parameteric nature. Policies in RL are usually very simple and overparameterized. However, it is well-known that an NN trained with maximum likelihood will not predict out-of-distribution uncertainty well, simply because it hasn't been trained on that. Both the mean and variance predictions will be very poor outside the training distribution. It also sometimes goes to zero, but I'm not aware of a formal analysis of this, or convinced that it always happens. As you note in L200, ensembles are sometimes used for predictions, so they can't all always go to zero.

- Figure 6: Clarify which "Non-KL" methods are used in the bottom row, I assume it is AWAC etc? It might also have been nice to compare against regular RL with pre-trained (BC) policies as a sanity check. 

- Section 5.3: Shouldn't you compare compute time against some RL algorithm *with* expert demonstrations? I understand that if it doesn't reach the reward thresholds without extra tuning, it becomes a bit apples to oranges, but this isn't ideal either. It's unclear how the overhead of the GP policy compares to the NN policy. It might be relevant to compare on the more complex benchmarks as well, as the simulator is usually the bottleneck in non-trivial RL applications.
limitations_and_societal_impact: Adequate for this type of paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a previously unrecognized pathology in KL-regularized RL with expert demonstrations. Specifically the commonly used parametric behavioral policies suffer from collapse of predictive variance at states far from the demonstrations. This collapse of variance hinders the algorithm’s ability to learn effectively. The authors then propose to use non-parametric behavioral policies and demonstrate its effectiveness in several control tasks.

main_review: Estimating predictive variance/uncertainty is an important topic that influences a broad range of ML research. Understanding and estimating the predictive variance in RL is in particular challenging due the instability of deep RL. This work provides an insightful analysis on KL-regularized RL algorithms and how the behavioral policy class influences the online learning process. The paper is well-written with clarity. The claims in the paper are supported with empirical experimental results and the designed experiments successfully highlight the problem of collapsed predicted variance at states not in demos in KL-regularized RL. This observation is novel and inspires the authors to propose a class of new algorithms that use non-parametric policies, which show comparable or better performance than prior methods in several high-dimensional control tasks. 

It would be nice if the authors could also comment on how the non-parametric policy class would compare with parametric policies regularized with MC-dropout [1], since MC-dropout has also been considered as a way to do bayesian approximation on the network weights, and is often compared with Ensembles in related literature for estimating aleatoric uncertainty of neural network. In addition, in the paper the authors only commented on how the problem setting is friendly for non-parametric methods, it would be interesting to know what the actual computational trade-offs are, if any, for using non-parametric policies.

[1] Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). PMLR. 


limitations_and_societal_impact: The authors discussed the limitations of the proposed method and how this work could influence the research community to focus effort on better understanding of how policy classes would influence learning in deep RL. The authors did not explicitly comment on potential societal impact. One possible societal impact the authors could mention is that accurately estimating the predictive variance of ML systems is critical for real-world risk-sensitive applications such as autonomous driving.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors identify a potential failure in KL-regularized reinforcement learning in which small predictive variance of the target policy may lead to exploding gradients, possibly destabilizing gradient-based learning algorithms. The authors discuss an uncertainty collapse of parametric models under Maximum Likelihood estimation in OOD regions of the expert, resulting in the aforementioned pathology. Finally, they suggest to use non-parametric models of the behavior policy (in which uncertainty collapse is not present), showing improved performance on several continuous control benchmarks.
main_review: The authors identify an interesting problem of KL regularized optimization and propose a solution to solve it. I find the problem interesting and the authors' solution to be well motivated and reasonable. In addition, the paper is written clearly, and the experiments were applied to reasonable tasks compared to similar / previous literature. The main limitations of the paper are as follows:

1. The model of the paper focuses on KL-regularized reinforcement learning. There are various modifications to this model which may trivially solve the authors' suggested pathology, yet the paper does not discuss them at all.
* First, KL-regularization is applied to the policies themselves and not the stationary distributions, d^\pi. Algorithms such as GAIL and AIRL can be combined with reward functions to construct regularized algorithms which utilize the expert data through stationary distributions. Would the issues the authors suggest still exist in this case? I would be interested to see comparison to such algorithms (with addition of reward).
* KL regularization is only one way to regularize the algorithm. Other f-divergences can be used which would not have the exploding gradient problem (e.g., TV-distance). Why do the authors focus on KL-regularized policies?

2. Scalability. While I agree that simple tasks would require small amounts of data, I find limited scalability as a major drawback of the paper. I hope for a solution that scales well with the amount of data provided, so that harder tasks could also be achieved by the proposed method.

3. Experimental Comparisons. As mentioned in 1 and 2 , the author's should compare their results to algorithms using stationary distributions (e.g., GAIL, AIRL), as well as other forms of f-divergences. I would also be happy to see comparison to CQL as well as DICE algorithms.
limitations_and_societal_impact: The main limitation of the proposed approach is its scalability, which the authors discuss in Section 4.1 implicitly. I would be happy to see a more explicit discussion of this limitation.
review_title: null

Meta-review: There has been a split amongst the reviewers for this paper. Although each expressed interest in the pathology of KL-regularization stated by the author, there has not been a consensus about whether the author's approach to this is impactful. I would recommend the position of the majority, 3 reviewers have found the set of baselines and architectures attempted are suitable to explore the problem at hand, and the emirical results are promising enough to be of value to the community as written (and with promised helpful additions in the discussion). Therefore, I recommend acceptance.

I do also sympathize with the position of reviewer hW6R, in that I may not be likely to use KL-regularization to solve the RL-from expert problem as of this year and have concerns about scaling GPs for high-dimensional problems. While these concerns are relevant, and could inform the authors about helpful discussion in the final version or their next papers, I still believe the contribution of a focused analysis of this widely used form of KL-regularization is sufficient in current form.
Decision: Accept (Poster)