On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations
Authors: Tim G. J. Rudner
Cong Lu
Michael Osborne
Yarin Gal
Yee Whye Teh
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Expert Demonstrations
Behavioral Cloning
Gaussian processes
Uncertainty Quantification

Abstract: KL-regularized reinforcement learning from expert demonstrations has proved successful in improving the sample efficiency of deep reinforcement learning algorithms, allowing them to be applied to challenging physical real-world tasks. However, we show that KL-regularized reinforcement learning with behavioral reference policies derived from expert demonstrations can suffer from pathological training dynamics that can lead to slow, unstable, and suboptimal online learning. We show empirically that the pathology occurs for commonly chosen behavioral policy classes and demonstrate its impact on sample efficiency and online policy performance. Finally, we show that the pathology can be remedied by non-parametric behavioral reference policies and that this allows KL-regularized reinforcement learning to significantly outperform state-of-the-art approaches on a variety of challenging locomotion and dexterous hand manipulation tasks.

On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations
Tim G. J. Rudner" University of Oxford
Cong Lu" University of Oxford
Michael A. Osborne
Yarin Gal
Yee Whye Teh University of Oxford
University of Oxford
University of Oxford
Abstract
KL-regularized reinforcement learning from expert demonstrations has proved successful in improving the sample efficiency of deep reinforcement learning al- gorithms, allowing them to applied to challenging physical real-world tasks. However, show that KL-regularized reinforcement learning with behavioral reference policies derived from expert demonstrations can suffer from patholog- ical training dynamics that can lead to slow, unstable, and suboptimal online learning. We show empirically that the pathology occurs for commonly chosen behavioral policy classes and demonstrate its impact sample efficiency and online policy performance. Finally, we show that the pathology can be remedied by non-parametric behavioral reference policies and that this allows KL-regularized reinforcement learning to significantly outperform state-of-the-art approaches on a variety of challenging locomotion and dexterous hand manipulation tasks.
1 Introduction
Reinforcement learning (RL) [15, 24, 46, 47] is powerful paradigm for learning complex behaviors. Unfortunately, many modern reinforcement learning algorithms require agents carry out millions of interactions with their environment learn desirable behaviors, making them of limited use for wide range of practical applications that cannot be simulated [8, 28]. This limitation has motivated the study of algorithms that can incorporate pre-collected offline data into the training process either fully offline or with online exploration improve sample efficiency, performance, and reliability [2, 6, 16, 23, 52, 53). An important and well-motivated subset of these methods consists of approaches for efficiently incorporating expert demonstrations into the learning process [5, 11, 18, 42].
Reinforcement learning with Kullback- (KL) regularization is a particularly successful ap- proach for doing so [3, 27, 29, 31, 44. 51]. In KL-regularized reinforcement learning, the standard reinforcement learning objective augmented by Kullback-Leibler divergence term that penal- izes dissimilarity between the online policy and behavioral reference policy derived from expert demonstrations. The resulting regularized objective pulls the agent's online policy towards the behavioral reference policy while also allowing it to improve upon the behavioral reference policy by exploring and interacting with the environment. Recent advances that leverage explicit or implicit KL-regularized objectives, such BRAC [51], ABM [44], and AWAC [27]. have shown that KL- regularized reinforcement learning from expert demonstrations able to significantly improve the sample efficiency of online training and reliably solve challenging environments previously unsolved by standard deep reinforcement learning algorithms.
'Equal contribution. Corresponding author: tim. www.erdddde...om ox. uk.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Parametric
Non-Parametric
1000
1000
75
5.0
25
255
0.0
0.0
5.0
1.5
7.5
10.0
10.0
-2
-5
0.003 0.006
0:009
0.012
0.012 0.020
0,028
0.008
0.044
0.052
Figure 1: Predictive variances of non-parametric and parametric behavioral policies on a low dimen- sional representation (the first two principal components) of a 39-dimensional dexterous hand manipula- tion state space (see "door-binary-v0" in Figure 5). Left: Parametric neural network Gaussian behavioral policy Tp
(s), o (s)). Right: Non-parametric Gaussian process posterior behavioral policy Top(-1 (-1 Da) = (P( (s s")). Expert trajectories D used to train the behavioral policies are shown in black. The GP predictive variance is well-calibrated: is small the expert trajectories and large in other parts of the state space. contrast. neural network predictive variance is poorly calibrated: is relatively small on the expert trajectories, and collapses to near zero elsewhere. Note the significant diffcrence in seales.
Contributions. this paper, we show that despite some empirical success, KL-regularized rein- forcement learning from expert demonstrations can suffer from previously unrecognized pathologies that lead to instability and sub-optimality in online learning. To summarize, our core contributions
follows:
We illustrate empirically that commonly used classes of parametric behavioral policies experi- ence collapse in predictive variance about states away from the expert demonstrations.
We demonstrate theoretically and empirically that KL-regularized reinforcement learning al- gorithms suffer from pathological training dynamics in online learning when regularized against behavioral policies that exhibit such a collapse in predictive variance.
We show that the pathology can be remedied by non-parametric behavioral policies, whose predictive variances are well-calibrated and guaranteed not to collapse about previously unseen states, and that fixing the pathology results in online policies that significantly outperform state- of-the-art approaches on a range of challenging locomotion and dexterous hand manipulation tasks.
The left panel of Figure shows an example the collapse in predictive variance away from the expert trajectories in parametric behavioral policies. In contrast, the right panel of Figure shows the predictive variance of non-parametric behavioral policy, which-unlike in the case of the parametric policy-increases off the expert trajectories. By avoiding the pathology, obtain stable and reliable approach to sample-efficient reinforcement learning, applicable to wide range of reinforcement learning algorithms that leverage KL-regularized objectives."
2
Background
We consider the standard reinforcement learning setting where an agent interacts with a discounted Markov Decision Process (MDP) [46] given by 5-tuple (S,A, r, 7), where and A are the state and action spaces, ( (s, a,) are the transition dynamics, r(s,, a;) is the reward function, and 7 is a discount factor. Pa (T:) denotes the action trajectory distribution from time t induced by policy л|- s.). The discounted return from time step is given by R(T)) Ekst - **((S,, a&) for t E No. The standard reinforcement learning objective to be maximized is the expected discounted
return J, (7p) E,_ (T.) ([T))) under the policy trajectory distribution.
2.1 Improving and Accelerating Online Training via Behavioral Cloning
We consider settings where we have a set of expert demonstrations without reward, Do ((Sn a.)) ) (S, A), which we would like to use to speed up and improve online leam-
Code and visualizations of our results can be found https: httt:p:://sitteee wttpssstttegggeeooooo.ooooo wwwgl/e.co//vi////pps..e.

---

ing [5, 42]. A standard approach for turning expert trajectories into a policy is behavioral cloning [1, 4] which involves learning a mapping from states in the expert demonstrations to their corresponding actions, that is, no S - A. As such, behavioral cloning does not assume or require access to reward function and only involves learning a mapping from states to action in a supervised fashion.
Since expert demonstrations are costly to obtain and often only available in small number, behavioral cloning alone typically insufficient for agents learn good policies in complex environments and has to be complemented by a method that enables the learner to build on the cloned behavior by interacting with the environment. A particularly successful and popular class of algorithms used for incorporating behavioral policies into online training is KL-regularized reinforcement learning [10, 37, 43, 48].
2.2 KL- KL-Regglarized Objectives in Reinforcement Learning
KL-regularized reinforcement learning modifies the standard reinforcement learning objective by augmenting the return with negative KL divergence term from the learned policy æ to reference policy no- given a temperature parameter a. The resulting discounted return from time step E No is then given by
R(T)) = Σ
> [r(ssk as) - aDKL ( (sk) || To [o(-|s)))]
and the reinforcement learning objective becomes (Tu) = (m) [[TT]]. When the reference policy is given by uniform distribution, recover the entropy-rege regularized reinforcement learning objective used in Soft Actor-Critic (SAC) [13] up to an additive constant.
Under uniform reference policy No, the resulting objective encourages exploration, while also choosing high-reward actions. contrast, when To is non-uniform, the agent is discouraged to explore areas of the state space S where the variance of To(- (- s) is low (i.e., more certain) and encouraged to explore areas of the state space where the variance of To s) is high. The KL- regularized reinforcement learning objective can be optimized via policy-gradient and actor-critic algorithms.
2.3 KL-Regularized Actor-Critic
An optimal policy π that maximizes the expected KL-augmented discounted return J, can be learned by directly optimizing the policy gradient J.. However, this policy gradient estimator exhibits high variance, which can lead to unstable learning. Actor-critic algorithms [7, 17, 32, 38) attempt to reduce this variance by making use of the state value function (T,ss) = Ep- (R(())) |s]] or the
state-action value function Q" (st, at) ( s, ar] to stabilize training.
Given a reference policy To st), the state value function can be shown to satisfy the modified Bellman equation
V"T = Ea, (e) 1Q" (st, ax)] - ( (s)) |1 (s))))
with a recursively defined Q f-uunction
Q" (st, = r(s,, a.) + PBs V (st+1)]-
Instead of directly optimizing the objective function J via the policy gradient, actor-critic methods alternate between policy evaluation and policy improvement [7, 13]:
Policy Evaluation. During the policy evaluation step. Q7 (s, a), parameterized by parameters 6, is
trained by minimizing the Bellman residual
(o) (s), (Qo(s,, a)) - (r(se, a.) + El "e (
(2)
where D a replay buffer and is stabilizing moving average of parameters.
Policy Improvement. In the policy improvement step, the policy næ. parameterized by parameters d, is updated towards the exponential of the KL--augmented Q-function,
(d) Eg,~D [aDKL(TTT) ( (- (x) || mo(- st)) - Eg,~D [Ear~ro( (s)
(3)
with states sampled from replay buffer D and actions sampled from the parameterized online policy Tg. The following sections will focus on the policy improvement objective and how certain types of
references policies lead to pathologies when optimizing Jx (Ø) with respect to ó.

---

3 Identifying the Pathology
In this section, we investigate the effect of KL-regularization on the training dynamics. To do so, we first consider the properties of the KL divergence to identify potential failure mode for KL-regularized reinforcement learning. Next, we consider parametric Gaussian behavioral reference policies commonly used in practice for continuous control tasks [13, 51] and show that for Gaussian behavioral reference policies with small predictive variance, the policy improvement objective suffers from exploding gradients with respect to the policy parameters Φ. We confirm that this failure occurs empirically and demonstrate that results slow, unstable, and suboptimal online learning. Lastly, we show that various regularization techniques used for estimating behavioral policies are unable to prevent this failure and also lead to suboptimal online policies.
3.1 When Are Kl.-Regularized Reinforcement Learning Objectives Meaningful?
We start by considering the properties of the KL divergence and discuss how these properties can lead to potential failure modes in re--regulrrzedd objectives. well-known property of KL-regularized objectives in the variational inference literature is the occurrence of singularities when the support of one distribution not contained in the support of the other.
To illustrate this problem, we consider the case of Gaussian behavioral and online policies commonly used in practice. Mathematically, the KL divergence between two full Gaussian distributions is always finite and well-defined. Hence, we might hope KL-regularized reinforcement learning with Gaussian behavioral and online policies to be unaffected by the failure mode described above. However, the support of a Gaussian online policy (- ss) will not be contained in the support of a behavioral reference policy #n(- 8;) as the predictive variance (s;) tends to zero, and hence DKL ( sx) To |s,)) - reference policy tends to zero and the behavioral distribution becomes degenerate, the KL divergence blows up to infinity [25]. While in practice, Gaussian behavioral policy would not operate in the limit of zero variance, the functional form of the KL divergence between (univariate) Gaussians,
(s:) O. In other words, as the variance of behavioral
ou(s;) (st)
2 (S:) + (HL(())) (o(s:))
DxL (- s:) || To(- ss)) oc log
201(s:)
implies a continuous, quadratic increase in the magnitude of the divergence (s decreases, further exacerbated by a large difference in predictive means, [Mo()))) - (s:)).
As a result, for Gaussian behavioral reference policies mo(-1s)) that assign very low probability to sets of points in sample space far away from the distribution's 's mean (s))s)) computing the KL divergence can result in divergence values so large to cause numerical instabilities and arithmetic overflow. Hence, even for a suitably chosen behavioral reference policy class, vanishingly small behavioral reference policy predictive variances can cause the KI. divergence to "blow up' and cause numerical issues at evaluation points far away from states the expert demonstrations.
One way to address this failure mode may be to lower-bound the output of the variance network (e.g.. by adding small constant bias). However, placing a floor the predictive variance of the behavioral reference policy is not sufficient to encourage effective learning. While it would prevent the KL divergence from blowing up, it would also lead to poor gradient signals, well-callin well-calibrated predictive variance estimates that increase on states far away from the expert trajectories are necessary keep the KL penalty from pulling the predictive mean of the online policy towards poor behavioral reference policy predictive means on states off the expert trajectories. Another possible solution could be use heavy-tailed behavioral reference policies distributions, for example, Laplace distributions, to avoid pathological training dynamics. However, in Appendix B.3 we show that Laplace behavioral reference policies also suffer from pathological training dynamics, albeit less severely.
In the following sections, we explain how an explosion in DRI. (T (s)) T- o affects the gradients of (o) in KL-regularized RL and discuss of how and why (s)) may tend to zero in practice.
ss)) caused by small
3.2 Exploding Gradients in KL-Regularized Reinforcement Learning Objectives
To understand how small predictive variances in behavioral reference policies can affect-and possibly destabilize-onlie training in KL-regularized RL, we consider the contribution of the behavioral

---

reference policy's variance to the gradient of the policy objective in Equation (3), Compared to entropy-reeullaizzdd actor-critic methods (SAC, Haarnoja al. [13]). which implicitly regularize against a uniform policy, the gradient estimator J (ő) KL-regullaried RL gains an extra scaling TT(. Is):
term Va, log (ar Se), the gradient of the prior log-density evaluated actions
Proposition 1 (Exploding Gradients in -l.-Regularized RI.). Let mo(- 8) be a Gaussian behavioral reference policy with mean (s)) and variance (s))., and let Tal- s) be online policy with
reparameterization a = fo(c;; Se) and random vector Et. The gradient of the policy loss with respect to the online policy's parameters & is then given by
Jz)(( = (a lo (a (t) VVa log To(a sc)
(4)
- Q(,, ar) VoJo(es;s) St) + V log (ar St))
with Va a log (a st) = a mo For fixed las mo(ss)) aa log (a(lss)) grows as (f()
Ola 2 thus,
(d))(()) + oo as og(ss) +0 whenever Vofoler:si) #0.
Proof. See Appendix A.I.
This result formalizes the intuition presented in Section 3.1 that behavioral reference policy with a sufficiently small predictive variance may cause KI.-regularized reinforcement learning to suffer from pathological training dynamics in wradient-based optimization. The smaller the behavioral reference policy's 's predictive variance, the more sensitive the policy objective's ' gradients will be to differences in the means of the online and behavioral reference policies. As result, for behavioral reference policies with small predictive variance, the KL divergence will heavily penalize online policies whose predictive means diverge from the predictive means of the behavioral policy-eve even in regions of the state space away from the expert trajectory where the behavioral policy's mean prediction is poor.
3.3 Predictive Uncertainty Collapse Under Parametric Policies
The most commonly used method for estimating behavioral policies is maximum likelihood estima- tion (MLE) [44, 51], where we seek my = Tg. with 3² = arg max, (E.s [log T. (a (a|s)]} for a parametric behavioral policy TΦ. In practice, Te is often assumed to be Gaussian, To (-1s) = (H (s)), with (s) and (s) parameterized by a neural network.
While maximizing the likelihood of the expert trajectories under the behavioral policy is sensible choice for behavioral cloning, the limited capacity of the neural network parameterization can produce unwanted behaviors in the resulting policy. The maximum likelihood objective ensures that the behavioral policy's 's predictive mean reflects the expert's 's actions and the predictive variance the (aleatoric) uncertainty inherent in the expert trajectories.
However, the maximum likelihood objective encourages parametric policies to use their model capacity toward fitting the expert demonstrations and reflecting the aleatoric uncertainty in the data. As result, for states off the expert trajectories, the policy can become degenerate and collapse to point predictions instead of providing meaningful predictive variance estimates that reflect that the behavioral policy ought to be highly uncertain about its predictions in previously unseen regions of the state space. Similar behaviors are well-known in parametric probabilistic models and well- documented in the approximate Bayesian inference literature [3].
Figure 1 demonstrates the collapse in predic- tive variance under maximum likelihood estima- tion low-dimensional representation of the ddoor-binary-ve dexterous hand manipulation environment. shows that while the predictive variance is small close to the expert trajectories (depicted as black lines), it rapidly decreases further away from them. Examples of variance collapse in other environments are presented in Appendix B.6. Figure 2 shows that the predic- tive variance off the expert trajectories consis- tently decreases during training. As shown in Proposition 1, such collapse in predictive variance can
10
Valiition Vantance
Valiiation Log Lielhhood
Epochs
Figure 2: Collapse in the predictive variance (in blue) of a Gaussian behavioral policy parameterized by neu- ral network when training via maximum likelihood es- timation. Lines and shaded regions denote means and standard deviations over five random seeds, respectivcly.
5

---

HaliChoetah-v2
Ant-v2
Walker2d-v2
Aserago Rotaras
400
4000
000
1300
200
3000
Dxul [5]
00
12
10"4
10
Timestope
Timostppe
Tirestoge
10
-
5
x 10
Figure 3: Ablation study showing the effect of predictive variance collapse on the performance of KI -e-guullarieed RL MuloCo environments. The plots show the average return of the learned policy, the magnitude of the KL penalty, and the magnitude of the average absolute gradients of the policy loss during online training. The lighter the shading, the lower the behavioral policy's 's predictive variance.
result in pathological training dynamics in Kl.-regularized online learning-steerie the online policy towards suboptimal trajectories in regions of the state space far away from the expert demonstrations and deteriorating performance.
Effect of regularization on uncertainty collapse. To prevent collapse in the behavioral policy's 's predictive variance, prior work proposed adding entropy or Tikhonov regularization to the MLE objective [51]. However, doing does not succeed in preventing collapse in predictive variance off the expert demonstration trajectories, as we show in Appendix A.3. Deep ensembles [20], whose predictive mean and variance computed from the predictive means and variances of multiple Gaussian neural networks, are a widely used method for uncertainty quantification in regression settings. However, model ensembling can be costly and unreliable, neural networks from scratch and does not guarantee well-calibrated uncertainty estimates [39, 49]. We provide visualizations in Appendix B.5 which show that ensembling multiple neural network policies does not fully prevent a collapse in predictive variance.
requires training multiple
3.4 Empirical Confirmation of Uncertainty Collapse
To confirm Proposition 1 empirically and assess the effect of the collapse in predictive variance the performance of KL-regularized RL, we perform an ablation study where we fix the predictive mean function of behavioral policy and vary the magnitude of the policy's predictive variance. Specifically, we set the behavioral policy's predictive variance different constant values the set {1 x 10 (following a similar implementation in Nair et al. [27]).² The results of this experiment are shown Figure 3, which shows the average returns, the KL divergence, and the average absolute gradients of the policy loss over training. The plots confirm that as the predictive variance of the offline behavioral policy tends zero, the KL terms and average policy gradient magnitude explode as implied by Proposition 1, leading to unstable training and collapse or dampening in average returns.
function that attains 60% of the optimal performance
10 30-3, 10 10-2) 2)
In other words, even for behavioral policies with accurate predictive means, smaller predictive vari- ances slow down or even entirely prevent learning good behavioral policies. This observation confirms that the pathology identified in Proposition occurs in practice and that it can have a significant impact on KL-regularized RL from expert demonstrations, calling into question the usefulness of KL regularization as means for accelerating and improving online training. In Appendix B. we show that an analogous relationship exists for the gradients of the Q-function loss.
We attempted smaller values, but the gradients grew too large and caused arithmetic overflow.

---

4 Fixing the Pathology
In order to address the collapse in predictive uncertainty for behavioral policies parameterized by a neural network trained via MLE, we specify a non nan-parametric behavioral policy whose predictive variance is guaranteed not to collapse about previously unseen states, Noting that KL-regularized RL with behavioral policy can be viewed approximate Bayesian inference with empirical prior policy [13, 21, 40), we propose Non-Parametric Prior Actor-Critic (N-PPAC), an off-policy temporal difference algorithm for improved, accelerated, and stable online learning with behavioral policies.
4.1 Nor-Parametric Gaussian Processes Behavioral Policies
Gaussian processes (GPs) [36] are models over functions defined by a mean m(-) and covariance function ((+,.). When defined in terms of a non-parametric covariance function, that is, covariance function constructed from infinitely many basis functions, we obtain a non-degenerate GP, which has sufficient capacity to prevent collapse in predictive uncertainty away from the training data. Unlike parametric models, whose capacity limited by their parameterization, non-parametric model's capacity increases with the amount of training data.
Considering a non-parametric behavioral policy, mo s). with
A s To( s) GP (m(s), kss,s'),
(5)
we can obtain a non-degenerate posterior distribution over actions conditioned on the offline data Do A} with actions sampled according to the
A Do Ta(- (-(-1s, Do) (P(Ho(s), Es)sss'))
(6)
with
(s))=m(s) + k(s, (), -A - mm(A) and ((s.s))=k(ss5')) + k(s,
To obtain this posterior distribution, we perform exact Bayesian inference, which naively scales as (NN5) in the number of training points N. but Wang et al. [50] show that exact inference GP regression be scaled to N > 000, 000. Since expert demonstrations usually contain less than 100k datapoints, non-parametric GP behavioral policies are applicable to wide array of real-world tasks. For empirical evaluation of the time complexity of using GP prior, Section 5.5.
Figure confirms that the non-parametra GP's predictive variance is well-calibrated: is small in magnitude in regions of the state space near the expert trajectories and large in magnitude in other regions of the state space. While actor-critic algorithms like SAC implicitly use a uniform prior to explore the state space, using behavioral policy with a well-calibrated predictive variance has the benefit that in regions of the state space close to the expert demonstrations the online policy learns to match the expert, while elsewhere the predictive variance increases and encourages exploration.
Algorithmic details. our experiments, we use KL-regularized objective with standard actor- critic implementation and Double DQN [14]. Pseudocode is provided in (Appendix C.1).
5 Empirical Evaluation
We carry out a comparative empirical evaluation of our proposed approach vis-à-vis related methods that integrate offline data into online training. We provide a detailed description of the algorithms we compare against in Appendix A.4. We perform experiments on the MuJoCo benchmark suite and the substantially more challenging dexterous hand manipulation suite with sparse rewards.
We show that KL-regularized RL with non-parametric behavioral reference policy can rapidly learn to solve difficult high-dimensional continuous control problems given only a small set of expert demonstrations and (often significantly) outperforms ffate-of-the-ae methods, including ones that use offline reward information- whhch our approach does not require. Furthermore, we demonstrate that the GP behavioral policy's predictive variance crucial for KL-rreularized objectives to learn good online policies from expert demonstrations. Finally, we perform ablation studies that illustrate that ponmmaraetrric GP behavioral reference policies also outperform parametric behavioral reference policies with improved uncertainty quantification, such deep ensembles and Bayesian neural
7

---

HaliChoctah-v2
Ant-v2
Walker2d-v2
1000
0000
4000
200
10000
0000
5000
2500
10:00
2000
3XX
500
-1000
B6K
3XCR
3000
DXK
200
TTansttgu
Timeettpp
Timestopps
N-PAAC (Oun)
BRAC
AWAC
A.M
SACI
SAC
BEAR
DAPG
Figure Comparison of N-PPAC (ours) vs. previous baselines standard MuJoCo benchmark tasks. Top:
KL-bbaedd methods (dashed lines), Bottom: -o--KKd-based methods (dash-dotted lines). Both top and bottom plots include N-PPAC (blue). BRAC uses the same actor-critic algorithm as N-PPAC, but uses a parametric behavioral policy, and results in slower learning and worse final performance.
networks (BNNs) with Monte Carlo dropout, and that the difference between parammerettic and parametric models exacerbated the fewer expert demonstrations are available. We use the expert data from Nair et al. [27], every experiment uses six random seeds, and we use fixed KL-temperature for each environment class. For further implementation details, see Appendix C.2.
5.1 Environments
MuJoCo locomotion tasks. We evaluate N-PPAC on three representative tasks: "Ant-v2", "HalfCheetah-v2", and "Walker2d-v2". For each task, we use 15 demonstration trajectories col- lected by pre-trained expert, each containing 1,000 steps. The behavioral policy is specified as the posterior distribution of a GP with a squared exponential kernel, which well-suited for modeling smooth functions.
Dexterous hand manipulation tasks. Real-world robot learning is a setting where human demon- stration data is readily available, and many deep RL approaches fail to learn efficiently. We study this setting suite of challenging dexterous manipulation tasks [35] using a 28-DoF five-fingered simulated ADROIT hand. The tasks simulate challenges common to real-world settings with high- dimensional action spaces, complex physics, and large number of intermittent contact forces. We consider two tasks in particular: in-hand rotation of a pen to match target and opening door by un- latching and pulling a handle. We use binary rewards for task completion, which is significantly more challenging than the original setting considered in Rajeswaran et al. [35]. 25 expert demonstrations were provided for each task, each consisting 200 environment steps which are not fully optimal but do successfully solve the task. The behavioral policy is specified as the posterior distribution of GP with a Matérn kernel, which is more suitable for modeling non-smonn non-smooth data.
5.2 Results
On MuJoCo environments, KL-regularized RL with non-parametric behavioral policy consistently outperforms related methods across all three tasks, successfully accelerating learning from offline data, as shown in Figure 4. Most notably, it outperforms methods such AWAC [27]-the previous cooo--ffthe-rr-w-hiiee attemppss to eschew the problem of learning behavioral policies but instead uses an implicit constraint. Our approach, N-PPAC, exhibits increase in stability and higher returns compared to comparable methods such as ABM and BRAC that explicitly regularize the online policy against parametric behavioral policy and plateau at suboptimal performance levels they are being forced to copy poor actions from the behavioral policy away from the expert data. In contrast, using a non-parametric behavioral policy allows to avoid such undesirable behavior.
On dexterous hand manipulation environments, KL-regularized RL with non-parametric behavioral policy performs on par outperforms all related methods on both tasks, as shown in Figure 5. Most notably, on the door opening task, achieves a stable success rate of 90% within only 100,000 environment interactions For comparison, AWAC requires 4x many environment interactions to

---

pen binar-o
door binary-v0 vO
1.00
1.00
0.30
0.50
0.25
0.35
1.00
1,00
0.50
0.30
0.25
0.25
Timesters
Timestens
N-PPAC (Ous)
BRAC
AWAC
AWR
ABM
SACID
SAC
BC
BEAR
DAPG
Figure 5: Left & Center: Comparison of N-PPAC (ours) vs. previous baselines on dexterous hand manipulation tasks. Top: KL-based methods (dashes), Bottom: Non-KL-based methods (dots and dashes). Both top and bottom plots include N-PPAC (blue). Right: The pen-binary-v0 (top) and door-binary-vo (bottom) environments.
achieve the same performance and is significantly less stable, while most other methods fail to learn any meaningful behaviors.
Alternative divergence metrics underperform Kl-regularization. KL-rregularized RL with non-parametric behavioral policy consistently outperforms methods that use alternative divergence metrics, as shown in the bottom plots of Figures 4 and 5.
5.3 Can the Pathology Be Fixed by Improved Parametric Uncertainty Quantification?
To assess whether the success of non non-paraman con-paraettric be- havioral reference policies is due their predictive variance estimates-4e suggested by Proposition 1- or due to better generalization from their predictive means, we perform an ablation study on the predictive variance of the behavioral policy. To isolate the effect of the predictive variance on optimization, we perform online training using behavioral policies with differ- ent predictive variance functions (parametric and non- parametric) and identical mean functions, which we to be the predictive mean of the GP posterior (which achieves success rate of -80%). the pathology iden- tified in Proposition I can be remedied by commonly used parametric uncertainty quantification methods, we would expect the parametric and non parametric behav- ioral policy variance functions result in similar on- line policy success rates. We consider the challenging ttoor-binary-v00" environment for this ablation study.
10
door- Shor-binary-vl Ablations
10
10
21 0K
50K
100K
150K
200K
Timesteps
Variance Type
Nonpwaaettric GP (Exeat Posterior)
Parametrie Caussian NN (Ensemble)
Parametric Caussian NN (MC-D BNN)
Paramettri Gaussian NN (MLE. + Entropy)
Paramotric Ganssian (MLF)
Figure Post-online training success rates with
different behavioral policy variance functions.
Parametric uncertainty quantification is insufficient. Figure shows that parametric variance functions result in online policies that only achieve success rates of up 20% and eventually deteriorate, whereas the non-paramctric variance yields an online policy that achieves a success rate of nearly 100%. This finding shows that commonly used uncertainty quantification methods, such as deep ensembles or BNNs with Monte Carlo dropout, do not generate sufficiently well-calibrated uncertainty estimates to remedy the pathology, and better methods may be needed [9, 39, 41].
Lower-bounding the predictive variance does not remedy the pathology. The predictive variance of all MLEE-based and ensemble behavioral reference policies in all experiments are bounded away from zero at a minimum value of ss 10-2. Hence, setting a floor on the variance is not sufficient to prevent pathological training dynamics. This result further demonstrates the importance of accurate predictive variance estimation in allowing the online policy to match expert actions in regions of the state space with low behavioral policy predictive variance and explore elsewhere.

---

5.4 Can a Single Expert Demonstration Be Sufficient to Accelerate Online Training?
To assess the usefulness of non- parametric behavioral reference poli- cies in settings where only few ex- pert demonstrations are available, we investigate whether the difference in
15 Expert Demonstrations
One Expert Demonstration
1000
10000
X000
5000
performance between online policies trained with non-paranan con-paamettric and para- metric behavioral reference policies, respectively, is exacerbated the fewer expert demonstrations are available. To answer this question, we consider the "HalfCheetahh-2" environment and compare online policies trained with different behavioral reference pareciis-non-ppaaaae
100K
B00
B00K
400K
300K
100K
300K Timesteps
200K
400k
500k
Timcsteps
X-PPAC (Ours)
MC-Dropout Behavioral Policy
Eusemble Behaviocaa Policy
No Rehavioral Palley (SAC)
Figure 7: Returns during online training with different behav- ioral policies and varying amounts of expert demonstration data on "HalfCheetah-v2".
GPS, deep ensembles, and BNNs with Monte Carlo dropout either from 15 expert demonstrations (i.e., 15 state-action trajectories, containing 15,000 samples) or from a single expert demonstration (i.e., single state-action trajectory, containing 1,000 samples).
A single expert demonstration is sufficient for non-paramctric behavioral reference policies. Figure shows the returns for online policies trained with behavioral reference policies estimated from the full dataset (top plot) and from only single expert itate-action trajectory (bottom plot). On the full dataset, we find that all three methods are competitive and improve on the prior state- of-the-art but that the GP behavioral policy leads to the highest return. Remarkably, non-parametric GP behavioral policies perform just as well with only a single expert demonstration as with all 15 (i.e., with 1,000 data points, instead of 15,000 data points). These results further emphasizes the usefulness of non-parametric behavioral policies when accelerating online training with expert demonstrations-eever when only very few expert demonstrations are available.
5.5 Are Non-Paramitric GP Behavioral Reference Policies Too Computationally Expensive?
Table I presents the time complexity of KL-regularized RL under non-parametric GP and parametric neural network behavioral reference policies, as measured by the average time elapsed per epoch on the fhoor-binary-voe and "HalfCheetah-v2"" environments. One epoch of online training on "door- binary-v0" and "HalfCheetah-v2" requires computing the KL divergence over 1,000 mini-batches of size 256 and 1,024, respectively. The time complexity of evaluating the log-density of a GP behavioral reference policy-needee needed for computing gradients of the KI. divergence during online training- scales quadratically in the number of training data points and linearly in the dimensionality of the state and action space, respectively. As can be seen in Table 1, non non-parametric GP behavioral reference policies only lead to a modest increase in the time needed to complete one epoch of training while resulting in significantly improved performance as shown in Figures and 5.
Table 1:
Time per epoch under different behavioral reference policies for expert demonstration data of varying size computed on GeForee RTX 3080 GPU. The first and second value in each entry of the table give the time required when using single parametric neural network and a GP behavioral reference policy, respectively.
Dataset
1,000 Data Points
5,000 Data Points
15,000 Data Points
HalfCheetah--2
12.00s/ / 16.06s 19.62, 23.78s
11.59s / 18.31s 19.62s / 33.62s
12.00s / 46.54s
boor-binary-v0
6 Conclusion
We identified previously unrecognized pathology in KL-regularized RL from expert demonstrations and showed that this pathology can significantly impede and even entirely prevent online learning. To remedy the pathology, we proposed the use of non-parametric behavioral reference policies, which we showed can significantly accelerate and improve online learning and yield online policies that (often significantly) outperform current state-of-the-art methods on challenging continuous control tasks. We hope that this work will encourage further research into better model classes for deep reinforcement learning algorithms, including and especially for reinforcement from image inputs.
10

---

Acknowledgments and Disclosure of Funding
We thank Ashvin Nair for sharing his code and results, as well as for providing helpful insights about the dexterous hand manipulation suite. We also thank Clare Lyle. Charline Le Lan, and Angelos Filos for detailed feedback on an early draft of this paper, Avi Singh for early discussions about behavioral cloning in entropy-regullarized RL, and Tim Pearce for useful discussion on the role of good models RL. TGJR and CL are funded by the Engineering and Physical Sciences Research Council (EPSRC). TGJR is also funded by the Rhodes Trust and by Qualcomm Innovation Fellowship. We gratefully acknowledge donations of computing resources by the Alan Turing Institute.
References
[1] Michael Bain and Claude Sammut. framework for behavioural cloning. In Machine Intelli- gence 15, pages 103-129, 1995.
[2] Philip J Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 619-629. PMLR, 18-24 Jul 2021.
[3] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, pages 182-189. JMLR Workshop and Conference Proceedings, 2011.
[4] Ivan Bratko, Tanja Urbancic, and Claude Sammut. Behavioural cloning: phenomena, results and problems. IFAC Proceedings Volumes, 28(21):143-144 1995.
[5] Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement learning from demonstrations through shaping. In Twenty-fourth International Juint Conference on Artificial Intelligence, 2015.
[6] Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer in offline RL, 2021.
[7] Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. Proceedings of the 29th International Coference International Conference Machine Learning, ICML' 12, page 179-186, Madison, WI, USA, 2012. Omnipress.
[8] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges real-world reinforce- ment learning, arXiv preprint
12901, 2019.
[9] Sebastian Farquhar, Michael A. Osborne, and Yarin Gal. Radial bayesian neural networks: Beyond discrete support in large-scale bayesian deep learning. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1352-1362. PMLR, 26-28 Aug 2020.
[10] Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized RL. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[11] Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.053/1 2018.
[12] CW Groetsch. The theory of Tikhonov regularization for Fredholm equations. Boston Pitman Publication, 1984.
[13] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications, 2019.

---

[14] Hado V. Hasselt. Double Q-learning. In J. D. Lafferty, C. K. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, Advances Neural Information Processing Systems 23, pages 2613-2621, 2010.
[15] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey, Journal of artificial intelligence research, 4:237-285, 1996.
[16] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21810- 21810-21824 21823. Curran Associates, Inc., 2020.
[17] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pages 1014, 2000.
[18] George Konidaris, Scott Kuindersma, Roderic Grupen, and Andrew Barto. Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research, 31 (3):360-375, 2012.
[19] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems 32, pages 1174--17994 2019.
[20] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Informa- tion Processing Systems 30, pages 6402-6413, 2017.
[21] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
[22] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[23] Cong Lu, Philip J. Ball, Jack Parker-Holder, Michael A. Osborne, and Stephen J. Roberts. Revisiting design choices in model-based offline reinforcement learning, 2021.
[24] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
[25] Kevin P. Murphy. Machine learning: A Probubilistic Perspective. MIT Press, 2013.
[26] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration reinforcement learning with demonstrations. In 2018 IEEE International Conference on 2018.
Robotics and Automation (ICRA), pages
[27] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online rein- forcement learning with offline datasets, 2020.
[28) Nicolás Navarro-GGuerreoo, Cornelius Weber, Pascal Schroeter, and Stefan Wermter. Real-world reinforcement learning for autonomous humanoid robot docking. Robotics and Autonomous Systems, 60( 1): 1400-1477, 2012.
[29] Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, 100, page 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc.
[30] Aldo Pacchiano, Jack Holder, Yunhao Tang, Krzysztof Choromanski, Anna Choro- manska, and Michael Jordan. Learning to score behaviors for guided policy optimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7445-7454. PMLR, 13-18 Jul 2020.
12

---

[31] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage Advantage-weighted regression: Simple and scalable off- off-po reinforeement learning, 2019.
[32] Jan Peters, Sethu Vijayakumar, and Stefan Schaal. Natural actor-critic. In European Conference on Machine Learning, pages 280-20 1:00-291..00 Springer, 2005.
[33] Joaquin Quiñonero Candela and Carl Edward Rasmussen. A unifying view of sparse approxi- mate Gaussian process regression. Mach. Learn. Res., 6:1939-1959, December 2005, ISSN 1532-4435.
[34] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
[35] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018.
[36] Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
[37] Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII, 2012.
[38] Michael T Rosenstein, Andrew G Barto, Jennie Si, Andy Barto, and Warren Powell. Supervised actor-critic reinforcement learning. Learning and Approximate Dynamic Programming: Scaling Up the Real World, pages 359-380, 2004.
[39] Tim G. J. Rudner, Zonghao Chen, and Yarin Gal. Rethinking function-spant variational inference in Bayesian neural networks. In Third Symposium on Advances in Approximate Bayesian Inference, 2021.
[40] Tim G. J. Rudner, Vitchyr H. Pong, Rowan McAllister, Yarin Gal, and Sergey Levine. Outcome- driven reinforcement learning via variational inference. In Advances Neural Information Processing Systems 34. 2021.
[41] Tim G. J. Rudner, Freddie Bickford Smith. Qixuan Feng, Yee Whye Tch, and Yarin Gal. Continual learning via fumetion-space variational inference. In ICML Workshop on Theory and Foundations of Continual Learning. 2021.
[42] Stefan Schaal et al. Learning from demonstration. Advances in neural information processing systems, pages 1040-1046, 1997.
[43] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft Q-learning. arXiv preprint 1rxivv1704.0644 2017.
[44] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
[45] Alex Smola, Arthur Gretton, Le Song. and Bernhard Schölkopf. A hilbert space embedding for distributions. In Marcus Hutter, Rocco A. Servedio, and Fiji Takimoto, editors, Algorithmic Learning Theory, pages 13-31, Berlin, Heidelberg, 2007. 2007.Sprine Springer Berlin Heidelberg.
[46] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
[47] Gerald Tesauro. Temporal difference learning and td- Id-gammon. Communications of the ACM, 38(3):58-68) 1995.
[48] Emanuel Todorov. Linearly-solvable markov decision problems. In B. Schölkopf, J. Platt, and Hoffman, editors, Advances Neural Information Processing Systems, volume 19, pages 1369- 1376. MIT Press, 2007.
13

---

[49] Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. Improving deterministic uncertainty estimation in deep learning for classification and regression, 2021.
[50] Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gor- don Wilson. Exact Gaussian processes a million data points. In H. Wallach, H. Larochelle, A. Beygelzimer, d' Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, 2019.
[51] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911: 1161, 2019.
[52] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14129-144 14142. Curran Associates, Inc., 2020.
[53] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization, 2021.
14


Reviews:
[{"review_id": "iTCNnGFDKlK", "reviewer": "Reviewer_Vbhn", "summary": "This paper presents a previously unrecognized pathology in KL-regularized RL with expert demonstrations. Specifically the commonly used parametric behavioral policies suffer from collapse of predictive variance at states far from the demonstrations. This collapse of variance hinders the algorithm’s ability to learn effectively. The authors then propose to use non-parametric behavioral policies and demonstrate its effectiveness in several control tasks.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Estimating predictive variance/uncertainty is an important topic that influences a broad range of ML research. Understanding and estimating the predictive variance in RL is in particular challenging due the instability of deep RL. This work provides an insightful analysis on KL-regularized RL algorithms and how the behavioral policy class influences the online learning process. The paper is well-written with clarity. The claims in the paper are supported with empirical experimental results and the designed experiments successfully highlight the problem of collapsed predicted variance at states not in demos in KL-regularized RL. This observation is novel and inspires the authors to propose a class of new algorithms that use non-parametric policies, which show comparable or better performance than prior methods in several high-dimensional control tasks. \n\nIt would be nice if the authors could also comment on how the non-parametric policy class would compare with parametric policies regularized with MC-dropout [1], since MC-dropout has also been considered as a way to do bayesian approximation on the network weights, and is often compared with Ensembles in related literature for estimating aleatoric uncertainty of neural network. In addition, in the paper the authors only commented on how the problem setting is friendly for non-parametric methods, it would be interesting to know what the actual computational trade-offs are, if any, for using non-parametric policies.\n\n[1] Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). PMLR. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations of the proposed method and how this work could influence the research community to focus effort on better understanding of how policy classes would influence learning in deep RL. The authors did not explicitly comment on potential societal impact. One possible societal impact the authors could mention is that accurately estimating the predictive variance of ML systems is critical for real-world risk-sensitive applications such as autonomous driving. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "S_K0Bfjlap4", "reviewer": "Reviewer_vDkn", "summary": "The authors identify a problem with KL-based reinforcement learning from expert demonstrations. Poor out-of-sample variance predictions of NN expert policies ends up unreasonably penalizing RL exploration outside the expert data distribution, potentially causing numerical instability. To remedy this they propose to instead use non-parametric expert policies based on a scalable GP approach.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This seems like a solid incremental improvement. They thoroughly analyze an important problem in previous work and propose a fix. RL from expert demonstrations is a relevant niche with real-world applications. The paper is very well-written and the experimental results appear to improve on prior state-of-the-art.\n\nSome comments:\n\n- L43: \"without ad-hoc design choices\" - Like what? this is both needlessly vague and ends up sounding a bit objectionable.\n\n- L178-185: I don't believe it is the limited capacity from its parameteric nature. Policies in RL are usually very simple and overparameterized. However, it is well-known that an NN trained with maximum likelihood will not predict out-of-distribution uncertainty well, simply because it hasn't been trained on that. Both the mean and variance predictions will be very poor outside the training distribution. It also sometimes goes to zero, but I'm not aware of a formal analysis of this, or convinced that it always happens. As you note in L200, ensembles are sometimes used for predictions, so they can't all always go to zero.\n\n- Figure 6: Clarify which \"Non-KL\" methods are used in the bottom row, I assume it is AWAC etc? It might also have been nice to compare against regular RL with pre-trained (BC) policies as a sanity check. \n\n- Section 5.3: Shouldn't you compare compute time against some RL algorithm *with* expert demonstrations? I understand that if it doesn't reach the reward thresholds without extra tuning, it becomes a bit apples to oranges, but this isn't ideal either. It's unclear how the overhead of the GP policy compares to the NN policy. It might be relevant to compare on the more complex benchmarks as well, as the simulator is usually the bottleneck in non-trivial RL applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequate for this type of paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "PLwybpUMpee", "reviewer": "Reviewer_hW6R", "summary": "The authors identify a potential failure in KL-regularized reinforcement learning in which small predictive variance of the target policy may lead to exploding gradients, possibly destabilizing gradient-based learning algorithms. The authors discuss an uncertainty collapse of parametric models under Maximum Likelihood estimation in OOD regions of the expert, resulting in the aforementioned pathology. Finally, they suggest to use non-parametric models of the behavior policy (in which uncertainty collapse is not present), showing improved performance on several continuous control benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors identify an interesting problem of KL regularized optimization and propose a solution to solve it. I find the problem interesting and the authors' solution to be well motivated and reasonable. In addition, the paper is written clearly, and the experiments were applied to reasonable tasks compared to similar / previous literature. The main limitations of the paper are as follows:\n\n1. The model of the paper focuses on KL-regularized reinforcement learning. There are various modifications to this model which may trivially solve the authors' suggested pathology, yet the paper does not discuss them at all.\n* First, KL-regularization is applied to the policies themselves and not the stationary distributions, d^\\pi. Algorithms such as GAIL and AIRL can be combined with reward functions to construct regularized algorithms which utilize the expert data through stationary distributions. Would the issues the authors suggest still exist in this case? I would be interested to see comparison to such algorithms (with addition of reward).\n* KL regularization is only one way to regularize the algorithm. Other f-divergences can be used which would not have the exploding gradient problem (e.g., TV-distance). Why do the authors focus on KL-regularized policies?\n\n2. Scalability. While I agree that simple tasks would require small amounts of data, I find limited scalability as a major drawback of the paper. I hope for a solution that scales well with the amount of data provided, so that harder tasks could also be achieved by the proposed method.\n\n3. Experimental Comparisons. As mentioned in 1 and 2 , the author's should compare their results to algorithms using stationary distributions (e.g., GAIL, AIRL), as well as other forms of f-divergences. I would also be happy to see comparison to CQL as well as DICE algorithms.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of the proposed approach is its scalability, which the authors discuss in Section 4.1 implicitly. I would be happy to see a more explicit discussion of this limitation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "OfbzcjO10b", "reviewer": "Reviewer_3qqH", "summary": "This paper studies the problem of KL-regularized reinforcement learning for locomotion tasks, where the KL regularization penalizes deviations from expert demonstrations.\n\nThis work makes the observation that fitting expert demonstrations to a certain class of conditional neural density models (e.g. neural networks that output the parameters of a Gaussian distribution), results in policies whose variance collapses for states that are different enough from the expert data. \n\nThe paper argues that this collapse causes instabilities in learning KL-regularized policies: for Gaussian policies, the KL penalty grows quadratically to infinity as the expert policy variance goes to 0. To address this issue, the paper proposes to instead compute the KL penalty by fitting expert demonstrations to models that do not suffer from this collapse, in particular Gaussian Process regression models. Gaussian Process models result in variance that increases (depending on the choice of kernel) for states that are far enough from the expert data.\n\nThe paper provides some analysis on why the variance collapse affects the RL optimization process, experiments that  aim to show an empirical relationship between expert policy variance, KL penalty and magnitude of the policy gradients, and a comparison between various methods that use KL regularization with an expert policy.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As summarized above, this paper presents a method for KL regularized RL, in which the KL penalty computation uses a behavioural policy prior fit from expert data. The proposed method uses Gaussian Process models for fitting the expert data. The paper is clearly written, proposes a simple approach and provides empirical evidence of its usefulness. \n\nIt is notable that, in the experiments presented in the paper, the resulting learned policies produce visually smooth behaviours when regularized with a smooth prior (even if the policy class is not smooth). The paper analyses the problem of variance collapse in KL regularized RL, showing the relationship between variance collapse (which is empirically observed) with optimization instabilities due to exploding gradients.\n\nSome things to improve:\n\nThe paper limits its comparison to one between conditional Gaussian models with ReLU networks and GP models with smooth stationary kernels. While this is enough for the main point of the paper (addressing the variance collapse), it doesn't necessarily mean that parametric models will have variance that will invariably collapse, particularly in the case where modelling uncertainty is introduced in the parametric model fitting (e.g. the comparison with ensembles). The paper does not need to answer how parametric models could be made to work in this setting, but the main text should be explicit about what is specifically being compared.\n\nAn useful comparison that is missing in this story is the effect of expert dataset size on the performance of the algorithms. Is it always better to use GP regression for fitting the prior as the expert dataset size changes? \n\nWhat kind of ensemble was used for the experiments (bootstrap, different random seeds/initialization)? Since the paper introduces a comparison with ensembles, another useful comparison of the method would be against an ensemble model, as you let the size of the ensemble grow.  In the context of the discussion about ensembles, it would be helpful to include a discussion about other models that could be applied in this setting (e.g. bayesian neural networks). and their limitations.\n\nA comment on the robotics experiments: in general, human expert demonstrations are not readily available for robot tasks. Collecting human expert data is still expensive and Perhaps the authors meant that there exists publicly available datasets for learning from demonstrations for robot manipulation tasks.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Section 4.1, which the authors point to as a description of the model limitations at the end of the paper, only mentions the cubic complexity of fitting GP models with exact inference. Further, this limitation is presented as not really being a limitation due to existing algorithms based on conjugate gradient solvers for exact inference with GPs. The paper, however, fails to mention other important limitations, in particular ones affecting the current implementation: how well will the method work with non-smooth data? with high dimensional input data such as images? How can the model be extended to multimodal expert behaviours? While the paper does not need to provide solutions to these issues, the discussion of limitations would be more helpful if it pointed out problems that are not easily addressable.\n", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There has been a split amongst the reviewers for this paper. Although each expressed interest in the pathology of KL-regularization stated by the author, there has not been a consensus about whether the author's approach to this is impactful. I would recommend the position of the majority, 3 reviewers have found the set of baselines and architectures attempted are suitable to explore the problem at hand, and the emirical results are promising enough to be of value to the community as written (and with promised helpful additions in the discussion). Therefore, I recommend acceptance.

I do also sympathize with the position of reviewer hW6R, in that I may not be likely to use KL-regularization to solve the RL-from expert problem as of this year and have concerns about scaling GPs for high-dimensional problems. While these concerns are relevant, and could inform the authors about helpful discussion in the final version or their next papers, I still believe the contribution of a focused analysis of this widely used form of KL-regularization is sufficient in current form.
Decision: Accept (Poster)