Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
Authors: Yujie Zhu
Charles Alexander Hepburn
Matthew Thorpe
Giovanni Montana
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Learning from Demonstrations
Primary area: reinforcement_learning

Abstract: In reinforcement learning with sparse rewards, demonstrations can accelerate learning, but determining when to imitate them remains challenging. We propose Smooth Policy Regularisation from Demonstrations (SPReD), a framework that addresses the fundamental question: when should an agent imitate a demonstration versus follow its own policy? SPReD uses ensemble methods to explicitly model Q-value distributions for both demonstration and policy actions, quantifying uncertainty for comparisons. We develop two complementary uncertainty-aware methods: a probabilistic approach estimating the likelihood of demonstration superiority, and an advantage-based approach scaling imitation by statistical significance. Unlike prevailing methods (e.g. Q-filter) that make binary imitation decisions, SPReD applies continuous, uncertainty-proportional regularisation weights, reducing gradient variance during training. Despite its computational simplicity, SPReD achieves remarkable gains in experiments across eight robotics tasks, outperforming existing approaches by up to a factor of 14 in complex tasks while maintaining robustness to demonstration quality and quantity. Our code is available at https://github.com/YujieZhu7/SPReD.

Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
Yujie Zhu', Charles A. Hepburn', Matthew Thorpe', and Giovanni Montana 1.2
Department of Statistics, WWrwwick Manufacturing Group University of Warwick CV4 7AL (yujie. zhu, charles hepburn, matthew matthew.thorpe, g g.montana) @warwick. ac uk
Abstract
In reinforcement learning with sparse rewards, demonstrations can accelerate learning, but determining when to imitate them remains challenging. We propose Smooth Policy Regularisation from Demonstrations (SPReD), a framework that addresses the fundamental question: when should agent imitate a demonstra- tion versus follow its own policy? SPReD uses ensemble methods to explicitly model Q-value distributions for both demonstration and policy actions, quantifying uncertainty for comparisons. We develop two complementary uncertainty-awei methods: probabilistic approach estimating the likelihood of demonstration superi- ority, and an advantage-based approach scaling imitation by statistical significance. Unlike prevailing methods (e.g. Q-filter) that make binary imitation decisions, SPRcD applies continuous, uncertainty-proportinna proportional regularisation weights, re- ducing gradient variance during training. Despite its computational simplicity, SPReD achieves remarkable gains in experiments across eight robotics tasks, out- performing existing approaches by up to factor of 14 in complex tasks while maintaining robustness to demonstration quality and quantity. Our code is available at httts:///githut com/ eeiiieehu///aaa
Introduction
Reinforcement learning (RL) has proven effective for sequential decision problems in robotics [ and games 3 4] yet complex tasks require extensive iterations that introduce risks and costs when learning must occur in real-world settings. To accelerate learning, researchers have incorporated pre-collected demonstrations [5 7 particularly valuable for sparse-reward environments where agents struggle with minimal feedback. While dense reward functions could help, they require domain expertise and become increasingly difficult to design for complex dynamics. Sparse rewards, though simpler and less susceptible local optima [5] significantly intensify the exploration challenge macienggeemainggggggeee
making demonstrations essential for effective learning.
The field has produced numerous upproaches leverage demonstrations in RL, with varying degrees of success. Early methods employ prioritised replay mechanisms but require extensive parameter tuning and struggle with demonstration quality adaptation [5] g Other techniques
weighted behaviour cloning (BC) with predetermined decay factors that reduce demonstration influence over time regardless of their continuing utility 77 More recent methods attempt address limited or suboptimal demonstrations through reduced Q-values for undemonstrated actions [9] or variance estimates as weighted guidance signals [10], but are restricted to discrete action spaces. Jing al.
III treat demonstrations as soft constraint on policy exploration, formulating a constrained policy optimisation problem. Although they reduce overhead by applying local linear search its
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

dual, the approach still involves considerable computational complexity. Efficiently leveraging such demonstrations remains an underexplored problem.
The Q-filter approach 6 brought significant advancements by selectively applying behaviour cloning only when demonstration actions yield higher Q-values than policy actions. This intuitive method has become foundational in RL with demonstrations, yet our analysis reveals two critical limitations: relies on single point estimates without accounting for estimation uncertainty, and it makes binary decisions that introduce high variance during training. These limitations become particularly problematic with limited or suboptimal demonstrations common scenarios in practical applications.
To directly address these fundamental limitations of Q-filter, we propose Smooth Policy Regularisation from Demonstrations (SPReD), which reformulates demonstration utilisation as distributional comparison problem. Our approach employs ensembles of critics to model two distinet Q-value distributions: one evaluating demonstration actions and another evaluating current policy actions. Unlike Q-filter's binary approach, SPReD applies continuous weights to the behavioural cloning loss, determining how strongly each demonstration action should influence policy updates. These weights scale smoothly with our statistical confidence that demonstration actions outperform the current policy. We develop two complementary methods for this value distribution comparison: a probabilistic weighting based on the likelihood that demonstration actions are superior, and an exponential scaling inspired by weighted behavioural cloning that calibrates imitation strength based on the statistical significance of advantages. Both methods yield state-adaptive regularisation that enables smoother policy learning across diverse environments.
Our theoretical analysis establishes several key properties: continuous weights strictly reduce policy gradient variance compared to binary decisions; they adapt systematically to uncertainty levels; and they progressively diminish influence from suboptimal demonstrations as policy performance improves. These properties provide sound foundation for the empirical improvements we observe. Through experiments across eight robotic tasks, we demonstrate that SPReD consistently outperforms existing methods, achieving up 14x success rates with the same interaction steps in complex manipulation tasks like block stacking (0.920 vs. 0.064) and significant improvements even with severely limited or suboptimal demonstrations. Despite using ensembles, our implementation main- tains efficiency comparable to standard methods by leveraging the same critic networks for both target computation and uncertainty estimation. Our results demonstrate that our uncertainty-ave -aware approach to comparing value distributions enables effective learning from limited or noisy demonstrations.
Related work
Expert demonstrations enhance online RL across various ap- proaches [1]. Atkeson and Schaal 13 pioneered demonstration-aasse task model and reward function learning as initialisation for policy learning. More advanced methods leverage demonstra- tions throughout the entire learning process. DDPGfD [3] and DQFD [ utilise prioritised replay buffers combining demonstrations and interactions, but require extensive parameter tuning and lack mechanisms for handling suboptimal demonstrations. DAPG 7 implements an on-policy method with weighted BC loss based the advantage function, but its influence diminishes through non-adaptive decay factor regardless of demonstration quality. Our method differs by employing weighted BC with adaptive Q-value comparisons specific to each state-action pair, enabling smooth regularisation that effectively incorporates both expert and suboptimal demonstrations--dddee addressing a key limitation of previous approaches.
RL from expert demonstrations.
RL from suboptimal demonstrations. Recent research focuses on handling imperfect demonstra- tions adaptively. Nair et al. [6] introduced the Q-filter by incorporating demonstrations in DDPG without pretraining, using BC loss filtered by estimated Q-values selectively imitate demonstrators. Our ensemble method directly enhances this concept through uncertainty quantification. Other approaches include Assisted DDPG [14, which relies on external controllers, LIDAR [1), which considers the advantage of demonstrations, and NAC (9, which reduces Q-values of unobserved demonstration actions discrete spaces. While some methods require demonstration pretraining (9) our approach remains sample efficient without this step. Alternative approaches include constrained policy optimisation [T] BQND's pariance-based guidance [00) and RL.DD [66, which uses sy symmet- ric buffer sampling and ensemble critics with Layer normalisation. We select Q-filter and RLPD as

---

primary baselines given their comparable settings both handle suboptimal demonstrations without pretraining and provide state-of-the-art performance.
Offline-to-online RL. Offline RL enables learning policies that surpass static dataset performance
without online interactions [17 18 19 but remains constrained by dataset coverage and quality,
Offlneeto--nlinne RL addresses this through subsequent fine-tuning, facing challenges with inaccurate Q-value estimates for outrof-distribution state-action pairs (20]. IQL. [21] uses expectile regression to learn value function and performs advantage-weighted policy extraction to learn a policy without explicit bootstrapping from the policy. Methods like AWAC [22] implement fine-tuning through
implicit policy constraints, while others gradually relax BC constraints [23], employ ensemble pretraining with pessimistic Q-functions (20], or generate interactions from composite policies [24] Unlike our approach, these methods typically require extensive offline datasets for pretraining, whereas our method operates effectively with few demonstrations in a purely online manner.
Uncertainty in RL Uncertainty in RL is typically categorised as aleatoric (inherent environmental
randomness) or epistemic (model knowledge limitations) 25 While uncertainty quantification has proven valuable for pegloration-exploittaiii
balancing 26 27,, safety constraints [28], and
offline learning [29], its application to demonstration utilisation remains underdeveloped. Common approaches for uncertainty estimation include bootstrapping (30], ensemble techniques [31] and MC-
dropout (32), with some methods explicitly addressing both uncertainty types [33]. UWAC (34] an
offline RL method that weights critic and actor updates using dropout-baset uncertainty manage out- of-distriuution actions. Most online RL approaches employing uncertainty with demonstrations such as Active DQN (35), RCMP [36), and CHAT [37]- make binary decisions about demonstration usage. Our work differs by using ensemble- based epistemic uncertainty estimates to enable smooth, continuous regularisation based quantified confidence in demonstration superiority.
3 Preliminaries
Setup We consider the standard Markov decision process (MDP) framework M (S.A, P. R, y, Po) where an agent interacts with an environment E over discrete time steps. The initial state distribution is po. At each time step, the agent observes state s S, selects action a e A according to policy x, and receives reward = R(s, a) while transitioning to the next state s' according the environment dynamics P(s'ls, a). The transition tuples (s, a, r, s') are saved in replay buffer B with exploration noise added to actions, and mini-batcher are sampled from it for future learning. With discount factor y E [0, 1), the agent aims maximise the expected cumulative discounted return J = Brr.s,-EE,a [Ro] where R₂ = Si=t =I "r The state-action value function is defined as 9r (s, a) EE, [Rels, a), with the estimate (s, a). Additionally, we assume access to a set of demonstrations D = {(s, ad, ra. {'}] collected from an unknown policy TD, which may be suboptimal. Our goal is to effectively leverage these demonstrations to accelerate learning.
(s,
TD3 and HER experiments, we implement SPReD with the state-of-the- art TD3 (38) algorithm, which provides an ideal foundation due to its model-free nature and suitability for continuous state and action spaces. Crities are updated by minimising the mean squared error:
Our method is compatible with any off- off-iolicy actor-critic RL algorithm. For our
((0,) = -(aa))=(rr + min Qo,(s', (s.
1=1
where dual critic networks with target networks are employed in the target to address the overesti- mation bias (39. 38, and ä is the action selected by the target actor with Gaussian noise. The actor parameters & are updated using deterministic policy gradient 440 to maximise:
(( E(sa)) Qe, (s. (s)))).
For environments with sparse rewards, we employ Hindsight Experience Replay (HER) 41 to address exploration challenges in goal-conditioned tasks. HER stores transitions twice with desired goals or actually achieved goals, enabling learning from unsuccessful episodes. The goals are appended to states as inputs for actor and critic networks.
3

---

Q-filter To leverage demonstrations for accelerating learning, the Q-filter [6] stores demonstration data in separate buffer B,, During each training step. mini-batch of size ND is sampled from the demonstration buffer in addition to transitions from the standard replay buffer. Both are used for critic updates. The Q-filter technique incorporates a selective BC loss that only imitates demonstration actions when they are estimated to be superior to the current policy's actions:
(oc( (s (sa)) - aall 2
Q Q(aa Qlenaaa))
where 1l is the indicator function that equals when the single Q estimate of the demonstration action higher and 0 otherwise. The actor network is then updated by optimising the combined objective, -Xi + d2 LBC, where 2 and 22 are hyperparameters that balance policy improvement against demonstration imitation.
4 Methodology
The Q-filter mechanism 66] represents state-of-the-at approach for demonstration utilisation in reinforcement learning, particularly for continuous control tasks with sparse rewards. While effective, this approach makes binary decisions to accept or reject demonstrations based on point estimates of Q-values. We hypothesised that such binary filtering mechanisms fundamentally limit learning efficiency in two ways: they fail to account for estimation uncertainty inherent temporal difference learning, and they introduce gradient discontinuities during policy updates. These limitations would theoretically cause increasingly unstable learning as policies approach optimality, precisely when nuanced guidance from demonstrations becomes most valuable.
To address these limitations, we reformulate demonstration-oe based regularisation by directly comparing the quality of demonstration actions versus current policy actions while accounting for uncertainty. For each state-action pair in the demonstration buffer, we compare two distributions: the distribution of Q-values for the demonstration action {Qi(sa, aa) )m and the distribution of Q-values for the current policy action {Qa, (+())))) )= where E 1. indexes our ensemble of m independent critic networks. The variability across these ensemble estimates captures the epistemic uncertainty in our value approximation. The selection of uncertainty measure is discussed Appendix FF
Smooth policy regularisation from demonstrations Rather than using point estimates as in Q- filter, we leverage the full distributions of Q-values from our ensemble to determine how strongly each demonstration should influence policy learning. The key insight of our approach is that demonstration influence should vary continuously with our confidence in its superiority. Rather than making binary decisions, we introduce a state-adaptive weight p E [0, 1] that quantifies our statistical confidence that demonstration action outperforms the current policy action. This weight modulates a behaviour cloning loss:
LWBC (Bsa [p( (=)))=)(ss)) azl19)
where higher p values (the dependency on (sa, ad) is dropped in notation for simplicity) apply stronger regularisation toward demonstration actions when and lower values reduce imitation pressure when demonstrations appear less valuable.
more confident in their superiority,
Following the TD3 framework (38], we then update the actor network by combining standard deterministic policy gradient with this weighted behaviour cloning term:
( = - (s)))) + A2
This continuous weighting mechanism creates a smooth regularisation effect that adapts to the varying quality of demonstrations while accounting for uncertainty in value estimates. Despite its computational simplicity, this approach offers theoretical benefits (see Section 5) and remarkable empirical performance (see Section 66.
The central challenge now becomes:
how should we compute this weight p by comparing two value distributions? We present two complementary approaches for this distributional comparison: probabilistic method that estimates the likelihood of demonstration superiority, and an exponential method that scales imitation strength based on the statistical significance of advantages. Despite their different formulations, these methods are theoretically connected, with similar behaviour in highhunnertainty regimes we demonstrate in Property 5.3

---

SPReD-P: Probabilistic advantage weighting Our first method, SPReD-P, frames the above com- parison as probabilistic inference problem. Following common practice in uncertainty quantification, we model the Q-value estimates from our ensemble as Gaussian distributions 27 42 43
Q(sa. ~ N(Q((s. Q(sa, ~(((s)))))(((( TT()))))))))))))))))))))))))))))
where Q(s2, ad), (sa. To (sa)) and 0². 6² represent the empirical mean and variance of Q-value estimates across the ensemble. With these distributions established, compute the probability that the demonstration action outperforms the current policy action:
((()) - Q(sa, Te (sa))
pp P(Q ad) Q(sa, (s(s))))
Vol +0
where & represents the cumulative distribution function of the standard normal distribution.
This formulation creates continuous spectrum of imitation strengths that naturally adapts certainty levels throughout training. Early in training when Q-value estimates have high variance, probabilities tend toward intermediate values (#s 0.5), allowing partial learning examples. As uncertainty decreases with more training, the probabilities become more decisive, ap- proaching the binary case only when uncertainty becomes negligible: see Property [.] and empirical evidence in Appendix F The Gaussian assumption provides a computationally efficient approxi- mation while remaining analytically tractable; empirical comparisons in Appendix F confirm this assumption is plausible in practice.
from uncertain
SPReD-E: Exponential advantage weighting While SPReD-P provides principled probabilistic approach, we now introduce SPReD-E, which focuses on the magnitude of improvement rather than its likelihood. This complementary method scales imitation strength based on how significantly demonstration outperforms the current policy relative to estimation uncertainty. The core insight is that imitation should be proportional the size of the advantage, accounting for Q-value variability.
As before, using our ensemble of critics, we generate two distributions Q-vuuu for each state in the demonstration buffer: the distribution of demonstration Q-values and the distribution of current policy Q-values. To quantify how much better the demonstration is compared to the current policy, need to derive advantage measure A that respects the distributional nature of our estimates.
To simplify notation, let μ and y represent the Q-value distributions under the current policy and demonstration, respectively. Heuristically, we aim to define weight pæ that follows the current policy when its Q-values exceed those of demonstrations, and increasingly imitates demonstrations as their relative advantage grows. If the Q-value then we could define PE as a function of the difference, say A, between the values, for example pg A/3 1. To treat measures we look at comparing x in the support of μ to y in the support of v. Formally, this is done through a transport map T that rearanges fs to form v (which can be written V == Ty == == (-)).. We define = - - x) du(x)) to be the average difference between Q-values given the map T. Since (by change of variables) we can write A = Ey [Q] E, [Q]. there no dependence on and A is simply the difference between means. This short argument justifies computing the advantage directly the difference of mean Q-values:
single value (that is, a Dirac distribution),
= Emg
(s. ad)l - Ejg(m] [Q: (sa, ((s)))]
Having derived this advantage measure, we transform it into a weight using an exponential function inspired by advantage-weighted behavioural cloning 44 and policy improvement techniques:
PE A/B
where 3 controls sensitivity to advantage magnitude. We use a proportion of the interquartile range (IQR) of Q-value distributions as capture uncertainty in advantage estimates, providing robustness against outliers while achieving state-adaptive normalisation. This formulation is clipped to the range [0, 1] to ensures zero weight for inferior demonstrations (A < 0), proportional weighting for uncertain cases (small [A), and strong imitation for clearly superior demonstrations (large A). The subtraction of creates natural zero-threshold exactly when demonstration and policy actions equally valuable.
5

---

Theoretical properties
We now establish the theoretical foundations of SPReD, examining how both weighting mechanisms adapt to varying uncertainty conditions and demonstrating their advantages over binary filtering; full proof's are provided in Appendix B and empirical evidence are provided in Appendix F Our analysis focuses on key properties that characterise the behaviour and benefits of continuous weighting.
First, we formalise the fundamental advantage of continuous weights over binary decisions:
Lemma 5.1 (Gradient-variance gap), Assuming (AI) gradient norms are bounded and (A2) demon- strations are independently sampled, let Then
Xx 1 9k-
Yk Pk: 9k. 9k
V ((()) ax 1l?
Var I
Σn
1 ND
Xx Σx
Var
where 1, represents binary filtering decisions, ps [0,1] represents continuous weights, and ND is the batch size. Strict inequality holds if P(0 < pk < 1] O.
This result establishes that SPReD's smooth weighting produces BC gradient estimates with lower variance than binary Q-filter approaches. This variance reduction directly improves training stability and sample efficiency 45 46 47 48) particularly in the early stages of learning when policy updates are most sensitive to demonstration influence.
Next, we characterise how our weights respond to different uncertainty conditions:
Property 5.1 (Adaptive behaviour). Assume B = aB where 3 is the IQR of the mixture model with components (sa, a.)) and Q(sa, (sa)). ). Let 03 and o² be the variances and be the difference of
means under assumptions given in Appendix B As this variance varies, our weights satisfy:
(i) High-certainty: If 62 63 0 then pp + 1 (with pp 0.5 if and pe +
clip(e- - 1, A > 0 (with pE = 0 if f < 0).
2/g2+0 + à + oo then pp - 0.5 and pæ 0. (ii) High-uncertainge o
This property demonstrates how both methods adapt to uncertainty: approaching binary filtering when confident (accepting only positive advantages, with pg for negative advantages), while becoming conservative under high uncertainty. This adaptive behaviour is crucial for robust learning throughout training.
The following property addresses how SPRcD handles potentially suboptimal demonstrations as learning progresses:
Property 5.2 (Diminishing weight on suboptimal demonstrations). Let Q, be the Q-value distribution corresponding the policy Th Assume n² is the optimal policy with Q-value function Q" We assume that there is no uncertainty in Q* so that Q" is single value function (not distribution). Let (sa, aa) satisfy Q* (sa, a.) < Q" (s (sa)). Assume (A3) the mean Qe(s, a) of the random variable Qs(s, a) converges Q" (s,a) a) and the variance [ōa]i 0 uniformly over actions (so that the policy We is in a sense converging to the optimal policy x ). Then
lim pr(sd, ad) = 0, lim pe(sa,aa) = 0. t00
t+00
Crucially, this property enables SPReD to autonomously down-weight inferior demonstration actions as the policy improves and uncertainty decreases, allowing the agent to filter out misleading informa- tion and potentially surpass the performance of provided demonstrations without requiring explicit scheduling or quality estimation.
Finally, we establish fundamental connection between our two weighting schemes:
Property 5.3 (Parameter scaling relationship). When advantage magnitudes are small compared to estimation uncertainty (|A)/a where a = Vo + +Z²), Taylor expansion yields:
1 Pr
A + ((AAo))), PE PE= - (
+
2
This reveals that setting 3 av 2.5a causes both methods to exhibit similar rates of change with A in high-uncertainty scenarios, precisely when smooth regularisation is most beneficial. Under

---

this parameterisation. both approaches implement proportional forms of uncertainty- caution up to constans, differing only in higher-order terms.
This relationship reveals that our exponential weighting provides a non-parametric alternative that achieves similar uncertainty-awer adaptation without requiring Gaussian assumptions, while natu- rally placing greater emphasis on demonstrations with larger advantages. The theoretical connection also provides principled guidance for parameter selection (sce Appendix B for more details).
6 Experimental results
Environments and tasks We evaluate SPReD on eight challenging robotics tasks from OpenAI Gym's Fetch and Shadow Dexterous Hand environments [49), simulated in MuJoCo [50]. These environments feature sparse binary rewards (feedback only upon goal completion) and complex multi-goal structures, creating significant exploration challenges that make them particularly suitable for demonstration-baset learning. The Fetch tasks utilise 7-DoF robotic arm with parallel gripper for pushing, sliding, bock-and-placee and block stacking operations of increasing difficulty. For the stacking tasks, we use implementations and expert policies from Lanier [51]. The Shadow Hand tasks represent substantially higher complexity, requiring a 24-DoF anthropomorphic hand to achieve precise rotational control of objects (block, egg, pen) despite high-dimensional action spaces and control noise sensitivity. We exclude the trivial FetchReach task and use 1000 demonstration episodes for the challenging 3-block stacking task, with 100 demonstrations for all other tasks. See Appendix[D) and Appendix E for additional details about environments and demonstrations. We also experiment on locomotion tasks
(22
with dense rewards for more evaluation domains in Appendix
Baselines We evaluate SPReD against several state-of-the-ar baselines, all implemented with HER for fair comparisons. Our primary RL baseline is TD3 (38], which operates without demonstration utilisation. We include EnsTD3, an ensemble version using 10 critics where random pairs compute minimum target values and the ensemble mean guides actor updates (similar to REDQ (33), isolating ensemble effects without demonstrations. We compare against Q-fitter, the approach from Section 3 using binary-filtered BC with point Q-value estimates [6], and its ensemble variant EnsQ-filter that uses critic means for BC decisions, helping isolate benefits beyond simple ensembling. We evaluate against RLPD [6), a recent method leveraging ensemble critics with layer normalisation. implemented with author-recommender hyperparameters and input normalisation for optimal per- formance. We also include three variants of AWAC, a state-of-the- based on advantage weights: AWAC with no pretraining prior data, AWAC-p with pretraining, and AWAC-r without pretraining but keeping resampling demonstrations. Our proposed SPReD approach implemented in the two variants: SPReD-P and SPReD-E. Both methods leverage ensemble critics to quantify uncertainty but differ in how they transform uncertainty estimates into imitation weights. The pseudocode of our method and training details with computational cost analysis can be found in Appendix C The ablation tests on ensemble size, isolated contribution of different components, and normalisation constant of SPReD-E are presented in Appendix G
offline-to-onlig RL method
Main results Table presents success rates after million interactions (10 million for challenging
stacking tasks). quantifying sample efficiency across methods. Our incertainty-awan methods (SPReD-P and SPReD-E) consistently outperform standard Q-filter, its ensemble variant, RLPD and all variants of AWAC across all tasks, with SPReD-E achieving significantly higher success rates in seven of eight environments than baselines, and both methods are stable exhibiting relatively lower variance. AWAC struggles significantly in our setting with few demonstrations and environments with sparse rewards as SPReD) explicitly reasons about when demonstrations remain useful rather than relying on advantage estimates from limited data.
The sample efficiency advantage of our approach increases with task complexity and is most pro- nounced in intricate manipulation tasks. In FetchStack2, our methods achieve 14x the success rate of RLPD (0.920 vs. 0.064), despite using only 100 demonstrations. Even in the challenging sliding task, where demonstrations provide limited benefit due to sensitivity to precise force application, SPReD-E achieves a 50% higher success rate (0.240) than the next best method (0.160).
The entire learning curves in Figure D provide more comprehensive performance comparisons extend- ing beyond the 1-10 million interactions reported in Table showing longer-term behavior. Apart from the substantial improvement of sample efficiency, methods match exceed all baselines
7

---

Table I: Average success rate (with standard deviation) over 5 seeds after IM environment interactions (10M for stacking tasks). The highlighted results lie between the mean of the best performer and one standard deviation below it (i.e., if the best result is μ + o, all values 2 / - a are bold). The success rates of demonstrations range from 0.2-0.86 for standard tasks and 1.0 for stacking tasks.
Emriionnen
TD1
Q-fiter
En-Q-iller
RLPD
AWACC
AWAC-p
AWAC-r
SPR
SPReID-
FrsuaPask
20.272
0.722
0:227
8.015
0 0.044
0,568
8.016
2.112200:75
0,864 2.652
0,280
0007
0.020
0.964 z EU032
PrankSiste
01800 -1.052
0388
01070
0.104
0.01
0.180
6.112.0.:117
0.12
IUB4
FaschSuck2
00,06
0.000
0005900099
0,144 0.100
0,06 0.044
0.005
00006
0.008 00.05
=
(JI57
FascSttcck
10.0000.000
0.000 0.000
0.063 00.59
0.040 0.05
0.600 0.062
0.000 1100010.000
0.:5000.:00
0.000 0.000
0.246 1.246:10:1330
0.364 EL125
MangslaneBlok
IM02 +1255
0.136 0.008
0.700=0125
0,016
0:000
01018 -.01
01088 +0006
0.34
11864 = 0.065
0.833 tU09
Manipuaacaag
0.000:1000 0.000
0,000 0.000
0.055 0.054
0.035.1000
0.041
0.000
x 0.000
10,000
0.000.0..000
0.000 00000
0.218 2 0.052
0.067
ManipalaicPan
0.000
0.050
0.205
0.364 0,093
0.000 0.000
1 0.000
0.008 0.006
0.000:00.00
2 0.065
TD3
Q-fiter
EnsQ
RLPD
AWAC-p
ANAC-
SPReD-P
SPReD-E
FatchPush
FetchSiide
BerchFirkkndPaace
FatchStark?
1.0
1.0
1.0
1.0
0.9
Success
0.0
0.6
0.6
0.5
0.1
0.2
0.2
0.2
0.2
0.0
0.0
1.00
1.25
2,50
5.00
FelchStack3
MeriiuledeeBiccRolaaee
ManipulsteeggRotste
Manipulateeer Rotale
1.0
1.0
1.0
0.00
Success rete
0.8
0.8
0.8
0.48
0.6
0.8
0.6
0.4
0.4
0.24
0.2
0.2
0.2
0.0
0.0
0.0
0.00
18 24
30
0.0
0.5
1.0
1.5
2.0
5
10 15 Env steps (10")
20
0.0
1.2
2.4
3.6
4.5
6.0
Env steps (10")
Eny steps (10*)
EmV steps (10")
Figure 1: Performance comparison across eight robotics tasks. Solid lines represent mean success rates across 5 seeds, with shaded areas showing standard deviation. The learning curves are smoothed using a 5-point moving average. Horizontal dashed lines indicate the success rates of the demon- strations used for training. Our SPReD methods (red and brown) consistently outperform baselines across environments of varying complexity.
asymptotically. SPReD excels particularly in complex tasks like stacking, where demonstrations are most needed. In some extremely sensitive tasks where considerably suboptimal demonstrations provide limited guidance, EnsQ- filter and RLPD show competitive performance asymptotically but fail to adapt to different tasks.
From a computational perspective, instead of processing 10 critics sequentially, we stack their computations into batched tensor operations that execute in parallel with the vectorised critic. SPReD) maintains efficiency and achieves nearly the same throughput as TD3 despite using 5x more critics, while RLPD demands approximately twice the computation time (shown in Appendix Cl.
Impact of demonstration quality To systematically assess robustness to demon- stration quality, we conduct experiments at three dis- tinct quality levels in Fetch- PickAndPlace and Fetch- Push as illustrated in Fig- ure 2 and Appendix G
1D33
EnsTD3
Qittter
EnsQ fllter
RUPO
ANAC
AWACC
ARMC
SPReO
SPReD E
rate
0.8
0.0
stops (10*)
stops (100)
Env stops (1.0%)
Figure 2: Effect of demonstration quality in FetchPickAndPlace The demonstrations are expert, suboptimal and severely suboptimal from left to right with success rates shown as dashed lines.
With expert demonstrations, standard Q-filter performs
adequately since most demonstration actions merit imitation. However, its performance drops sharply as demonstration quality decreases, showing the weaknesses of binary filtering when dealing with uncertain Q-value comparisons. In the most extreme case-the the PinkAndPlace environment with only one expert trajectory among 99 random trajectories (the right plot in Figure standard Q-filter
8

---

actually performs worse than TD3 without demonstrations, effectively amplifying misleading exam- ples rather than filtering them. RLPD neither fully leverage the advantage of expert demonstrations nor discard the sub-optimality. In contrast, both SPReD variants maintain consistent performance across all quality levels. The automatic, continuous adaptation mechanism shown in Property [5.1 and Appendix | enaalees SPReD methods to extract maximum value from demonstrations of any quality level, maintaining superior sample efficiency across all test conditions without requiring explicit scheduling quality estimation procedures.
Impact of demonstration sample size available demonstrations (episodes) affects learning performance across methods in Figure g
We systematically evaluate how varying the sample size of
TD3
EnsTD3
O-filter
EnsO-fiiter
RLPD
AWAC
AWAG-P
AWAC-
SPReD-P
SPReD-E
5 demos
10 demos
20 demos
50 demos
0.8
846cess
0.4
0.0
Env steps (101)
Env steps (10")
Emv (10%)
Env (10%)
Figure 3: Effecc of demonstration size in FetchPickAndPlace The demonstrations collected from the same policy contain 5, 10, 20, or 50 episodes with success rates shown the dashed lines.
Our SPReD methods demonstrate superior performance even with extremely limited data (10 demon- strations), while maintaining improvement as sample size increased. With only 5 demonstrations, the ensemble Q-filter approach shows reasonable performance but exhibited substantially higher variance across random seeds, indicating less reliable learning. In general, SPReD-- demonstrates greater robustness limited sample sizes, while SPReD-E yields better asymptotic performance when provided with either larger sample sizes or higher-quality demonstrations, suggesting a potential trade-off between the two approaches depending on available demonstration characteristics.
Robustness To show the robustness of our SPReD methods in comparison to the baseline, we evaluate methods with noisy rewards, with results shown in Appendix SPReD remains the best method against the baselines, solving the task with noisy rewards where other methods fail to learn.
7 Conclusion
We have introduced SPReD, a framework for smooth policy regularisation from demonstrations that enhances RL in sparse-reward environments. Our key contribution is a principled approach to uncertainty-aware demonstration utilisation through ensemble-based Q-value modelling. We developed two complementary weighting methods: SPReD-P, which leverages probabilistic estimates of demonstration superiority, and SPReD-E, which scales imitation strength based on the statistical significance of advantages. Both methods significantly reduce policy gradient variance compared to binary filtering approaches. Our extensive evaluation across eight robotics tasks demonstrates substantial performance improvements, with up to 14x success rates in challenging manipulation tasks while maintaining robustness to varying demonstration quality and quantity. Based on our results, we recommend SPReD-E as the default choice for most applications, particularly for complex tasks, while noting that both methods achieve significant gains with minimal computational overhead.
Despite these advances, important limitations remain. While SPReD significantly accelerates learning in complex manipulation tasks, we observed that demonstration influence can sometimes slow later-stage learning in highly dexterous tasks, suggesting the need for automatic balancing between demonstration guidance and exploration. As with many deep RL approaches, practical considerations like hyperparameter sensitivity and sample complexity in extremely -igh--imensional tasks remain areas for continued refinement. Future work should address the automatic adaptation of demonstration influence throughout training, potentially through meta meta-learning approaches. Theoretical analysis of convergence properties would also strengthen the foundation of demonstration- based RL methods. Sim-to-real gap is another research direction as our evaluation based on the simulation. Finally, the simplicity of our approach makes readily applicable to other off- off-policy RI. algorithms beyond TD3, offering potential improvements across broader range of tasks and domains.

---

Acknowledgments
YZ acknowledges funding from the Department of Statistics, University of Warwick. CH acknowl- edges support from Innovate UK under the project "Fundamental Science and Outreach for Connected and Automated ATM: HyperSolver" (grant no. 1011 14820). MT would like to acknowledge the support of the Leverhulme Trust through the Research Project Award "Robust Learning: Uncertainty Quantification, Sensitivity and Stability" (RPG-2024-051) and the EPSRC Mathematical and Founda- tions of Artificial Intelligence Probabilistic AlHub (EP/Y007174/1) GM acknowledges support from a UKRI AT Turing Acceleration Fellowship (EP/V024868/1).
10

---

References
[1] Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1607-1612, 2010.
[2] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz. Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-00, 2020.
[3] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[4] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):444-88) 2016.
[5] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pictquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems with sparse rewards. arXiv preprint arXiv: 2017.
[6] Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojciech Zaremba, and Picter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292-6299. IEEE, 2018
[7] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 10087, 2017.
[8] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanetot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[9] Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.053/3. 2018.
[10] Fengdi Che. Bayesian Q-learning from Imperfect Expert Demonstrations. McGill University (Canada), 2021.
[11] Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 5109-5116 16, 2020.
[12] Jorge Ramírez, Wen Yu, and Adolfo Perrusquía. Model-free reinforcement learning from expert demonstrations: survey. Artificial Intelligence Review, 55(4):3213-33 2022.
[13] Christopher G Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML, volume 97, pages 12-20, 1997.
[14] Linhai Xie, Sen Wang, Stefano Rosa, Andrew Markham, and Niki Trigoni. Learning with training wheels: speeding up training with a simple controller for deep reinforcement learning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6276-6283. IEEE, 2018
[15] Xiaoqin Zhang, Huimin Ma, Xiong Luo, and Jian Yuan. Lidar: learning from imperfect demonstrations with advantage rectification. Frontiers Computer Science, 16:1-10, 2022.
[16] Philip J Ball, Laura Smith, llya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. International Conference on Machine Learning, pages 1577-1594. PMLR, 2023,
11

---

[17] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[18] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping entor reduction. Advances in neural information processing systems, 32, 2019.
[19] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[20] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference Robot Learning, pages 1702-1712. PMLR, 2022.
[21] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leaming. arXiv preprint arXiv:21 2021.
[22] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09335) 2020.
[23] Alex Beeson and Giovanni Montana. Improving td3-be: Relaxed policy constraint for offline learning and stable online fine-tuning. arXiv preprint arXiv:2211.1180) 2022.
[24] Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935, 2023.
[25] Owen Lockwood and Mei Si. review of uncertainty for deep reinforcement learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 18, pages 155-162, 2022.
[26] William Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-29 294, 1933.
[27] Richard Dearden, Nir Friedman, Stuart Russell, et al. Bayesian q-learning. Aaai/iaai, 1998: 761-768, 1998.
[28] Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From rearning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):41 1 1-444, 2022.
[29] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447. 2021.
[30] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016
[31] Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv: 100, 2017.
[32] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[33] William R Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905.06638, 2019.
[34] Yue Wu. Shuangfei Zhai, Nitish Srivastava, Joshua Susskind. Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.0m 10o::2815000800 2021.
[35] Si-An Chen, Voot Tangkaratt, Hsuan-Ties Tien Lin, and Masashi Sugiyama. Active deep g-laarning with demonstration. Machine Learning, 1999-77169917 2020.
12

---

[36] Felipe Leno Da Silva, Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Uncertainty- aware action advising for deep reinforcement learning agents. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5792-5799, 2020.
[37] Zhaodong Wang and Matthew E Taylor. Improving reinforcement learning with confidence- based demonstrations. In IJCAI, pages 3027-3033, 2017.
[38] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[39] Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 connectionist models summer school, pages 255-263. Psychology Press, 2014.
[40] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. Pmir, 2014.
[41] Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[42] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
[43] Brendan O'Donoghue, Ian Osband, Remi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. In International conference on machine learning, pages 3836-3845 2018.
[44] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00/177, 2019.
[45] Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov): 1471-1530, 2004.
[46] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
[47] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q-prop: Sample-efficient policy gradient with an off-policy critic. arXiv preprint drXiv:1611.02247, 2016.
[48] Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variance- reduced) policy gradient and natural policy gradient methods. Advances Neural Information Processing Systems, 33:7624-7636, 2020.
[49] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.004460 09444, 2018
[50] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IBBE, 2012.
[51] John Banister Lanier. Curiosity-driven multi-criteria hindsight experience replay. University of California, Irvine, 2019.
[52] Mark Towers, Ariel Kwiatkowski, Jordan Terry, John Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiy preprint arXiv:2407. 17032, 2024.
13

---

[53] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
[54] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisima Jayne. Imitation learning:
A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-55, 2017.
[55]
Chrystopher L. Nehaniv and Kerstin Dautenhahn. The correspondence problem. 2002.
[56] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[57] Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International confer- ence on machine learning, pages 3309-3318. PMLR, 2017.
[58] Samyeul Noh, Seonghyun Kim, and Ingook Jang. Efficient fine-tuning of behavior cloned policies with reinforcement learning from limited demonstrations. NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability.
[59] Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torne, and Pulkit Agrawal. From imitation to refinement-residual rl for precise assembly. arXiv preprint arXi::440.116677, 2024.
[60] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.600 2014.
[61] Barry Payne Welford. Note method for calculating corrected sums of squares and products. Technometrics, 4(3):419422, 1962.
[62] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
14

---

A Additional related work: imitation learning
Imitation learning (IL) aims to mimic expert behaviours by learning observation-to-ace mappings [54]. behaviour cloning (BC), prevalent IL method, uses supervised learning but cannot selectively incorporate demonstrations based on estimated performance like our approach. IL's primary challenge is generalising to unseen scenarios with limited demonstrations, stemming from state distribution shifts, demonstrator-leamet correspondence problems [55], and i.i.d. assumption violations. Methods like DAgger [56] and Deeply Aggre VaTeD [57] address accumulated emrors through continuous expert interaction during training an assumption our method avoids. While effective across domains, IL cannot surpass demonstrator performance and depends heavily on demonstration quality and quantity, unlike our approach which leverages even suboptimal demonstrations while exceeding their performance. Some approaches RL for post-learning refinement [58 [59, whereas seamlessly integrate demonstration guidance throughout learning. Crucially, traditional IL methods assume extensive expert demonstrations, while our work targets scenarios with limited, potentially suboptimal demonstration availability.
B Missing proofs and further theoretical results
Proof Lemma 5.1 By Assumption (A2), the demonstration samples are i.i.d., so suffices to show Var[Y] < Var[X] ] for single sample, where X 1y and Y py with g = V (T(()) all²
Let E denote the ensemble statistics for a given state-action pair (x, (s,a), which determine both 1 and p. Applying the law of total variance:
Var(X) = E|Var(X E)] Var(E[X E])
Since g is fixed given (s, a) and o, we have:
E EJ = Bi1 EJ Ei1 E]
For SPReD-P, p represents (sQ s, a) > Q(s, (s)) E), which is precisely E|1 | []] under our modeling assumptions. Thus, E[X E] gp Y. For the conditional variance term:
Var(X E Var(1g E)) 9² Var(1 E) g2pp(1 -1 0
since Il follows Bernoulli distribution with parameter y conditional E. Substituting back:
Var(X) P1 p)] Var(Y) 2 Var(Y)
The inequality is strict when E[g²p(1 [p] > 0, which occurs if and only if there exists a non- zero measure set where g o 0 and < p < simultaneously. Since g is typically non-zero (by Assumption A1), this simplifies to requiring PPOO < <<1)
L
Remark on SPReD-F. Note that the key step in Lemma 5.1 is the identity pp = [QI > Qx) E| which lets us write E[X E| ppr. In the exponential variant pg is not by construction this exact conditional expectation. Nevertheless, pr [0, 1] and it tracks the true probability pp closely shown in Property 5.3 (pp - PE - Empirically (Figure 7. SPReD-E therefore exhibits nearly the same reduction gradient variance as SPReD-P. Extending Lemma 5.1 to cover any smooth, bounded weight remains an interesting direction for future theoretical work.
Extension of Property 5.1
We restate Property 5. by separating into the two models: the first for probabilistic advantage weighting and the second for exponential advantage weighting. Property 5. a summary of Property B. and Property B.2
Property B.1 (An exhaustive version of adaptive behaviour for pp). Assume Q(sa, aa) and O(sa,To(sa) are Gaussian's 's with variances 0 and 8² respectively. Let A be the difference of their means. We define pp = 0 when A 0 and 8² + + = 0. As the variance varies, our probabilistic advantage weights satisfy:
(i) High-certainty: 2 o3 + 0 then pp + Laso (with pp 0.5 if A 0).
15

---

(ii) High Sighhuuncetaiitt: 1f +02 + xc then pp - 0.5.
Proof. Since pp =
A
Vog+0>
If A and 02+ + 0² + pp - (+00)
If A < 0 and 0 + 62 - 0: pr - 40 -00 =
If A 0 and 03 + 0² 0: pr = ((0) 1
xo: If 62 + 62 xo pp + (0) = 2
Property B.2 (An exhaustive version of adaptive behaviour for pε). Assume 3 aB where B is the IQR of the mixture model with components Q(8s, aa) and Q(8s; = Let oa and ò² be the variances and A be the difference of means of Qsa,, aa) and Q(sa, (s())) Assume (for convenience) that both distributions are continuous and symmetric. We define pe = 0 when A = 0 and 6² 0 0. As the variance varies, our exponential advantage weights satisfy:
(i) High-certainty: ff + o². + 0 then pæ - clip(e - 1,,0,) when A and pe when A < O.
(ii) High-uncertainty: If the 4th moments of Q(s,, aa) and Q(sa, To(sa)) scale like 0 and a respectively then as 6² + + + 0c, Pø + 0.
Proof. For notational convenience, let Q1 (sa ad) and Q2 Q(ss, To (sa)). and for consistency
we redefine the variances 0₁ = δd and d2 = 6. We let my be the means of Qi and Q be the mixture model Q = Q; with probability 0.5 for i = 1, We choose 3 to be the IQR of Q and B = a& for some a > which we fix. this notation A m1 m2.
(i) High-certainty limit: 01 t + 0 we claim that .3 + A. By definition, assuming continuous
and symmetric distribution for Q. the IQR, 3, satisfies P([Q - ml S 2 = 0.5 (the value of 0.5 does not matter, larger or smaller quantile bound could be equivalently considered) where mn mi+-m +m2 the mean of Q. Now for any we can apply Chebyshev's 's incquality infer
P Q ml < 8 - P (1Q) mil 2 8) + _ 2
of
0
(lQ2 m2 2 6) <
+0
0
28
28
01 + 02 + O. Hence, limo,-0 8 >> - 28 for all å > 0. In particular, lim 3 > A. On the
other hand, applying Chebyshev's inequality again implies
P P ml
A + 8
2 P (1Q1 mil
8) PP (QQ2 -mzl <)
= 1 - 1
- -mil mmm2000 2) (
P (1Q2 - -mal 2 8)
2
2
2 1 of o3
28
282
+1
σi 002 - for any 8 > 0. This implies lim A 28 for all 8 > 0. In particular,
lim,,-0 3 < A.
Now for A > we have
PE = clip(e - 1, clip(e - -,,0, + clip(e- - 1,0,
For A < 0 and 01 + o0 > 0, we have $ < 0 since > 0, so Pg = clip( - 1,0,1) 0.
(ūi High-uncertainty limit: As either σι - +oo or 02 + oo we claim / - co. Let = (Q 2_ Then EZ is the variance of the mixture model Q which is straightforward compute o²
16

---

c + 4. IM is an upper bound on the 4th moment Q1 and Q2 then a direct computation with Jensen's 's incquality gives the bound
EZ I
M 4m 6m M 3m I CM S Co
for some C, C > 0. By the Paley-Zygmund inequality,
R
(EZ)²
1
R
P P(Q ml z Rt) = PP (Z 122)
2
EZ
EZ²
C
for any R > O. If we choose R² o² 1+
3C
then we have
4
TP (1Q m
R) 0.75.
follows that B R. As + oo then 33 - Dc.
It is straightforward to now conclude that
pE clip(e # - 1,,0,) clip( 0.
Remark. For < 0 and 0 + 02 = 0, m22 - m1, so
DE clip c - 0, 1 clip (eas -1,0,1 clip (e - 1,0,1) 0.
For A = 0 and 01 + 02 = O. we define pr. = 0 for consistency.
Remark. When A = and o2 6² 0. (sa. ad) ((sa. To(sa) almost surely, which leads to
pp 0 if using a strict inequality and pp 1 if using a non-strict inequality in the definition of pp,
neither desirable. Hence, we define pp 2
Proof of Property
5.2 Fix (sd, aa) with
AQ" ("as n" (sa)) - (sd, ad) > O.
Let
A (sa, ad) ((s. To. (sa)). By (A3), there exists a sequence E, + t + 00 such that
sup Qc(sa: a) - Q=a a)l S Et.
In particular,
1Qr(sa; ua) (sd, aa)l S 6, and QQi(a, Te, (8a)) (sa, Tor (8a))) S Se.. Since the policy improves, for any 8 > 0 there is T with
(" ne. (sa)) 2 (sd, r"" (sa)) - vt 2 T.
Hence, for 2 T,
A S (Q" (sa, a²) + +.) (Q2 गt" (sa)) 6 E1) = -Q" + 22, 8. Choosing T large enough that 25, + 5 < AQ" for all t 2 T implies A, < 0. By the high-certainty limit of Property 5.I as the ensemble variance vanishes we get
PP(sa, 0a) 1A:00 = 0, PE(sa, aa) = 0.
Proof of Property 5.3
Proof. For the standard normal CDF,
1
x²
+ ((x)
+
0
2
2m
6/2m
2m
Setting A/o yields the expansion of pr ( (A/o). For pr: = (xp()//) (where clipping inactive when Al/3 & 1), we use the power series of the exponential at 0:
y²
yd
exp(y) =1 =1
+ Olr²)
Thus Pg -1
42
1
((A/3)) Comparing the linear terms of both
23
633
expansions, we see that pp 2/2
~
A
and PE ~
These terms match when 8 = ov 2T. 0
2V2T
17

---

Remark. Empirically, our ensemble Q-values are near- near-Gaussinn Gaussian, and also the difference between two independent distributions, where IQR x 1.35 a. Given the theoretical relationship 3 = = and substituting this into 3 a IQR, we can determine a principled starting point for and a. While a requires tuning for specific task characteristics, the theoretical relationship provides meaningful baseline that ensures proper uncertainty scaling across environments.
C Algorithms and implementation details
Algorithm overview SPReD incorporates uncertainty quantification to enable adaptive demonstra- tion utilisation through principled ensemble-based approach. We maintain ensemble of critic networks (0,), each with a corresponding target network, alongside a standard actor network (6) with its target network. The algorithm maintains two separate buffers: standard experience replay buffer B and demonstration buffer Bp containing pre-collected demonstration transitions.
Algorithm 1 Reinforcement Learning with Smooth Policy regularisation from Demonstrations
Initialise critic networks 0, actor network and their target networks 0 02, o' Φ. where
= -1,2,.., m and m is the ensemble size
2: Initialise replay buffer B ( and demonstration buffer BD with transitions in demonstrations
(sa, ag. ra, $/2 9a)
3: for episode e 1 to MM do
4: for -= to T do
5:
Execute action a ~ (8))) ()) + with exploration noise e -c, where e is the maximum action. Observe reward r and new state s'
6:
Store the transition (s, a, r, s')g) in
7:
Update state s end for
8:
9:
for t= 1 to T do
10:
Store the transition (s, a, r, 9a) ) in where Ia is the actual achieved goal in this episode
11:
end for
12:
for
iteration = 1 to k do Sample mini-batch of size NR from B and mini-batch of size No from Bp Sample two critics uniformly from the ensembles Update all critic networks by minimizing
14:
[(0:) E(,,,))B.BB) (r +
min Q0 (s',a) Qo, (s)
16:
where à = Tal TTr t, -l)(( if ( mod
then
18:
Update the actor network by minimizing Equation Update target networks:
20:
8. - T0; + (1 T)
21:
- T + (1 - g
22:
end if
23:
end for
24:
end for
Our Algorithm | operates in two main phases. During environment interaction (lines 4-8), the agent follows its current policy with added exploration noise bounded to the action space collect experiences. Following the HER approach [41]. we augment collected transitions by storing them again with actually achieved goals (lines 9-11), enabling learning from unsuccessful episodes in sparse-reward environments. The learning process (lines 12-21) integrates several key components:
Coordinated sampling: Each update draws transitions from both experience and demonstra- tion buffers with fixed proportions, ensuring consistent demonstration influence throughout training.
Ensemble-based target computation: We randomly select two critics from the ensemble to compute target values, following the REDQ approach 53 to mitigate overestimation bias while maintaining computational efficiency.
18

---

Uncertainty-awai policy updates: The actor loss combines standard deterministic policy gradient with our mncertainty-weighie behaviour cloning loss that smoothly regularises the policy.
Weighting mechanisms During the actor update, we compute the weight for each demonstration using either SPReD-P SPReD-E. For SPReD-P, we compute the probability described in Section 4 which naturally produces values in the [0, 1] range through the CDE For SPReD-E, we practically take
==a [OR aa) IQR (
and apply the truncation the basic exponential form:
PE clip(e 1,0,1)
This truncation: (1) ensures PE = 0 for negative advantages, preventing imitation of demonstrably
inferior actions; (2) creates a smooth exponential ramp for modest positive advantages; and (3) caps the weight pe = 1 when A/3 2 In 2, providing full imitation only for clearly superior demonstrations.
Remark. Both our choice of 3 and 3 = Lovv22 2T, which connects two variants of our method, measure the uncertainty of the advantage. The choice of 3 is not sensitive, and provides proportional bounds for 3 With near- Gaussian Q-value distributions, our $ 1.35(6+ba) By the Cauchy-Schwer inequality,
(( + + (8 2 03) s 2n( +a)2.
Then we have
2/8
5 <
2v/2 3 since 3 and 3 non-negative. 1.35
1.35
Computational considerations Despite its theoretical sophistication, SPReD introduces minimal computational overhead. The entire probability or advantage calculation and weighting process requires only a few simple operations beyond standard TD3, with the ensemble critics serving dual purposes of target value computation and uncertainty estimation. The actor update occurs less frequently (every d steps) to allow critic networks stabilise between policy updates. Our implementation maintains the same overall complexity as standard RI algorithms while gaining the benefits of uncertainty-awarr demonstration utilisation. The practical computational cost of our method and all baselines we compare are presented in Figure 4 SPRcD requires Rs 2.6 hours per 4M environment steps, nearly identical TD3 (2 critics), while RLPD (also 10 critics) requires 444
4.8 hours. SPReD processes &e 427 environment steps/second, compared to TD3's steps/second only 3.8% decrease despite using 5x more crities. Given that SPReD achieves up to 14x better success rates than baselines on complex tasks, this < 4% throughput cost is well justified. Our SPReD method requires almost the same computational resources the standard RL algorithm, while a single running of RLPD takes nearly double time.
Runiig (hours)
TB2
EneTD3
Q-liter
Bneq filter
RLPD
BNAC.r
SRReD P
SPReD-E
Method
Figure 4: Computational cost for the individual experimental runs for FetchPickAndPlace with 4e6 steps.
19

---

Network architecture and hyperparameters We implement our approach using hyperparameters consistent with prior work [6]: mini-batch sizes NR 1024 and No 128 for experience and demonstration buffers respectively, discount factor 7 0.98, and loss weights X1 = 10-3 and A2 = 128 Both actor and critic networks employ identical architectures consisting of two hidden layers with 256 neurons each and ReLU activations. The actor's output layer uses a tanh activation to bound actions within the environment's 's range. We use the Adam optimizer (60] with learning rate 10t for all networks.
For SPReD-E, the scaling constant a was set to 10 based on preliminary experiments. This value provides sufficient caution with uncertain estimates while still allowing clear advantages to receive significant imitation weights. The ablation test is presented Appendix [G]
State and goal observations are normalised before being processed by the networks:
Normalised value = clip clip(original value, 200, 200) μ 5,.5
+ 10-6
where μ and a represent the running mean and standard deviation, updated at each step using Welford's 's online algorithm [6]] For methods using demonstrations, these statisties are initialised using the demonstration data. The small constant (10 ") prevents division by zero.
For exploration, we employ clipped Gaussian noise with 0.1. Following standard TD3 imple- mentation (38), we add smoothing noise clip(N((, 0.2), -0.5, 0.5) to actions during critic updates to enhance Q-function smoothness across similar actions.
We maintain replay buffer capacity of 10 transitions and begin training after collecting 10.V initial transitions. Critic networks are updated twice per iteration while the actor network is updated once, with target networks updated via Polyak averaging using т 10-3 Our ensemble consists of 10 independent critic networks with ablation test presented in Appendix G Policy performance evaluated regularly over 25 test episodes without exploration noise. All the experiments were performed with a single GeForce GTX 3090 GPU and an Intel Core 19- 1900K CPU at 3.50GHz.
D Further environment details
We evaluate our approach on eight robotics tasks implemented in the OpenAl Gym framework 449 simulated using the MuloCo physies engine [50] These environments feature sparse rewards and multi-goal structures, providing an ideal testbed for demonstration based learning approaches.
Fetch robotic arm tasks The Fetch environment employs a 7-DoF robotic arm with a parallel gripper for manipulation tasks of increasing complexity:
FetchPush: Moving objects to target positions on a tabletop
FetchSlide: Striking objects toward targets beyond the arm's reach
FetchPickAndPlace: Lifting and positioning objects 3D space
FetchStack2 and FetchStack3: Precisely arranging multiple blocks specified configura- tions
In these environments, the action space is 4 4-dimensional, with the first three dimensions controlling the gripper's 's movement and the fourth dimension controlling gripper opening/closing. Observations are 25-dimensional, containing position and velocity information for both the gripper and manipulated objects, and goals are specified as 3-dimensional target positions in Cartesian coordinates for standard tasks. Additional dimensions included for stacking tasks to accommodate multiple objects.
Shadow dexterous hand tasks control challenges:
The Shadow Hand environment presents significantly more complex
ManipulateBlock: Manipulating cube a target orientation
ManipulateEgg: Orienting egg-shaped object
ManipulatePen: Precisely controlling pen-shaped object
20

---

Goals in these environments are specified as 7-dimensional vectors representing target positions (3D Cartesian coordinates) and orientations (quaternions). The action space is 20-dimensional. corresponding to the absolute angular positions of the hand's actuated joints. Observations are 61-dimensional, containing comprehensive kinematic information about both the hand and the manipulated object.
All environments employ sparse binary rewards, with agents receiving 0 when successfully achieving goals (within a specified tolerance) and -I otherwise. For the stacking tasks, an additional reward of is provided when the gripper moves away from the blocks after successful completion, encouraging proper task termination.
E Demonstration data quality
We collect demonstrations with varying levels of quality evaluate the robustness of approach across different conditions. For each environment, we use 100 demonstration episodes, with the exception of FetchStack3 where we use 1000 episodes due to its greater complexity.
Table Categorisation of environments by demonstration quality used in main results reported by Table and Figure
Quality Level Success Rate
Environments
Expert
0,86-1.00
FetchStack2 (1.00), FetchStack3 (1.00), ManipulateBlock (0.86)
Moderate
0.49-0.53
FetchSlide (0.53). FetchPickAndPPace (0.49)
Low
0.20-0.39
ManipulateFgg (0.39), ManipulatelPen (0.37), FetchPush (0.20)
For the challenging stacking tasks (FetchStack2 and FetchStack3), utilise expert demonstra- tions from policies developed by Lanier et al. (51), achieving perfect success rates (1.0). The ManipulateBlock environment also features high-quality demonstrations with success rate of 0.86.
For the remaining environments, we generate demonstrations of varying quality using policies trained with EnsTD3+HER and introducing controlled levels of noise. FetchSlide (0.53) and FetchPickAnd- Place (0.49) use moderate- quality demonstrations, while ManipulateEgg (0.39), ManipulatePenn (0.37), and FetchPush (0.20) employ lower-quality demonstrations. Table categorises these environments by demonstration quality level.
For experiments explicitly analyzing sensitivity to demonstration quality (Figure and Figure 10p we generate three distinct quality levels for each environment:
1. Expert: Demonstrations collected from well-trained policies with minimal added noise.
2. Suboptimal: Generated by adding moderate Gaussian noise to expert actions.
3. Severely suboptimal: Created with substantial noise that significantly degrades demonstra- tion quality.
This systematic variation enables us to precisely characterise how each algorithm's 's performance scales with demonstration quality, isolating this factor from other variables. For the extreme test case, we also evaluate performance when provided with 99% random trajectories mixed with just 1% expert demonstrations.
The deliberate inclusion of diverse demonstration qualities reflects real-world scenarios where perfect demonstrations may be unavailable or prohibitively expensive to collect. An algorithm's 's ability to extract useful information even from imperfect demonstrations particularly important for practical applications.
F Empirical evidence to support methods
Uncertainty measures As we mention in Section 2 there are different methods of the uncertainty measure. We investigate dropout-based uncertainty as an alternative to our ensemble approach. Contrary the intuition that dropout might reduce overhead, our experiments show it actually increases computational cost while degrading performance:
21

---

Table Running time and success rate for SPReD-P with different uncertainty measures in Fetch- PickAndPlace. The ensemble size for ensemble method is 10. For dropout method, the dropout rate 0. 1, and there are 500 forward passes per critic (2 crities in TD3).
Method
Time (4M steps)
Success rate (1M steps)
Ensemble 2.6h
0.832 = 0.111
Dropout
20.3h
0.600 = 0.057
The dropout approach is 8x slower due to requiring 1000 stochastic forward passes (500 x 2) to estimate uncertainty, while our ensemble uses a single vectorised pass through crities parallel on GPU. Although both methods eventually learn an expert policy within 4M steps, dropout shows worse sample efficiency (28% drop), likely due to: (1) noisier uncertainty estimates, (2) overconfident predictions on unseen data [62]. and (3) computational bottlenecks from repeated passes that hinder efficient batch processing.
These results are consistent with prior findings 622 that ensembles yield better uncertainty estimates than dropout, and modern GPUs enable highly efficient ensemble parallelisation.
Bootstrapping presents other computational challenges: (1) separate data samples per model prevent batch-sharing. (2) models process different minibatches, blocking parallelisation, and (3) storing multiple bootstrap samples increases memory demands. In contrast, our ensemble processes the same batch across all crities via single vectorised operation, achieving near-linear speedup. Moreover, Bootstrapped DQN [30] supports this strategy, suggesting that diversity from random initialisations of deep NN climinates the need of explicit data bootstrapping.
These results confirm that vectorised ensembles offer the best balance between uncertainty estimation quality and computational efficiency.
Locomotion tasks We also evaluate our methods relatively easy OpenAl Gym locomotion tasks
52] with dense rewards, where goals and HER are excluded from the algorithm. The tasks are to move the following robots in the forward direction:
Hopper: a wwo-dimensional one-legged figure
HalfCheetah: a two-dimensional robot with 9 body parts and 8 joints
Walker2d: a two-dimensional bipedal robot
Ant: three-dimensional quadruped robot
Humanoid: a three-dimensional bipedal robot which simulates a human
According to the initial sample efficiency at 200K steps shown in Table 4 SPReD method surpasses all variants of AWAC for HalfChectah and Ant, and is particularly outstanding on HalfCheetah (SPReD-P gains 12% improvement from EnsQ-filter and 72% improvement from Q-filter). AWAC is competitive for other tasks with near-expert demonstrations, and RLPD has remarkable sample efficiency initially. However, the learning scores of AWAC and RLPD asymptotically lower than SPReD for all locomotion tasks we tested, which is visualised by learning curves in Figure 3 Even in setting with dense rewards, SPReD method is the robustest with relatively high sample efficiency, consistent improvement and the best converging performance, confirming that our uncertainty-aa approach effectively transfers across different continuous control domains.
Table 4: Average score (with standard deviation) over 5 seeds after 200K interactions for locomotion tasks. The highlighted results lie between the mean of the best performer and one standard deviation below it (i.e., if the best result is + σ, all values 2 a are bold). The scores of demonstrations
range from 2500-7000.
Favinomment
Methods
TD3
EasT133
Q-titer
Enst)-Biter
RLPD
AWAC
AWAC-P
AWAC-
SPRe1>-P
SPReD-E
Hopper
1468 à 8/9
2164 : 966
2818 438
2030 5.16
2744 s 580
2855 4: 707
3228 50
2461 1012
3246 : 18
2740 3.4
HeltCh
3728 2. 419
3175 1756
4671 760
7188 2 934
0721 3542
4835 a 854
51501 801
4209 a 1069
N060 1 171
7336 601
WalkerBd
1861 1159
2334 - 772
1485 580
3576 528
4519 e 154
3987 + 108
3343 + 1696
4005 - 433
3351 842
2403 + 1154
Ant
2028
3895 =
2478 960
5862 292
6968 42
3470 1325
1776
3465 997
5636
$779 4:18
Hummaoid
853 242
4701
4592
22

---

TD3
EnsTD3
Q-filter
EnsQ filter
RUPD
AWACC
ANAC-p
NWACY
SPReD-P
SPReD-E
Hopper
HairCheetah
Walker2d
5000
3000
9000
3750
2000
1000
3000
1250
0.00
0.25
0.50
0.73
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Emv steps (10%)
Env steps 110%)
Env steps (10")
Ant
Humanold
6000
6000
4000
4500
2000
3000
1500
-200
0.75
0.00
0.75
Eny shops (3.0%)
Env stops (3.0%)
Figure 5: Performance comparison across five locomotion tasks. Horizontal dashed lines indicate the scores of the demonstrations used for training. Our SPReD methods (red and brown) consistently outperform baselines across different tasks.
AWAC's poor performance on both locomotion and manipulation domains stems from a fundamental mismatch with problem setting: AWAC assumes large offline datasets IM transitions) for effective pretraining, but we have only 5K demonstration transitions. Without sufficient pretraining data, and since demonstrations are quickly diluted in the replay buffer, their impact fades early, effectively reducing AWAC to standard RL. Moreover, AWAC's advantage weighting assumes the offline data covers substantial portion of the state space, which doesn't 't hold with sparse demonstrations.
Gaussian assumption Our SPReD-P method relies on the assumption that Q-value estimates across the ensemble follow a Gaussian distribution. To validate this assumption, compare the performance of our Gaussian approach against two nonparametric alternatives that make no distribu- tional assumptions: (1) Sooppaaa_paiwisse which randomly pairs critic networks and makes pairwise comparisons, and (2) Nonpara_cross which performs all possible cross-comparisons between critics (10 10 pairs).
As demonstrated in Figure 6 the Gaussian approximation consistently outperforms both nonpara- metric methods in the FetchPickAnd environment. These results validate our modeling choice, suggesting that the Gaussian approximation effectively captures the underlying uncertainty while
1.0
0.8
Success rate
0.6
0.4
Gaussian
0.2
Nonpara_pann ppairwise
Nonpara_crmm cross
0.0
Env steps (10%)
Figure 6: Comparison between Gaussian and nonparametric methods in FetchPickAndPlace. Gaus- sian approximates two distributions of Q-values as Gaussian distributions using the sample mean and variance. Nonpara pairwise pairwise compares 10 pairs of Q-values, while Nonpara _ross crosswise compares 10 x 10 pairs of Q-values. The dashed line shows the success rate of demonstrations.
Nonpara_pairwise
23

---

providing computational advantages over nonparametric approaches. The superior performance likely stems from the Gaussian model''s ability leverage the entire ensemble's information in statistically efficient manner, whereas the nonparametric approaches may suffer from higher variance in their comparisons.
Variance reduction Through weighted BC, the gradient variance of policy updates is significantly reduced our SPReD method as proved by Lemma [.1] Figure confirms our theoretical prediction and establishes the benefit of smooth policy regularisation from demonstrations.
1e-6
EnsQ-filter
SPReD-P
Gradient variance
SPReD-E
2
www
0.00
0.25
0.50
0.75
1.00
Env steps (10%)
Figure 7: Empirical gradient variance of actor updates in the FetchPush environment, demonstrating that two variants of our SPReD method significantly reduce policy update variance compared to binary imitation decisions.
Adaptive mechanisms Figure provides insight into the adaptive mechanism behind the robustness of our SPReD method with various demonstration qualities. Both probabilistic and exponential variants progressively reduce the influence of these extremely suboptimal demonstrations (with only 0.06 success rate), effectively eliminating their impact policy updates. Examining SPReD-P's weighting mechanism in detail (Figure Sa). we observe three distinct phases which coincide with our theoretical expectation in Property B. (1) initial uncertainty phase where most weights cluster around 0.5, reflecting limited confidence in Q-value comparisons; (2) transition phase around one million interactions where the policy improves and weights begin polarizing; and (3) final
0.12
Probabilistin weight
Fxponential weight
0.6
0.4
0.2
0.03
0.0
0.00
2
3
2
3
LnV steps (10%)
Lnv steps (10%)
(a) SPReD-P
(b) SPReD-E
Figure 8: Evolution of behaviour cloning weights in the FetchPickAndPlace environment with severely suboptimal demonstrations (99% random trajectories). SPReD-P (left) shows three distinct phases: initial uncertainty (weights s 0.5). transition (polarizing weights), and final expert policy (most weights - 0). SPReD-E (right) displays similar trend with generally lower weight magnitudes for generally inferior demonstrations. Both methods automatically reduce the influence of poor demonstrations learning progresses, demonstrating the adaptive nature of the uncertainty-awarr weighting mechanisms.

---

phase where the vast majority of demonstration weights approach zero, with only genuinely superior demonstration actions retaining influence. SPReD-E (Figure shows similar qualitative behaviour, though with generally lower weight magnitudes due to its exponential scaling and conforms the theoretical results in Property B.2 with pessimistic normalisation a.
Noisy rewards We also evaluate the robustmess of our methods noisy rewards. We consider two types of noisy rewards: with probability of 0.1, flipping rewards of -1 (failure) to be O (success) or adding Gaussian noise to the rewards of -1. Note that reward computation function in HER still accurate and not affected.
TD3
EnsTD3
Q-filter
EnsQ-filter
RLPD
AWAC
AWAC-p
AWAC-r
SPReD-P
SPReD-E
Flipping noisy Rewards
Gaussian noisy Rewards
1.0
1.0
0.8
0.3
Success rate
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.6
0.3
1.0
Env steps (10%)
Env steps (10%)
Figure 9: The robustness to noisy rewards in FetchPush. The success rate of demonstrations is shown as dashed lines.
With noisy rewards, the efficient utilisation demonstrations becomes more crucial. With either kind of noisy rewards, the standard Q-filter fails to learn the policy due to the disturbed Q-values. However, our SPReD methods remain the best among all baselines, benefiting from smooth and uncertainty- aware regularisations.
G Ablation study:
impact of ensemble size and normalisation constant of SPReD-E on performance
Additional experiments of demonstration quality trends in FetchPickAndPlann confirming the robustness of our methods to demonstration quality.
The experiments in FetchPush exhibit similar
TD3
EnsTD3
Q-filter
EnsQ-fiiter
RLPD
AWAC
AWAC-p
AWAC-r
SPReD-P
SPReD-E
1.0
Success
0.6
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Env steps (109)
Env steps (10%)
Env steps (109)
Figure 10: Effect of demonstration quality in FetchPush. The demonstrations are expert, suboptimal and severely suboptimal from left to right with success rates shown as dashed lines.
Ensemble size We systematically evaluate the effect of ensemble size on learning performance by varying it from 2 to 30 critics across multiple environments. Our analysis reveals deseddeendd sensitivity to ensemble size. As shown in Figure II
performance in the FetchPickAndPlace envi-
25

---

EnsSize2
EnsSize3
EnsSize5
EnsSize10
EnsSize20
EnsSize30
FetchPickAndPlace
ManigulateEggRRotate
1.0
0.48
Succoss rate
0.36
0.6
0.4
0.24
0.12
0.0
0.00
Env steps (109)
Env steps (109)
(a) SPReD-P (Probabilistic method)
FetchPickAndPlace
ManigulateEggRotate
0.75
0.60
Success ratc
0.6
0.45
0.4
0.30
0.15
0.00
Env steps (10%)
Env steps (10%)
(b) SPReD-E (Exponential method)
Figure 11: Effect of ensemble size on learning performance. Left: FetchPickAndPlaae enviromment shows minimal sensitivity ensemble size for both methods. Right: ManipulateFgg environment demonstrates improved performance with larger ensembles, with diminishing returns beyond size 10. Ensemble sizes tested: 3, 5, 10, 20, and 30.
ronment (left) remains relatively consistent across different ensemble sizes for both SPReD variants, suggesting that even small ensembles (> 2) capture sufficient uncertainty information for this task.
In contrast, the more complex ManipulateEgg manipulation task (right) shows clearer performance improvement with increasing ensemble size. This suggests that challenging control tasks with higher-dimensional action spaces benefit from the improved uncertainty estimates provided by larger ensembles. However, the performance gains diminish noticeably beyond ensemble size of 10, with minimal additional improvement at sizes 20 and 30 despite the substantial increase in computational cost.
Based on this analysis and computational efficiency considerations, we select an ensemble size of 10 for main experiments. This choice provides favorable trade-off between performance and computational requirements, and maintains consistency with the RLPD baseline which also uses 10 critics. Further scaling the ensemble provides diminishing returns that do not justify the additional computational overhead for most practical applications.
Isolated contribution Since SPReD introduces both continuous weights and ensemble-based uncertainty estimation, it is important to understand their individual and combined effects. We conduct systematic ablation study to isolate these contributions. By varying ensemble size (2 vs. 10) and regularisation type (binary vs. continuous), we can assess each component's 's impact (with success rates at MM steps).
From results in Table we have the following key findings. First, continuous regularisation the primary driver. Even with minimal ensemble (size 2), SPReD-P improves over Q-fitter by 28%
26

---

Table The success rates of different methods in FetchPickkAndPPace at 1M steps to assess isolated contributions of ensemble and continuous weights.
Q-filter
EnsQ-iilter
SPReD-P (size 2) SPReD-E (size
SPReD-P (size
SPReD-E (size 10)
0.608 0.047
0.688 0.069
0.776 # 0.070
0.152 = 0.304
0.832 = 0.111
0.888 = 0.064
(0.776 vs 0.608). This demonstrates that smooth, envertainty-prooortinn weights fundamentally better than binary decisions, regardless of ensemble size. Then ensembles add complementary value. Expanding from 2 to 10 critics further boosts performance for both SPReD variants. However, the gains are method-specific- Saadspeecfic-SPRRDD4.4 shows modest improvement (+7%, 0.776 vs 0.832) while SPReD-E shows dramatic improvement (+484%, 0.152 0.888 This differential benefit reveals an important insight that the methods have different uncertainty requirements. SPReD-E's exponential weighting critically depends on accurate uncertainty estimates through With only 2 critics, the IQR calculation is noisy, leading to suboptimal scaling. In contrast, SPReD-P's probabilistic approach inherently more robust limited ensemble sizes. The ablation confirms that while both components contribute independently, they work synergistically. Continuous regularisation provides the foundation for better learning, while larger ensembles enable more precise uncertainty quantification particularly crucial for SPReD-E's exponential scaling mechanism. This validates our design decision combine both innovations rather than pursuing either in isolation.
Normalisation constant The theoretical scaling 3 a V Property 5.3 builds connection with our probabilistic advantage weighting method SPReD-P, but there works best. While both a and IQR measure Q-value distribution spread, the IQR-based approach provides better empirical performance, particularly in more challenging tasks (17% improvement in FetehPickAndPlace with a 10 after 1M steps) as shown in Figure 12 The advantage is even more pronounced with lower-quality demonstrations (44% higher success rate after 4M steps). This advantage likely stems from IQR's robustness to outliers in Q-value estimates during early training when ensemble variance is high. According task and demonstration quality, but the performance of our SPReD-E not very sensitive to the choice of There no general trend indicating that smaller larger value of would be better. This is a task-specific hyperparameter which can be tuned to achieve better performance for specific task. In our work, take a 10 for the consistent comparisons, which performs well overall.
guarantee that it
experimental results, the effect of depends
SPReD.P
alpha=2
alpha=5
alpha=10
alpha=20
thearetical
Suboptimal demos
Severely suboptimal demos
1.0
1.0
0.8
0.8
success rate
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0
1
2
Env steps (10%)
Env steps (105)
Figure 12: Effect of normalisation constant of SPReD-E in the FetchPickAndPlac. environment with suboptimal and severely suboptimal demonstrations (success rates shown as dashed lines). The performances SPReD-P SPReD-E with theoretical 3 2 are given as the baseline of comparison.
27

---

H
Broader impacts
Our research on uncertainty-aware reinforcement learning from demonstrations offers several societal benefits: accelerating robotic automation across industries, enabling safer operation in hazardous environments, and democratizing access to robotic solutions through reduced demonstration require- ments. However, these advances may also contribute to workforce displacement in sectors reliant on manual labor, potentially devalue certain specialised demonstration skills, and introduce challenges for accountability in systems that require minimal human supervision. We acknowledge that com- plementary workforce development programs and economic policies will be important alongside technological advances in automation to address potential negative impacts employment.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claim of reduced gradient variance, higher sample efficiency and robust- ness with quality and quantity of demonstrations obtained by smooth uncertainty-awer policy regularisation made in the abstract and introduction is verified both theoretically and experimentally in Section Section and Appendix B
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We present comprehensive experiments on tasks with different complexity, demonstration quality and demonstration size in Section In Section we explicitly discuss limitations including: demonstration influence potentially slowing later-stage learning in highly dexterous tasks, hyperparameter sensitivity challenges, and remaining sample complexity issues in extremely high-dimensional tasks that require further refinement.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
29

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The complete presentations and proofs with corresponding assumptions of our theoretical results in Section | provided in Appendix B We also provide the empirical
evidence support our theory Appendix F
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
Our algorithm is well-explained with pseudocode and all implementation The demonstrations used in this paper are also explained in
details provided in Appendix C Appendix E for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
30

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our implementation code with detailed instructions for environment setup, model training, and experiment reproduction on GitHub. The demonstration datasets used across all experiments are also included to enable complete reproduction of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/ //nips.cc/pplle wttpcccppppppppeeeeeeeeeeeeeeeee
httpl////giieee/eeSSSuiiisisssss Cooeeeoomdsssiiiieccea sonooooocsss for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We briefly introduce the experimental settings in Section 6 The full implemen- tation details are included in Appendix C with detailed explanations about the environments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
31

---

Justification:
We report the standard deviations across seeds as error bars for all results shown the shaded area in all applicable figures and included Table with explanation in captions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eτror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the relevant information in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our experiments are based on the simulated environments with open source with no human participants.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
32

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix H we discuss both positive impacts of our technology, ac- knowledging the need for complementary workforce development alongside technological advances.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our experiments simulations robotic manipulation tasks without concern privacy or misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
33

---

Justification: We properly cite all external assets used in our research, including OpenAI Gym environments, MuloCo physics engine for simulation, and expert stacking task policies from Lanier et al. All these assets are used in accordance with their open-source licenses and academic citation requirements.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, snnerswwthcode com/co//caaoooocoo con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code for our method is provided with document for instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmdel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
34

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy //np:s:////urraie cc. Cc//oonferencee/22255///// for what should or should not be described.
35


Reviews:
[{"review_id": "iq4tvurDRr", "reviewer": "Reviewer_JFdB", "summary": "The paper introduces Smooth Policy Regularisation from Demonstrations (SPReD), an ensemble-based extension of the Q-filter that turns the binary “imitate / ignore” rule into a continuous weight proportional to the critic ensemble’s epistemic uncertainty. Two instantiations are proposed:\n\n- SPReD-P: probabilistic weighting via the likelihood that a demo action’s Q-value exceeds the policy action’s.\n\n- SPReD-E: exponential weighting that scales with an advantage-to-uncertainty ratio.\n\nThe authors prove lower gradient variance than the Q-filter and present large empirical gains on eight MuJoCo manipulation tasks with sparse rewards.", "questions": "Besides the points of Weaknesses:\n\n- How does SPReD-E perform if β is set using the theoretical scaling (β = σ√2π) instead of a heuristic IQR-based α?\n- Is it possible to use dropout-based uncertainty or bootstrapping to reduce compute overhead, and how does performance compare?\n- Why was AWAC (with HER) not included as a baseline? Please report AWAC on at least FetchStack2.", "limitations": "The paper included limitations in the conclusion section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The need for smoother, uncertainty-aware demonstration regularization is well-motivated. Lower variance in policy updates is shown both theoretically and empirically.\n- Only requires minor changes to TD3+HER setups; ensemble critics are used both for targets and uncertainty estimates.\n- Results are robust across eight tasks, with substantial improvements, especially on harder manipulation and stacking environments.\n\n**Weaknesses**\n- The paper does not cite key related methods: IQL [1], UWAC [2], which combine advantage-weighted or uncertainty-weighted imitation in RL. \n- The cost of ensembles (10 critics) is not directly compared against single-critic baselines. Throughput is mentioned, but energy and wall-time under fixed hardware are not reported.\n- The approach is only tested in MuJoCo. No robustness analysis under sim-to-real gap, nor any explicit discussion of failure cases under noisy or misaligned reward signals.\n\n**References**\n[1] \"Offline Reinforcement Learning with Implicit Q-Learning\". ICLR 2022.\n[2] \"Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning\". ICML 2021.", "was_revised": true, "final_justification": "The rebuttal results have addressed most of my concerns. I recommend authors to include the results in the revision. And I am happy to increase my score.", "rebuttal": "Thank you for your constructive feedback. We address each of your concerns below:\n\n**Missing citations:** Thank you for highlighting these methods. We will include IQL and UWAC in our related work section. While both share ideas with SPReD, such as advantage weighting (IQL) and uncertainty-based weighting via dropout (UWAC), they address different problems:\n\n- **IQL** focuses on offline and offline-to-online RL, using expectile regression to learn a value function and performing advantage-weighted policy extraction to learn a policy without explicit bootstrapping from the policy.\n- **UWAC** is an offline RL method that weights actor updates using dropout-based uncertainty to manage out-of-distribution actions.\n\nIn contrast, **SPReD** tackles online RL with few demonstrations (~5000 transitions), where the agent must continuously balance exploration and selective demonstration guidance. Its core contribution is determining when demonstrations remain useful during ongoing learning.\n\nWe will include them in our revised related work section with a clear discussion of the differing problem settings.\n\n**Computational cost analysis:** We have included computational cost analysis in Appendix C (Figure 4, page 18). Our experiments on a single GeForce GTX 3090 GPU show:\n\n- **Wall-time**: SPReD requires ~2.6 hours per 4M environment steps, nearly identical to TD3 (2 critics), while RLPD (also 10 critics) requires ~4.8 hours.\n- **Throughput**: SPReD processes ~427 environment steps/second, compared to TD3's ~444 steps/second—only a 3.8% decrease despite using $5×$ more critics.\n- **Energy**: While we did not measure energy consumption directly (as is standard in RL papers), the similar wall-time to TD3 suggests comparable energy usage.\n\nThe key to our efficiency is *vectorised* critic implementation: instead of processing 10 critics sequentially, we stack their computations into batched tensor operations that execute in parallel on GPU. This is why SPReD achieves nearly the same throughput as TD3 despite using $5×$ more critics. Given that SPReD achieves up to $14×$ better success rates than baselines on complex tasks (Table 1), this $<4$% throughput cost is well justified. We will add these throughput metrics to the main paper for transparency.\n\n**Regarding sim-to-real gaps and reward misspecification:** While important, they remain orthogonal to our contribution. In most online RL research, the rewards are assumed to be accessible and reliable, and we continue with this assumption since misleading reward is not the primary problem we want to address in this work. If you deem the consideration of it necessary, we are willing to do the experiments with noisy rewards to check the robustness and add the results to the final manuscript. Our uncertainty-aware approach should actually provide more robustness than binary methods under such conditions. We will update our paper to clarify sim-to-real gaps as remaining limitations in the appropriate section and regard this as future work.\n\n### Questions\n\n**Q1**: We evaluated both approaches on multiple tasks and gave the success rate at 1M steps in the table. The results show that our IQR-based $β$ (with $α=10$) consistently outperforms the theoretical scaling:\n\n| Task | SPReD-E ($α=10$) | SPReD-E ($β=σ\\sqrt{2π}$) |\n| --- | --- | --- |\n| FetchPush | 0.984 ± 0.032 | 0.976 ± 0.020 |\n| FetchPickAndPlace (Suboptimal demos) | 0.840 ± 0.119 | 0.720 ± 0.104 |\n| FetchPickAndPlace (Severely suboptimal demos) | 0.304 ± 0.194 | 0.088 ± 0.059 |\n\nThe theoretical scaling $β=σ\\sqrt{2π}$ builds a connection with our probabilistic advantage weighting method SPReD-P, but there is no guarantee that it works the best. While both $σ$ and IQR measure Q-value distribution spread, the IQR-based approach provides better empirical performance, particularly in more challenging tasks (17% improvement in FetchPickAndPlace). The advantage is even more pronounced with lower-quality demonstrations (44% higher success rate after 4M steps). This advantage likely stems from IQR's robustness to outliers in Q-value estimates during early training when ensemble variance is high. Additionally, the hyperparameter $α$ allows task-specific tuning as shown for FetchPickAndPlace in Appendix G (Figure 9, page 24), whereas the theoretical scaling is fixed. We will include complete learning curves comparing both approaches in the revised manuscript.\n\n**Q2:** Yes, we investigated dropout-based uncertainty as an alternative to our ensemble approach. Contrary to the intuition that dropout might reduce overhead, our experiments show it actually increases computational cost while degrading performance:\n\n| Method | Time (4M steps) | Success Rate (1M steps) |\n| --- | --- | --- |\n| SPReD-P (Ensemble size10) | 2.6h | 0.744 ± 0.103 |\n| SPReD-P (Dropout) | 20.3h | 0.600 ± 0.044 |\n\n*Dropout configuration: rate=0.1, 500 forward passes per critic (2 critics).*\n\nThe **dropout approach is 8× slower** due to requiring 1000 stochastic forward passes (500 × 2) to estimate uncertainty, while our ensemble uses a single vectorised pass through 10 critics in parallel on GPU. Although both methods eventually learn an expert policy within 4M steps, dropout shows worse sample efficiency (19% drop), likely due to: (1) noisier uncertainty estimates, (2) overconfident predictions on unseen data [1], and (3) computational bottlenecks from repeated passes that hinder efficient batch processing.\n\nThese results are consistent with prior findings [1] that ensembles yield better uncertainty estimates than dropout, and modern GPUs enable highly efficient ensemble parallelisation.\n\nBootstrapping presents other computational challenges: (1) separate data samples per model prevent batch-sharing, (2) models process different minibatches, blocking parallelisation, and (3) storing multiple bootstrap samples increases memory demands. In contrast, our ensemble processes the same batch across all critics via a single vectorised operation, achieving near-linear speedup. Moreover, Bootstrapped DQN [2] supports this strategy, suggesting that diversity from random initialisations of deep NN eliminates the need of explicit data bootstrapping.\n\nThese results confirm that vectorised ensembles offer the best balance between uncertainty estimation quality and computational efficiency.\n\n**Q3:** We have added AWAC+HER as a baseline (with and without pretraining). We first compare our method and baselines with AWAC on locomotion tasks where AWAC has good performance [3]. The demonstrations contain 5 suboptimal trajectories ($≤ 5000$ transitions) for each task. The table shows scores at 0.2M steps for locomotion tasks, success rates at 10M steps for FetchStack2 and 1M steps for other fetch tasks.\n\n| Task | AWAC | AWAC  (pretraining) | AWAC (resample demos) | SPReD-P | SPReD-E |\n| --- | --- | --- | --- | --- | --- |\n| Hopper | 2855 ± 707 | **3176 ± 94** | 2461 ± 1041 | **3246 ± 18** | 2740 ± 344 |\n| HalfCheetah  | 4835 ± 854 | 4310 ± 963 | 5115 ± 823 | **8060 ± 171** | 7336 ± 601 |\n| Walker2d  | **3987 ± 108** | **3980 ± 353** | **4005 ± 433** | 3351 ± 842 | 2403 ± 1154 |\n| FetchPush | 0.056 ± 0.020 | 0.056 ± 0.032 | 0.072 ± 0.030 | **0.976 ± 0.020** | **0.984 ± 0.032** |\n| FetchSlide | 0.000 ± 0.000 | 0.000 ± 0.000 | 0.016 ± 0.020 | 0.112 ± 0.096 | **0.240 ± 0.044** |\n| FetchPickandPlace | 0.048 ± 0.030 | 0.040 ± 0.025 | 0.120 ± 0.080 | **0.744 ± 0.103** | **0.840 ± 0.119** |\n| FetchStack2 | 0.008 ± 0.016 | 0.008 ± 0.016 | 0.008 ± 0.016 | 0.840 ± 0.110  | **0.920 ± 0.057** |\n\nSPReD method surpasses all variants of AWAC for HalfCheetah. AWAC is competitive for Hopper and Walker2d with near-expert demonstrations. However, the learning score of AWAC is asymptotically lower than SPReD for all three locomotion tasks we tested, which will be visualised with the figure of learning curves in our final manuscript. \n\nAlthough AWAC achieves moderate performance in locomotion tasks, our experiments show that AWAC struggles significantly in our setting with few demonstrations and environments with sparse rewards. To give AWAC the best chance, we also implemented a variant that resamples demonstrations during online training (similar to our approach).  However, the benefit of continuously sampling demonstrations is negligible for AWAC on both locomotion and manipulation domains. SPReD's uncertainty-aware weighting significantly outperforms AWAC because it explicitly reasons about when demonstrations remain useful rather than relying on advantage estimates from limited data. AWAC's poor performance stems from a fundamental mismatch with our problem setting:\n\n1. **AWAC assumes large offline datasets** (~1M transitions) for effective pretraining, but we have only ~5K demonstration transitions. Without sufficient pretraining data, and since demonstrations are quickly diluted in the replay buffer, their impact fades early, effectively reducing AWAC to standard RL.\n2. **AWAC's advantage weighting** assumes the offline data covers a substantial portion of the state space, which doesn't hold with sparse demonstrations\n\nWe will include complete AWAC results on other tasks and learning curves in the revised manuscript.\n\n### References\n\n[1] Lakshminarayanan, B., Pritzel, A., Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. *Advances in Neural Information Processing Systems*, 2017, 30.\n\n[2] Osband, I., Blundell, C., Pritzel, A., et al. Deep exploration via bootstrapped DQN. *Advances in neural information processing systems*, 2016, 29.\n\n[3] Nair, A., Gupta, A., Dalal, M., et al. Awac: Accelerating online reinforcement learning with offline datasets. *arXiv preprint arXiv:2006.09359*, 2020."}, {"review_id": "aXarBFuXd4", "reviewer": "Reviewer_npaw", "summary": "The paper proposes a novel method for combining demonstrations with online off-policy reinforcement learning. This is motivated by the observation that existing methods, like Q-filter, make hard, binary decisions about when to imitate demonstrations based on point estimates of Q-values. The authors instead propose to learn an ensemble of Q-functions to capture epistemic uncertainty. Deciding whether to imitate a demonstrated action can then be phrased as comparing the distributions of Q-values of the demonstrated and policy action induced by the Q-function ensemble. Two variants of this idea are presented: SPReD-P assumes Q-values follow a Gaussian distribution and computes behavioral cloning (BC) weights from the probablity that the mean of demonstration Q-values is higher than the mean of the policy Q-values. SPReD-E computes the BC weights based on an exponential of the difference of the means of the two Q-value distributions, normalized by their interquartile range. Both variants result in continuous BC weights that take the uncertainty of the Q-value estimates into account. A theoretical analysis shows that this reduces the variance of the BC weights compared to Q-filter, and that the weights behave intuitively in different limits. An experimental evaluation demonstrates that both versions of SPReD outperforms competitive baselines on a range of continuous control tasks.", "questions": "### Questions\n* Why is there no initial state distribution mentioned in the “Setup” paragraph (line 108)? In line 114 the state is sampled from $E$ but $E$ is never introduced, I think.\n* In line 261 f. in Property 5.3, the statement “both weighting schemes implement similar degrees of caution” seems a bit odd because at $A=0$ on of them has weights 0.5 and the other one 0.0.\n### Suggestions\n* The sentence “Adapting to continuous spaces, policy with soft constraint from demonstrations [11] introduces substantial computational complexity.” in line 31 sounds awkward.\n* The learning curves look a bit crowded. As a result, it’s sometimes hard to discern which method is better. Smoothing them before plotting (and noting so somewhere) could help.\n* Line 112: “in replay buffer $R$\", forgot article. Also $R$ is already used for the reward function so it might be better to choose another symbol.\n* Also in line 112: “with exploration noise added to actions”. This sounds like the noise is only added when saving the actions to the replay buffer. However, this is not correct, right?\n* In Property 5.1 in line 241 an $\\alpha$ makes an appearance. Should this be a $\\beta$?\n* The caption of Figure 1 states that the SPReD methods are shown in red and purple but it’s actually red and brown as far as I can see.", "limitations": "Yes, the authors discuss limitations in the conclusion.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths: \n* The paper is well-written and structured.\n* The motivation for the proposed algorithm is made clear early on.\n* The theoretical analysis is a good addition to the paper.\n* The empirical results are strong, in particular on challenging tasks like stacking, and back the claims in the introduction.\n* The ablation studies help understand how SPReD and the baselines perform for different demonstration qualities and sample sizes.\n### Weaknesses: \n* The proposed method builds on Q-filter but uses smooth, uncertainty-aware BC weights. That makes it somewhat incremental. However, the clear motivation for doing this and the strong empirical results make the paper relevant nevertheless.", "was_revised": true, "final_justification": "I recommend acceptance of the submission for the following reasons:\n* The contributions of the paper in the area of combining learning from demonstrations and online interactions are significant, well-motivated, and relevant to the community.\n* The paper is well-written.\n* The empirical results are strong.\n* The theoretical analysis matches the proposed algorithm variations and help understand their properties.\n\nDuring the rebuttal the authors adressed my questions and outlined revisions to the camera-ready paper that will improve consistency and presentation of the paper. All points raised in my review were sufficiently addressed.", "rebuttal": "We sincerely thank you for your thorough review and positive assessment of our work. We appreciate your recognition of our contributions and your constructive feedback that helps improve the clarity of our presentation. Below, we address your specific questions and incorporate your helpful suggestions.\n\n### Questions\n\n**Q1: Missing initial state distribution (line 108).** Thank you for identifying this oversight. We have added the initial state distribution $ρ_0$ to the MDP framework in the Setup paragraph. To clarify the notation in line 114: $E$ represents the environment defined by our MDP, and $s_0 \\sim ρ_0$ denotes sampling the initial state from this distribution. The complete MDP is now properly defined as $\\mathcal{M} = (\\mathcal{S}, \\mathcal{A}, P, R, γ, ρ_0)$, ensuring notational consistency throughout the paper.\n\n**Q2: Property 5.3 - apparent inconsistency at $A=0$.**  We appreciate you highlighting this potentially confusing point. While it may seem odd that $p_P = 0.5$ and $p_E = 0$ at $A=0$, this reflects their different interpretations of uncertainty:\n\n- **SPReD-P** (probabilistic): When $A=0$, the Gaussian distributions overlap completely, yielding $\\mathbb{P}(Q_d > Q_π) = 0.5$.\n- **SPReD-E** (exponential): Using $e^{A/β} - 1$, which equals 0 when $A=0$, enforces imitation only for positive advantages.\n\nThe key insight from Property 5.3 is that despite this difference at $A=0$, both methods exhibit similar *rates of change* near $A=0$. Specifically, when $β = σ\\sqrt{2π}$, we show that $p_P - 1/2 ≈A/β≈ p_E$, meaning both follow similar trends when advantages are small compared to the uncertainty. This relationship is visualised in Figure 7 (Appendix, page 22). We have clarified this interpretation in the main text.\n\n### Suggestions\n\n- **Line 31**: Revised to “Jing et al. [11] treat demonstrations as a soft constraint on policy exploration, formulating a constrained policy optimisation problem. Although they reduce overhead by applying a local linear search on its dual, the approach still involves considerable computational complexity.”\n- **Learning curves**: We have applied 5-point moving average smoothing to improve readability and noted this preprocessing in all figure captions.\n- **Notation**: Changed replay buffer notation from $\\mathcal{R}$ to $\\mathcal{B}$ throughout to avoid conflict with the reward function.\n- **Exploration noise**: Clarified that noise is added in two contexts: (1) during action execution for exploration, and (2) during critic target computation for smoothing (both following TD3 design).\n- **Property 5.1**: Corrected to use $α$, which is the normalisation constant in $β = α ×$ IQR as defined in Appendix C (page 18). We have improved consistency between the methodology section and theoretical properties.\n- **Figure 1 caption**: Corrected to accurately describe SPReD methods as \"red and brown\" lines.\n\nAll the corrections and clarifications mentioned above have been incorporated into our revised manuscript. Since we cannot upload a revised PDF during the rebuttal phase, these changes will be reflected in the camera-ready version if the paper is accepted. We are happy to provide any additional clarifications if needed."}, {"review_id": "eLQDa60Teg", "reviewer": "Reviewer_hmL7", "summary": "This paper tackles the setting of demonstration-guided RL and when the agent should imitate a demonstration versus prioritizing the RL objective based on the estimated Q-values.  The authors propose an uncertainty-aware, smooth regularization weight which has lower gradient variance compared to prior methods that apply a binary weight.  They learn an ensemble of Q-functions and propose two variants of their method: 1) based on the probability that the demonstration Q-value is greater than the policy's and 2) based on the magnitude of difference between the two Q-values.  The theoretical results demonstrate that continuous regularization weights result in lower gradient variance and characterize the behavior of the weights.  The empirical results show their method outperforms other binary weighting methods in multiple manipulation tasks.", "questions": "1. Why did you not include experiments in other domains or use some of the benchmarks from prior work?\n\n2. Why did you use different quality demonstrations for each task?  Was the performance sensitive to how sub-optimal the demonstration were depending on task?\n\n3. Why did you pick 1 million timesteps for the results in Table 1?  Given the learning curves in Figure 1, Table 1 does not seem necessary, especially since each task seems to converge at a different rate.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Identified a clear limitation of prior methods in doing binary regularization weighting for demonstrations.\n2. Methods are sound for the most part and theoretical results provide some understanding for how they work.\n3. Strong empirical results.\n4. Well-presented and written.\n\nWeaknesses:\n1. The derivation and contribution of SPReD-E are unclear to me.  The paper claims that this measure should \"respect the distributional nature of our estimates\" and \"compare $x$ in the support of $\\mu$ to $y$ in the support of $v$\", suggesting that this measure should be defined probabilistically.  However, the final measure is a simple difference between means, so it cannot really be called uncertainty-aware.\n\n2. My main concern is that this paper lacks evaluation in other domains.  All the results are on OpenAI manipulation tasks.   Meanwhile the prior works the authors cited had experiments in D4RL, locomotion, and game tasks.  This is the main reason I am on the fence for acceptance.\n\n3. It's unclear to me what the contribution of the ensemble is versus the continuous regularization weights, especially for SPReD-E.  Do you have results ablating the two?", "was_revised": true, "final_justification": "I raised my score from a 4 to 5 since my main concern was that the experiments were all done in one domain.  The authors provided additional experiments in locomotion tasks in the rebuttal which addressed this concern.", "rebuttal": "We sincerely thank you for your thoughtful review and constructive feedback. We appreciate your recognition of our key contribution—identifying and addressing the limitation of binary regularisation in demonstration-based RL—as well as your positive assessment of our theoretical analysis and empirical results. Your acknowledgment that our methods are sound and the paper is well-presented is particularly encouraging. We address your concerns and questions below to further strengthen our contribution.\n\n**SPReD-E's uncertainty-aware nature:** We appreciate the reviewer's careful reading of our derivation. While the final formulation $A = \\mathbb{E}[Q_d] - \\mathbb{E}[Q_π]$ is indeed a difference of means, this emerges from a principled approach that considers the distributional nature of our estimates. In the end, we need a partial ordering on the set of probability measures. We imagine there are many such orderings that are more exciting than the one we have used here. However, we justify our choice based on (1) our belief it is better to start simpler and (2) empirical performance of the method suggests it is state-of-the-art.\n\nSPReD-E incorporates uncertainty through the normalisation term $β$, which captures the spread of both Q-value distributions:\n\n**$β = α × \\frac{1}{2}[IQR(Q(s_d,a_d)) + IQR(Q(s_d,π(s_d)))]$.**\n\nThis makes SPReD-E fundamentally uncertainty-aware:\n\n**$p_E = clip(e^{A/β} - 1, 0, 1)$**\n\nThe key insight is that the same advantage $A$ produces different weights depending on our estimation uncertainty:\n\n- **High uncertainty** (large $β$): $e^{A/β} ≈ 1 + A/β$ → small weights (conservative)\n- **Low uncertainty** (small $β$): $e^{A/β}$ grows rapidly → larger weights (confident)\n\n**Example**: For $A = 2.0$:\n\n- High uncertainty ($β = 10$): $p_E = e^{0.2} - 1 ≈ 0.22$\n- Low uncertainty ($β = 1$): $pE = e^{2} - 1 ≈ 6.39 → 1.0$ (after clipping)\n\nThe transport map formulation (lines 206-211) provides the theoretical foundation for why comparing distributions reduces to comparing means when using optimal transport. While the final implementation is computationally simple, this derivation justifies why $A = \\mathbb{E}[Q_d] - \\mathbb{E}[Q_π]$ is a principled choice for measuring distributional differences. The uncertainty-awareness comes from scaling this advantage by the combined uncertainty of both distributions.\n\n**Limited domain evaluation:**  We appreciate your interest in seeing broader domain evaluation. Our choice of manipulation tasks was deliberate and well-motivated—these environments feature sparse rewards and precise control requirements where demonstration guidance provides the most value, making them ideal for evaluating our method. Our evaluation includes all robotic manipulation tasks tested in the most relevant baseline—Q-filter and hand manipulation tasks in addition. \n\nD4RL, used by methods like RLPD, serves a fundamentally different purpose. It provides massive offline datasets (1-2M transitions) for pure offline learning, whereas SPReD tackles online RL with a relatively small number of demonstrations (~5K transitions). Testing SPReD on D4RL would require either:\n\n- Discarding 99.5% of the data to match our sparse demonstration setting\n- Abandoning our core research question of learning from limited demonstrations\n\nNeither approach would meaningfully evaluate our contribution. Notably, our main baseline Q-filter was also evaluated exclusively on manipulation tasks, providing appropriate comparison.\n\nTo address your comments, we have now expanded our evaluation to Hopper, HalfCheetah and Walker2d in locomotion domain to demonstrate generalisation:\n\n| Task | TD3 | EnsTD3 | Q-filter | EnsQ-filter | RLPD | SPReD-P | SPReD-E |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Hopper | 1468 ± 849 | 2144 ± 966 | 2818 ± 438 | 2930 ± 546 | 2744 ± 580 | **3246 ± 18** | 2740 ± 344 |\n| HalfCheetah  | 3728 ± 419 | 3775 ± 1756 | 4671 ± 760 | 7188 ± 934 | 6721 ± 3542 | **8060 ± 171** | 7336 ± 601 |\n| Walker2d  | 1861 ± 1159 | 2334 ± 772 | 1485 ± 589 | 3576 ± 528 | **4519 ± 154** | 3351 ± 842 | 2403 ± 1154 |\n\nThe table presents scores at 200K steps to assess the sample efficiency. Though EnsQ-filter without continuous regularisation and RLPD achieve competitive performance in these environments with dense rewards and near-expert demonstrations, SPReD achieves high performance for all three tasks and is particularly outstanding on HalfCheetah (SPReD-P gains 12% improvement from EnsQ-filter and 72% improvement from Q-filter), confirming that our uncertainty-aware approach effectively transfers across different continuous control domains. \n\nWhile we believe our current evaluation comprehensively validates our approach, we agree that exploring additional domains is helpful and will include more results (e.g., other locomotion tasks Ant and Humanoid with demonstrations) with learning curves in our final manuscript.\n\n**Contribution of ensemble vs. continuous regularisation:** We totally agree—since SPReD introduces both continuous weights and ensemble-based uncertainty estimation, it's important to understand their individual and combined effects. We conducted a systematic ablation study to isolate these contributions. By varying ensemble size (2 vs. 10) and regularisation type (binary vs. continuous), we can assess each component's impact (with success rates at 1M steps):\n\n|  | Q-filter | EnsQ-filter | SPReD-E (Ensemble size  2) | SPReD-P (Ensemble size  2) | SPReD-E (Ensemble size 10) | SPReD-P (Ensemble size  10) |\n| --- | --- | --- | --- | --- | --- | --- |\n| FetchPickandPlace | 0.584 ± 0.041 | 0.648 ± 0.039 | 0.120 ± 0.240 | 0.712 ± 0.064 | 0.840 ± 0.119 | 0.744 ± 0.103 |\n\nOur key findings are as follows:\n\n- *Continuous regularisation is the primary driver:* Even with a minimal ensemble (size 2), SPReD-P improves over Q-filter by 22% (0.712 vs 0.584). This demonstrates that smooth, uncertainty-proportional weights are fundamentally better than binary decisions, regardless of ensemble size.\n- *Ensembles add complementary value*: Expanding from 2 to 10 critics further boosts performance for both SPReD variants. However, the gains are method-specific—SPReD-P shows modest improvement (+4%, 0.712 vs 0.744) while SPReD-E shows dramatic improvement (+24%, 0.120 vs 0.840 ).\n- *The methods have different uncertainty requirements:* This differential benefit reveals an important insight: SPReD-E's exponential weighting critically depends on accurate uncertainty estimates through $β$. With only 2 critics, the IQR calculation is noisy, leading to suboptimal scaling. In contrast, SPReD-P's probabilistic approach is inherently more robust to limited ensemble sizes.\n\nThe ablation confirms that while both components contribute independently, they work synergistically. Continuous regularisation provides the foundation for better learning, while larger ensembles enable more precise uncertainty quantification—particularly crucial for SPReD-E's exponential scaling mechanism. This validates our design decision to combine both innovations rather than pursuing either in isolation. We will include this ablation study, with full learning curves, in the revised manuscript to make these contributions explicit.\n\n### Questions\n\n**Q1:**  Please see our response to Weakness 2 above, where we present new locomotion results and explain why benchmarks like D4RL (designed for offline RL with 1-2M transitions) are incompatible with our sparse demonstration setting.\n\n**Q2:**  We deliberately varied demonstration quality across tasks to evaluate robustness in realistic scenarios where expert demonstrations may not always be available.\n\nTo directly address sensitivity: Figure 2 systematically compares all methods across three quality levels (expert, suboptimal, severely suboptimal) on fixed tasks. SPReD consistently outperforms baselines at every quality level, demonstrating robustness rather than sensitivity. Even with 99% random demonstrations mixed with 1% expert data, SPReD methods successfully filter out noise and achieve reasonable performance, while baselines fail catastrophically.\n\nThis robustness is by design—our uncertainty-aware weighting naturally adapts to demonstration quality. Poor demonstrations lead to low advantage and high uncertainty with respect to Q-estimates, resulting in smaller weights that diminish their influence. All demonstration-based methods perform better with higher-quality data, but SPReD consistently outperforms other demonstration-based methods and is not misled by low-quality demonstrations.\n\n**Q3:**  Table 1 provides a standardised comparison point, following prior fundamental work (e.g., Table 1 in TD3 [1], Table S3 in A3C[2]). This enables fair comparison across methods and reproducible benchmarking for future work. We chose 1M steps as it allows meaningful learning progress while remaining computationally tractable for all baselines. \n\nThe combination of Table 1 and Figure 1 provides complementary information: Table 1 enables quick numerical comparison of sample efficiency, while Figure 1 shows complete learning dynamics and convergence behaviour. Together, they offer a more comprehensive view than either would individually.\n\n### References\n\n[1] Fujimoto, S., Hoof, H., Meger, D. Addressing function approximation error in actor-critic methods. In *International conference on machine learning*, pages 1587–1596. PMLR, 2018.\n\n[2] Mnih, V., Badia, A.P., Mirza, M., et al.  Asynchronous methods for deep reinforcement learning. In *International conference on machine learning*, pages 1928-1937. PMLR, 2016."}]

Author rebuttal:

Meta-review: Summary: The paper introduces SPReD (Smooth Policy Regularisation from Demonstrations), a framework for reinforcement learning with demonstrations that replaces binary Q-filter decisions with continuous, uncertainty-aware weighting. Two variants are proposed: SPReD-P, which uses probabilistic estimates of demonstration superiority, and SPReD-E, which applies exponential advantage weighting. Theoretical analysis shows reduced gradient variance and adaptive down-weighting of suboptimal demonstrations, while empirical results across eight robotics tasks demonstrate strong improvements over TD3, Q-filter, EnsQ-filter, and RLPD, with gains especially pronounced in complex stacking environments.

We received three expert reviews, with scores, 4, 5, 5, and the average review is 4.67. The reviewers appreciated theoretically grounded approach, including the variance reduction guarantees and properties showing adaptive handling of uncertainty and suboptimal demonstrations. The reviewers also appreciated strong empirical results. Additional experiments provided during rebuttal (locomotion benchmarks) strengthened generality claims. The paper is well-written, logically structured, and supported by ablations on demonstration quality and size.  While the reviewers raised some concerns and questions, the authors have successfully addressed all of them, and the reviewers have acknowledged and appreciated this effort. 

Overall, the paper offers a principled and effective contribution to RL with demonstrations. While not a significant departure from prior work, the demonstrated improvements and theoretical grounding make it a valuable contribution for the community.
Decision: Accept (poster)