Tackling Heavy-Tailed Q-Value Bias in Offline-to-Online Reinforcement Learning with Laplace-Robust Modeling
Authors: Ruibo Guo
Lei Liu
Rui Yang
Junjie Shen
Guoping Wu
Jie Wang
Bin Li
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Offline-to-Online
Q-value Estimation
Laplace Distribution
Primary area: reinforcement learning
TL;DR: In offline-to-online reinforcement learning, we reveal that the Q-value estimation bias often follows a heavy-tailed distribution and propose a novel Laplace-based method to enhance training stability and performance under these biases.

Abstract: Offline-to-online reinforcement learning (O2O RL) aims to improve the performance of offline pretrained agents through online fine-tuning. Existing O2O RL methods have achieved advances in mitigating the overestimation of Q-value biases (i.e., biases of cumulative rewards), improving the performance. However, in this paper, we are the first to reveal that Q-value biases of these methods often follow a heavy-tailed distribution during online fine-tuning. Such biases induce high estimation variance and hinder performance improvement.
To address this challenge, we propose a Laplace-based robust offline-to-online RL (LAROO) approach. LAROO introduces a parameterized Laplace-distributed noise and transfers the heavy-tailed nature of Q-value biases into this noise, alleviating heavy tailedness of biases for training stability and performance improvement. Specifically, (1) since Laplace distribution is well-suited for modeling heavy-tailed data, LAROO introduces a parameterized Laplace-distributed noise that can adaptively capture heavy tailedness of any data. (2) By combining estimated Q-values with the noise to approximate true Q-values, LAROO transfers the heavy-tailed nature of biases into the noise, reducing estimation variance. (3) LAROO employs conservative ensemble-based estimates to re-center Q-value biases, shifting their mean towards zero. Based on (2) and (3), LAROO promotes heavy-tailed Q-value biases into a standardized form, improving training stability and performance. Extensive experiments demonstrate that LAROO achieves significant performance improvement, outperforming several state-of-the-art O2O RL baselines.

Published as a conference paper at ICLR 2026
TACKLING HEAVY-TAILED Q-VALUE BIAS IN TFFLINE-TO-ONIEE REINFORCEMENT LEARNING WITH LAPLACE-ROBUS, MODELING
Ruibo Guo', Lei Liu' ,, Rui Yang", Junjie Shen', Guoping Wu', Jie Wang', Bin Li* University of Science and Technology of China
China Mobile
fruiboguo, yr0013, shenjunjie, (ooping)mmali 1. ustc. edu. cn [liulei13, jiewangx, binli]@usto, eerebustceeuu.....
ABSTRACT
Offline-to-onling reinforcement learning (020 RL) aims to improve the perfor- mance of offline pretrained agents through online fine-tuning. Existing O20 RI methods have achieved advances in mitigating the overestimation of Q-value bi- ases (i.e., biases of cumulative rewards), improving the performance. However, in this paper, we are the first to reveal that Q-value biases of these methods often follow a heavy-tailed distribution during online fine-tuning. Such biases induce high estimation variance and hinder performance improvement. To address this challenge, we propose Laplace- robust offline-to-online RL (LAROO) ap- proach. LAROO introduces a parameterized Laplace-distribute noise and trans- fers the heavy-tailed nature of Q-value biases into this noise, alleviating heavy tailedness of biases for training stability and performance improvement. Specifi- cally, (1) since Laplace distribution is well-suited for modeling heavy-tailed data, LAROO introduces a parameterized
aaplace-distribute noise that can adaptively capture heavy tailedness of any data. (2) By combining estimated Q-values with the noise to approximate true Q-values, LAROO transfers the heavy-tailed na- ture of biases into the noise, reducing estimation variance. (3) LAROO employs conservative ensemble-based estimates to re-center Q-value biases, shifting their mean towards zero. Based on (2) and (3), LAROO promotes heavy-tailed Q-value biases into a standardized form, improving training stability and performance. Ex- tensive experiments demonstrate that LAROO achieves significant performance improvement, outperforming several state-of-the-art 020 RL baselines.
INTRODUCTION
Offline-to-oniin reinforcement learning (020 RL) has drawn considerable attention in recent re- search, concentrating on enhancing the performance of offline pretrained agents by online fine- tunings (Lee et al. 2021b Nair et al. 2020 Zhao et al. 2022). Since the performance of pretrained agents heavily limited by the quality and state-action space coverage their offline datasets (Jin) ct al. 2) 020 RL proposes to fine-tune pretrained agents in online environments. It has the appealing potential to achieve notable performance improvement within just a few online environ- mental interaction steps (Feng al. 2024).
Recently, several 020 RL methods (Lee et al. 2021b Feng et al., 2024) highlight that pretrained agents confront significant aatt-ncciinn distribution shift between offline and online data during the fine-tuning process. This shift causes pretrained agents to inaccurately estimate Q-values in online data and misguide update directions, resulting in slow performance improvement and instability dur- ing online fine-tuning (Zhang et al.] (2024). Therefore, the aforementioned methods employ various techniques improve the Q-value estimation. Specifically, some methods incorporate a conser- vative penalty into the estimated Q-values (Zhao et al.) 2022 Nakamoto et al.) 2023) or employ
ensemble models (Zhao al. 2022 2024 to enable conservative and stable Q-value estimation in
"Corresponding author.

---

Published as a conference paper at ICLR 2026
CalQu
PEEX
502
Pereerzage (h6
20-
25-
Mosn: $5.12 Variance: 11552.79
Karioss: 6.54
Parcersage (60
12:
14-
Variance: 21:381 94
Kurtuiss 8,48
78.81
Pouenntage
5.0
40
Kurossii 9.00
15199.64
Parcentage (75)
30
Mean: 22.04
Kertoiic 8.97
Vorsance: 15 667.75
20
0
-100
100 200 300
450 300
0
200 403
6:0
.100
0
100
209 s0o
400 500
sco
-100
150
200 300 400 300 690
Q biss
Q
bins
Q hras
has
Figure| 1: The heavy-tailed Q bias distribution. We present Q bias distributions of four 020 RL
methods (Cal-QL (Nakamoto et al. (023). PEX (Zhang al. (023), ENOTO (Zhao et al. 2024), SO2 ZZang et al., (024)) for online fine-tuning at the 100k online step in the Walker2d-medium task. Please refer Figure and Figure [ for more training steps and various tasks.
online data. Other methods (Fng et al. 2024 Zhang et al. 2024) propose to increase the update frequency of models to facilitate the accurate Q-value estimation.
In this paper, based on a comprehensive study of the Q-value estimation in 020 RL. we reveal that Q-value estimation bias (i.e.. Q bias) often follows a heavy-tailed distribution during fine-tuning for the first time. In Figure [ we illustrate the distribution of Q bias for several 020 RI. methods. The results clearly show that the Q bias displays substantially heavier tails than the normal distribution, and exhibits positive skewness with notably high biases in the right tail. This heavy-tailed property of Q bias is inherently caused by distribution shift problem and persists throughout online fine- tuning. Consequently, the heavy-tailed Q bias challenges the widely adopted assumptions of finite variance Gaussian modeling for such biases in prior research (Duan al. 2021 D' D'Eramo al. 20216 Chen et al., 2021), thereby questioning their broad applicability.
In practice, the heavy-tailed Q bias significantly affects both Q-value estimation and policy updates in 020 RL. These heavy-tailed biases lead to extreme overestimation of Q-values and misguide agents to choose poor actions, causing large performance degradation during online fine-tuning. Meanwhile, high Q biases can lead to large fluctuations and even collapse of Q-value estima- tion (Zhang al., 2024: Yang al.] (0244). It is widely recognized that heavy-tailed nature of bias induces large estimation variance and impedes the convergence of Q-values, resulting in slow performance improvement (Shao et al.) 20188] Zhuang Suil 202T). Since the aforementioned O20 methods fail to account for these heavy-tailes Q biases and mitigate their impact, these methods continue to face challenges of instability and inefficiencies in performance improvement.
To tackle these challenges, we propose novel Laplace- based robust offline-to-onlnee RL (LAROO) approach. The central idea of LAROO to explicitly model and mitigate heavy-tailed Q-value bi- ases by transferring their heavy-tailed nature into parameterized Laplace-distributed noise. Since the Laplace distribution is well-suited for modeling heavy-tailed data (Bai) 1995 Yang et al. 2019), this formulation accurately represent the nature of Q-value biases, thus enhancing the Q-value esti- mation. Specifically, LAROO is built upon three key components: (1) Adaptive laplace-distributed noise modeling. LAROO introduces parameterized Laplace- Laplace-distributed noise that adaptively cap- tures heavy-tailed characteristics, providing a principled way to represent such heavy-tailed distri- butions. (2) Variance reduction via learning process that incorporates noise. By combining noise to approximate the true Q-values, LAROO
estimated Q-values with the Laplace-distribee transfers the heavy-tailed nature of biases into the noise distribution, effectively reducing the high estimation variance of Q values. (3) Bias Re-centering. For stability, LAROO further employs con- servative ensemble-bast based estimates to re-center Q-value biases, shifting their mean toward zero. By jointly leveraging (2) and (3), LAROO promotes the heavy-tailed Q-value biases into a standardized form, thereby significantly enhancing training stability and performance.
This study incorporates the Laplace-distributedd noise model and conservative Q-value ensemble models for robust Q-value estimation in 020 RL, facilitating efficient and stable performance im- provement. It extends prior analysis of Q bias beyond the mean variance, to the full distribution in 020 RL. Moreover, we theoretically show that LAROO reduces the single-step estimation bias compared with the l2 loss during training, thereby accumulating less bias over time. Experimental results show that LAROO represents a meaningful advancement by effectively mitigating the impact of heavy-tailed Q bias. We summarize our contributions as follows:
2

---

Published as a conference paper at ICLR 2026
To the best of our knowledge, for the first time, we reveal that the Q-value estimation bias in existing 020 methods often follows a heavy-tailed distribution during online fine-tuning, which leads to instability and inefficiencies in performance improvement in O20 RL.
We propose LAROO, a aaplcc--based robust deffline-to-oningg RL approach that introduces Laplace-distributed noise to alleviate heavy-tailed biases and uses conservative ensemble- based estimates to re-center bias, improving training stability and performance.
We provide theoretical analysis demonstrating that LAROO reduces single-step estima- tion bias compared to the conventional l2 loss function used in Q-value updates.
Extensive experiments demonstrate that LAROO achieves significant performance im- provement, outperforming several state-of-the-art baselines across various tasks.
2 PRELIMINARIES
We follow the standard protocol that formulates a RI. environment as a Markov decision process (MDP). The MDP M is often described as a tuple M (S, R. ?). where S is the state space, A is the action space, P. S x A + (SS) is the transition function, R S x A - IR is the reward function and y E [0, 1) the discount factor. The agent aims to learn policy π T[a () that maximizes the expected return Ex EE== r'r,]. where
R(s,,a))
Off Off-policy RL methods employ a Q-value network Qa a) estimate the expected cumulative dis- counted return starting from the state-action pair (s, a). defined as E,, IE y're sp S, ag = a]. which also known as the true Q-value of (s, a). However, the Q-value network inevitably in- troduces estimation biases in estimated Q-values, i.e., Q biases. Off-policy RL employs policy evaluation (i.e., Q-learning) to update the Q-value network Qo(s,
Jo(Qu) == -B(aaar,e') ((Q(ss.)-TQu(ssa))))))
- TQu(s,
(I)
where B is the replay buffer, (Qo a) is the Q-value update target defined as R(s, a) + (.Es-T(()ss.) Imax 2B (s',a') and the temporal difference (TD) error is denoted as ds(s,a) Qa(s,a) - TQo(s, a). In this study, given a (s,a) pair, we denote the true Q-value as ((s,a).
Q- VALUE ESTIMATION BIAS IN O20 RL
In this section, reveal the heavy- Q bias and analyze their influence. We further highlight the limitations of previous 020 methods in effectively mitigating their impact.
The estimation bias follows heavy-tailed distribution in online data during fine-tuning. We evaluate several existing offline and 020 RL. methods across various tasks and compute the Q bias in online data. Specifically, we collect online data into the replay buffer at every online step. For each data, we obtain its true Q-value using Monte Carlo strategy and the estimated Q-value with the Q network, then calculate their difference to get the Q bias. As shown in Figure I 5 and 6 these results present that Q biases exhibit a heavy-tailed nature in online data, and this heavy-tailed nature universal across various tasks and persists throughout the online fine-tuning process.
We employ the metric Kurtosis k = =======a> D. ( of heavy-tailedness relative normal distribution (Mardia 1970; Garg al. 202T.. Additionally, we compute the mean and variance of the Q bias. A positive mean indicates Q value overestimation, whereas negative mean indicates underestimation. Our observations reveal that 020 RL, offline algorithms typically underestimate Q-values, whereas existing 020 RI methods tend to overesti- mate them during fine-tuning. Importantly, regardless of whether the Q-values are overestimated underestimated, the Q bias consistently demonstrates high heavy-tailedness. We present detailed experiments for Q bias in Appendix A.2
(a;; 2) 2
to measure the degree
Explaining the Heavy-Tailed Q-Bias in Offline-to-Online RL. The heavy-tailed nature of Q-bias is attributed to two main factors: (i) the Q-function with Max operator amplifies overestimation by selecting large bellman target, and (ii) non-uniform distribution shift between online and offline data. Specifically, pretrained Q-value networks have large Q bias in online samples that lie far from the offline data distribution, which manifest as the bias distribution's 's tails. In contrast, online
3

---

Published as a conference paper at ICLR 2026
samples closer the offline distribution result in minor biases and constitute the central mass of Q bias distribution.
We validate factor (ii) through experiments by computing the shifted distance of online data from the offline distribution using two distance metrics. We validate a positive correlation between Q bias and shifted distance with Spearman's 's rank correlation coefficient. As shown in Figure 2 the Q bias of online samples tends to increase as their distance from the offline data distribution grows. Please refer Appendix A.3 for details.
bupper modur eppere2
Ieposteedaa exped-12
Crucly Noine
155
Nommllooe ioetm
Pazs Dcasity
Valoe
092
as.
Cuuctty Nuis:
Noise
20
Dox Dogg Tismne))
Decy (UNN Dezazze)
Timcalopo
Timostgs
Figure 2: Left figures: The non-uniform distribution shift contributes to heavy-tailed Q bias. The Q bias of online samples often increases as their distance (DRNN. DRR) from offline data distribution grows. Right figures: Validation of the instability caused by heavy-tailed Q biases. Heavy-tailed Cauchy noise causes greater training instability and lower performance.
The heavy-tailed Q bias could induce instability during the online fine-tuning process. When Q bias exhibits high heavy-tailedness, the back propagation of Q-learning amplifies these extreme bi- ases over time, exacerbating this issue. Additionally, large Q biases result in inaccurately estimated Q-values, which may cause Q-value estimation collapse (Zhang et al.] 2024). As a result, addi- tional training steps are required to correct these inaccuracies, resulting in inefficient performance improvement. Furthermore, existing 020 RL methods often rely on the l2 loss function for Q-value updates. This reliance may cause heavy-tailed Q biases to disproportionately influence the Q-value loss and dominate the update gradients, further destabilizing Q-value updates.
We also validate the issues caused by the heavy-tailed bias through experiments. For approximat- ing the heavy-tailed Q bias, we incorporate heavy-tailed noise into the estimated Q-values during Q-value updates. Specifically, for fair comparison, we respectively introduce Gaussian noise and heavy-tailed Cauchy noise, both with the same mean and variance, into two agents from the same pretrained one. The results in Figure 9 show that the heavy-tailed Cauchy noise leads more inac- curacies in Q-value estimation and lower final performance. See details in Appendix A.4
It is necessary to take into account the heavy-tailed Q bias in Q-value estimation. Previous re- search mainly focuses on reducing the mean or variance of Q bias, neglecting its underlying dis- tributional characteristics. For instance, Cal-QL (Nakamoto al.] (2023) employs conservative Q- learning to decrease positive Q bias. ENOTO (Zhao et al. 2024) introduces ensemble models with SUNRASE (Lee et al.) (021a) to reduce the mean and variance of Q bias. However, as results shown in Figure and 6 while these methods can effectively reduce the mean of Q bias. they still struggle to decrease its variance, and the kurtosis values remain high. This phenomenon highlights that ex- biases and reduce their impact. Therefore, it is crucial to address these biases in Q-value estimation to enhance training stability.
isting methods still struggle to mitigate heavy-tailed
In summary, in fiffine--t-onling RL, the Q bias exhibits high tailedess in online data during fine-tuning, regardless of whether the Q-value is overestimated or underestimated. These heavy- tailed Q biases arise from the non-uniform distribution shift and cause Q-value estimation to cillate, further resulting in training instability and slow performance improvement. Therefore, is essential to take into account the distribution of Q bias and enhance the training stability of agents under these heavy-tailed Q biases for stable fine-tuning.
4 METHOD
In Section 4.1 introduces an adaptive noise following a Laplace distribution to explicitly capture the heavy-tailed nature of Q-value biases, improving Q-value estimation. We propose a robust 020 approach in Section 4.2 and provide theoretical analysis for its effectiveness in Section 4.3
4

---

Published as a conference paper at ICLR 2026
4.1 LAPLACE-BASER BIAS DISTRIBUTION FOR Q-VALUE ESTIMATION
In robust regression research (Song et al., 2014, Lu & Chang 2022 a promising approach to handle non-normal error distributions is modeling each known for its ability to model data with outliers and connection with the robust regression criteria.
distribution as a Laplace distribution, which is
Inspired by aforementioned approaches, we introduce an independent approximation noise term using the Laplace distribution to explicitly capture the heavy-tailed characteristic of Q bias. Then, we incorporate the Laplace-based noise into the Q-value loss for robust Q-value estimation.
Adaptive Laplace-distribuie noise modeling for heavy-tailed Q bias. To facilitate the analysis of estimation bias, we begin with the assumption for the noise term following (Duan al., 2021
Assumption 4.1 For each (s.a) pair, ((s. a) is the true but unknown Q-value, Ea and are inde- pendent random variables, which denote the approximation noise in target Q-value and estimated Q-value respectively. The approximation noise is assumed to be independent from the (8, a) and O. Then we have:
TQo(s, R(s, ( 7ES s.a)
[max Q3 [s',,')] @(s.a) = (Q,(s. Eg ((x, = Qo(s, a) +
The noise characterizes the discrepancy between true Q-values and their estimates, arising from system noises and function approximation (Duan et al. (202]). Based on the definition of Q-value bias, Bias (Q, a)) TE (s, a)] ((s, a.,, obtain the relationship Bias (as((Q(s,,))) = E[
Then, to better capture the heavy-tailed Q bias, we model the noise using the Laplace distribution as follows, and update adaptively.
Assumption 4.2 (Laplace-basee noise). Eg and ee follow Laplace distributions with the same mean μ4, i.e., Eg ~ Laplace(j, b1). Eq ~ Laplace(μ, b2), P E IR, b1, b2 E R+
Here, is the mean parameter and b is the scale parameter. Specifically, larger values of b correspond heavier tails and higher variance in the Laplace-based noise. We introduce the adaptive updates of noise in next section. Based on the assumption above, we can derive the Laplace- likelihood of Q(s, a) given Qo a). and the Laplace-base based likelihood of ((s, given TQo(s,a)
af Qo(e,)))((
1
(-10(s [Q(s.a) a) Qois, a)
exp
2b2
b2
(3)
p [(( a) TQo(s.a):))
10(s. a) TQols, a)
exp
b,
The Laplace likelihood incorporates the distributional information of Q bias for robust Q-value updates. Then, discuss how to update the Q-value network and how to reduce the impact of extreme biases under the Assumption [.2] Since the likelihood p (Q(s, a) TQQos, ()) incorpo- rates true environmental return information (i.e., future rewards), it serve to the Bayes-optimal posterior (i.e., p( (C a)lall environmental evidence)), compared with the a (Q(s, a) Qo(s, a)). We explain this process in detail Appendix B.2 Therefore, we can train Q-network by minimizing the Kullback-Leibler (KL) divergence between likelihoods of the true -alue, i.e., p [(s,, a) TQQoss, a)) and q (Q(s, a) ())s.
closer approximation
Given a dataset D = (St,, s/ r the Q-value network Q9, and the target Q-network Qø, at the t-th training step, we can estimate or+1 as Equation 4
N
P arg min
(
| [(ss(s an) (s., (s)
(4)
1=0
By integrating over all values of Q(s:,, az). the KL divergence can be transformed as Equation (5) See proof in Appendix B.2 Based on the Bellman equation, the Q-network progressively approxi- mate the true Q-value.
DrL. (1(2(s1))) (T) ((+s (()))(()))))))))
(TQulln.)) - Qu(o),a))) [TQ a.) Qo(si, (
(5)
bi exp
-log b1
b2
b2
5

---

Published as a conference paper at ICLR 2026
We now discuss how the KL divergence addresses the issue of heavy-tailed Q bias, leading more robust training process. When using a Laplace likelihood a)) Qø(s, a)), the resulting negative log-likelihood is proportional to (Q(s,a) (s,s, a)}, which grows linearly rather than quadratically with the Q bias. This down-weights outliers and drives optimization using the central mass of the Q bias, rather than being influenced by rare, extreme biases in the tails.
The Laplace likelihood also better preserves the distributional information of Q bias compared to other robust functions (e.g, 1, loss). If the heavy-tailed bias in bellman target is ignored, i.e., the (((s, a) TQa (s, a)) is a Dirac delta function centered on T (Qa(s, a) the KL divergence reduces the negative log likelihood [g(T2p( a))Qo(s, a)), which exactly equivalent to applying an 1 loss under a Laplace-noiss noise assumption. Detailed analysis is provided in Appendix B.2
Robust Q-value loss function with Laplace noise modeling. We propose a robust function Dx(x) in replace of l2 function for Q-value updates. Compared with Equation [] we set b = b by for practical implementation with fewer hyperparameters. This setting imply states that the Eo and Ea follow the same distribution. We conduct the Mann-Whitne U test (Hettmansperger & McKean Furthermore, we show the derivatives of Dr(x)) even at a 0 and
2010 to validate in C.2.1 provide its more details in B
22 b
Dx(x)= exp
(6)
Finally, we demonstrate the robustness of Dz(x)) in the context of online fine-tuning through two properties: (1) the exponential term exp(-|>=)/ down-weights large Q-value losses in the heavy tails, reducing their impact. (2) the gradient bounded within the range I +) and does not increase with æ, in contrast to the l2 function, where the gradient increases linearly with x. These properties validate the effectiveness of the KL divergence in Equation 4
We observe that the robust property of Dz(x) is inherently connected with the Laplace distribution: as the heavy-tailed Q biases become more frequent, the Laplace-based noise capture the heavy- tailedness by increasing its scale parameter, the gradient bound of Dạ(x) tightens. Consequently, these mechanisms alleviate heavy-tailedness of Q biases, reduce the high estimation variance with noise model, and improve training stability with (z(x) function.
4.2 A ROBUST 020 RL METHOD WITH LAPLACE DISTRIBUTED Q BIAS
We develop a complete 020 method with the Laplace modeling for Q bias in this section.
Adaptively Updating the Suplace-Distribuue the heavy-tailed nature of Q biases with our Laplace noise model, it must adapt to the changing distri- bution of these biases during online fine-tuning. This requires continuously updating the parameters of the Laplace distribution, particularly the scale (variance) parameter b. However, estimators of un- biased sample variance are highly sensitive kurtosis and unreliable in the presence of significant outliers (Yuan et al. 2005). Robust variance estimators that account for kurtosis have been proposed in prior work (Searls& & Intarapanich| 1990 Wencheko & Chipoyera 2009). Among these, we adopt the MSE-best biased estimator (MBBE) (Wencheko & Chipoyera 2009), defined as follows.
Noise with Robust Statistics. To effectively capture
Lemma 4.3 (MBBE of variance). 82 is the unbiased sample variance s²= - E. (X, X)2, n is the sample size and & is the population kurtosis. Then, MSE-best biased estimator 2z is:
n n+1
2
12
71
In practice, the true Q biases not accessible during training. We the readily available TD- error as a statistical surrogate to approximate the variance of Q-bias. This surrogate is reasonable for variance approximation, because we have TQp(s, a) -Qo(s,,) Eg Eg under Assumption 4.1 Then the variance of TD-errors twice the variance of Q bias. We then
(Q(s, a) -a)) (Q(s, a)-Ea)
compute the scale parameter using the MBBE of the TD-errors within a batch (b- S /V2). Our empirical experiments confirm that the scale parameter estimated from TD-errors closely tracks the one estimated from true Q biases, as shown in Figure 17 Crucially, our noise model to capture the heavy-tailedness of Q bias and improve Q Q-luuue estimation according to Section 4.1
adaptively update

---

Published as a conference paper at ICLR 2026
Re-centering Heavy-Tailed Q Bias through Ensemble Models. In our empirical analysis, ob- served that urge-magnitude Q biases in the heavy tails persist under standard training. To mitigate this, we adopt ensemble models known for effectively reducing estimation bias, and compute the target Q-value by taking the minimum over random subset of Q-function estimators (Chen et al.) 2021]. This strategy effectively reduces highly positive biases and re-centers the Q bias distribu- tion because: (1) different Q-functions are approximately independent, positive errors and negative errors are independently sampled across different heads. (2) the probability of selecting the most overoptimistic head is reduced. Then, we integrate the ensemble model with D& (z(x) and derive the loss function for robust Q value estimation as Equation
8 where K the ensemble size and M is
the subset size.
(o:) (e,,,,,,"')
1 K
Do
(Q) -ymin (s,
(8)
0% %. (6, ISKSM
Ymin a, r,,,)) "7,,8'=11
max
min
a'EAA
LAROO mainly consists of two components: the noise model and the ensemble models, which together tackle heavy-tailed biases through two primary effects: (1) By selecting a lower Q-value from the ensemble models, LAROO explicitly re-centers the Q bias distribution, shifting its mean towards zero and mitigating the strong overestimation introduced by large Q biases. (2) Based on the Laplace-based noise model, LAROO alleviates the heavy-tailed biases and reduces their impact with robust function (I) on training stability, as discussed in Section 4.1
In summary, LAROO transforms the poorly-behaved, heavy-tailed Q-bias distribution into a more standardized form during Q-value updates. As result, LAROO effectively reduces the estimation variance and mitigates heavy-tailed Q biases, improving training stability and efficiency without the need for exploration constraints under distribution shifts. Additionally, we compare it with the standard batch normalization in Appendix B.7 and summarize the complete framework of LAROO in Al Algorithm] 1
4.3 THE THEORETICAL ANALYSIS
We further provide a theoretical analysis to prove that LAROO can reduce estimation bias compared with the l2 loss function for Q-value updates. We begin to introduce the estimate bias in single update step, following prior research (Duan et al. 2021). For simplicity, we denote the greedy target R(s a) + + [max.. Qi (8', "',a') as y. Conventionally, the Q-value network Qo is updated by minimizing the mean square loss (y 1y-Quc (s,s, a)) /2 using gradient descent methods, resulting in Qe..... Then, the post-update Q-value Qe.... (s, (s can be approximated by linearizing around Onew using Taylor's expansion, as shown in Equation 9 where the learning rate and is sufficiently small.
n 0 B(y Qo VoQo(ss)
Q0 a) = (s) + Bly - Qo (o(o.
Similarly, let 0id represent the ideal post-update parameter obtained based on true target ğ R(s, a) + >lE max. Q(s's ad)], and we can approximate the ideal post-update Q-value Qe (8,a) as:
Qu (s, a) = Qo(s.a) + 0ū - Qo(s, a)) ()) (ooo 1)
(10)
Then, in expectation, we can define the single-step estimation bias of the la update, denoted as:
(s, Eeg [Qo..n (8, a) Qa. a)) ((E,, 5) (V
where Eg is the approximation noise in target y. Moreover, previous research (Duan et al. 2021 Hasselt 2010) has verified that the estimate Q-value is usually overestimated due to the max operator and it is clear that: Ee, [v] ÿ 2 O, (s, a) 2 0.
Building on definitions above, we now demonstrate the robustness LAROO in Q-value estimation. When the Q-value network is updated with D2(x)) function, the following theorem holds:
Theorem 4,4 When b > 1, the single-step estimation bias of post-update Q-value Qrnen func- tion (x) is smaller than that with la loss function, i.e., D. A
7

---

Published as a conference paper at ICLR 2026
Table I: Normalized returns before and after the online fine-tuning, Each result is averaged with five seeds. 8aum 60.10. (0.1M) denotes the sum performance improvement on all tasks within IM steps. The best results are highlighted in bold, while the second-bes, results are underlined. The gray "offline" refers to the offline performance of both BOORL and LAROO, which use the same offline backbones, Please refer to Appendix C.2.2 for more results and training curves.
Task
Type
PEX
SO2
Cal-QL
ENOTO
Offine
BOORL
LAROO
random
6.6-117.5
12.9-887.3
7.3-46.9
7.4-666.5
7.9
88.3
100.6110.3
medium
49.1-77855
59.7-494.4
70.5-490.8
57.1-96.6
56.6
102.1
106.7:124
Hopper
medium-replay
60.0-87.55
98.9-+100.3
89.2-492.8
78.2-+102.7
80.5
105.1
109.5-1.8
medium--cppert
85,4-880.2
99.3-498.7
104.9-1108.3
94.5-1100.2
93.6
107.5
112.3=0.7
expert
65.7-47.88
86.7-484.7
106.0-+110.0
111.3-485.6
109.2
103.2
112.0::1.3
random
7.9-+37.0
4.7-420.8
9.7-+16
0.9-338.3
1.5
6.4
71,626.8
medium
60.5-+42.5
91.7-+100.6
83.3-48.77
83.6-110.22
83.6
98.6
120.4=2.9
Walker2d
medium-replay
32.8-+40.5
53.7-4920
75.1-+80.7
78.0-+110.2
76.8
104.6
124.8=2.00
medium-expeet
106.1-778.0
109.7>11108
106.0-:110.2
110.6-118..0
110.0
109.1
126.7=1.0
expert
102.1-990.8
93.2-+102.6
109.0-+110.1
110.0-120.5
111.2
110.5
128.9
random
15.8-+39.8
29.6-462.9
19.9-+12.2
9.8-82.4
10.4
90.6
87.4
medium
48.7-455.3
713--844
47.6-452.2
48.2-844
47.5
89.7
V2.5 1.2
Halfcheetah
medium-replay
44.5-152.1
66.1-476.2
45.8-+47.9
44.7-78.5
45.5
74.0
84.7+
medium--xppert
91.4-+90.3
90.6-85.6
51.3-+91.2
93.8-19.8
92.6
95.5
9.
expert
91.9-173.4
65.3-+100.8
85.8 85.8-193.3
97.4-89.3
96.4
93.5
101.1 13.1+33.8
6samm0.000 (0.1M)
355.4
550.4
Theorem 4.5 With Assumption 4. and 4.2 when >> the variance of post-update Q-value Qe with (x) function is smaller than that with l2 loss function.
The theorems above confirm that using D2(x) to update the Q-value, instead of the /2 function, reduces both the overestimated bias and estimation variance each update step. This leads to smaller accumulated bias and more robust Q-value estimation, theoretically reinforcing the effectiveness of LAROO.
We show the detailed proof in Appendix and B.5
5 EXPERIMENTS
We present experimental results to evaluate the effectiveness of LAROO. In Section 5.2 we demon- strate how LAROO significantly outperforms existing state-of-the-arr 020 methods across various tasks. Finally, we conduct ablation studies to further evaluate the effectiveness of LAROO.
5.1 EXPERIMENT SETTING
2020), which provides various continuous control tasks and training datasets. We choose MuJoCo and the more challenging sparse-reward environment, AntMaze, as our testing tasks. We compare LAROO with several state- of-the-art baselines in O20 RL, including PEX (Zhang et al. 2023), Cal-QU -L- Nakamooto et al., 2023), SO2 (Zhang et al. 2024), ENOTO (Zhao et al., 2024), BOORL (Hu et al. 2024). We also compare with the RLPD (Ball al. 2023). which effectively leverages offline datasets to train agents from scratch in online environments. We rerun these baselines with their official implementations. We Wtwpthubw..o//SSSSCAAAEEEEEEEEEE
Benchmark. We conduct our experiments on the D4RL. benchmark (Fu et al.
provide LAROO's ' codes
Experimental implement details. In our experiments, we first offline pretrain agents for one million gradient steps and then online fine-tune agents for 100K steps, this training setup is also adopted by (Feng et al.] 2024). For Mujoco tasks, LAROO adopts TD3BC (Fujimoto & Gu 2021 and TD3 (Fujimoto et al. 2018) with ensemble Q-functions as the backbone algorithms for the offline and online phases, respectively. Since TD3BC cannot handle with sparse-reward tasks, LAROO uses LAPO (Chen et al.] 2022) as the backbone for Antmaze tasks. These choices of backbone algorithms follow those in the ENOTO baseline (Zhao al. 2024). Please refer to Appendix C for more details and parameter settings.
5.2
MAIN RESULTS
The performance of LAROO. We evaluate both the final fine-tuning performance and the perfor- mance improvement dewm during online fine-tuning. Each experimental result is averaged over five
8

---

Published as a conference paper at ICLR 2026
walker2d medium
6025-
Laplac (liia Mo.6)
Maee 4.99
Nomalize Rotum
Normalized Rotum
120-
Kuraad
100-
ino-
0,020
so-
80
Density
6.15
60
40
60-
60:0
20
0005
ENOTO ENOTOMCI
Cal-OL Cal-OL
I03
10ss
20-
LAROD wio ensemble w/o hoise
hiis
Figure 3: (a) Normalized returns of existing 020 RL methods combining with LAROO on the walker2d tasks. (b) Ablation for components LAROO: the performance without ensemble models or without noise model respectively in the maalkerdd-edduu--rplla task. (c) Laplace-based noise model: we plot the Laplace-based noise against the empirical Q bias.
seeds. As shown in Table [ LAROO achieves superior final performance and notable performance improvement compared with other 020 RL methods, with an average score improvement of 4f4+5.8%% over the second-best method BOORL (Hu et al. 2024). In Figure [1] we present the online training curves of different methods Mujoco and Antmaze tasks. It is evident that LAROO consistently exhibits significant sample efficiency and training stability across various tasks.
Compatibility of LAROO with other 020 methods. To demonstrate the compatibility of LAROO with previous O20 RL methods for further performance improvement, we replace the l2 loss func- tion in previous methods with Do(x)) for Q-learning and evaluate the performance improvement. We select TD3, Cal-QL (Nakamoto et al., 2023) and ENOTO (Zhao et al., (202)) for evaluation, as these three methods exhibit varying levels of performance O20 RL. As shown in Figure (a) and
Table TO, the experimental results validate that our proposed robust loss function is highly effective and can serve as a plug-in in method to further enhance the performance existing methods.
Training stability of LAROO. We evaluate training stability in 020 RL based on two metries: (1) the variance, (2) the Normalized Cumulative Performance Drop (NCD), which quantifies the degree of performance degradation during training (Feng et al. 2024). Lower values of variance and NCD indicate better training stability. The results are shown in Table [ Specially, LAROO generally achieves a superior balance between performance improvement and training stability.
Empirical validation for Laplace-disrributed noise model. We plot the Laplace-distribtted noise against the empirical bias, shown in Figure 3e) and Figure TO. We note that the Laplace assumption better fits the empirical data than the Gaussian. We further discuss the advantages and limitations of using Laplace distribution to capture heavy-tailed nature of Q bias in Appendix B.
5.3 ABLATION STUDIES
Ablation for components of LAROO. We present distributions of Q bias without ensemble models without the noise model, respectively, in Figure 15 Ensemble models effectively reduce overes- timated Q bias, shifting its mean toward zero, but still suffers from high heavy- -tily-taisedness. The noise model alleviates heavy-tailedness of Q biases but remains prone overestimation. By combining these two components, LAROO promotes the heavy-tailed Q bias into a more standardized form.
We further compare the contribution of ensemble models with Laplace noise model LAROO. We assess their contributions by plotting training curves and computing related metrics. As shown in Figure [6] and Table [2] the results indicate that the Laplace noise model plays more critical role in LAROO. Please refer to Appendix C.4 for detailed ablation studies.
Ablation for UTD ratio and ensemble size. We conduct ablation experiments for the update fre- quency of data (UTD) and the ensemble size. A higher UTD has been widely used for improving sample efficiency, while the ensemble of Q function can help mitigate Q bias and improve train- ing stability in 020 RL (Zhao et al.) 2024). For a fair comparison, we set the UTD 1 and the ensemble size N = 1 in LAROO, then compare it with Cal-QL and PEX. Other baselines, which also leverage high UTD or ensemble models (i.e., BOORL, ENOTO and SO2), are excluded from the comparison. As shown in Table I0 LAROO outperforms Cal-QL in 10 out of 13 experiments, demonstrating its effectiveness even without the benefits of high UTD ensemble models.

---

Published as a conference paper at ICLR 2026
Ablation for the Q-value fescction We evaluate the performance with and without the proposed Q-value loss function Lo(x). Additionally, we compare Lo(z) with other commonly used robust loss functions, such as the Huber loss and the Cauchy loss functions, which prior studies have shown to effectively handle outliers in training data (Zahra et al. 2014). The results in Table | demonstrate that (x) outperforms these alternatives for better capturing these heavy-tailed Q biases.
RELATED WORK
Offine-to-online RL. Offline-to-Online RL primarily faces the challenge of significant distribu- tion shift between offline and online data during the fine-tuning process. This shift leads to inaccu- rate Q-value
overestimated-ffo out-of-distribution online data, which results in performance degradation during fine-tuning (Lee et al.) 2021b; Zhang et al. 2024 Zhou et al., 2025]. Various strategies have been proposed to improve training stability. Specifically, some meth- ods retain constraints from offline stage or introduce additional conservative techniques to ensure training stability. For example, Off2OnRL (Lee et al.) 2021b) leverages a balanced replay buffer and utilizes pessimistic Q-ensemble networks. Cal-QI. NNakamoto et al. (2023) employs calibrated Q-lcarning to learn a conservative Q-value function. However, Feng et al. (2024) argue that these constrained methods improve training stability at the cost of sufficient performance improvement. FineTTuneRL (Wagenmaker & Pacchiano) 2023) proposes the actor-critic alignment step to bridge the gap between online and offline Q-value learning. mitigating performance drop. Some stud- ies (Feng et al.] 2024] Zhang et al. (224)) increase the update frequency of Q-value networks to enhance Q-value estimation. ENOTO (Zhao al. 2024) uses Q-ensemble models reduce the Q-value estimation bias. BOORL (Hu et al. 2024) introduces Bayesian approach to guide agent's 's online fine-tuning.
modeling of Q-values. Distributional reinforcement learning aims to model and predict the full distribution of returns, for more robust and risk-sensitive maaking (Bellemare) et al. 2017 2023). Duan et al. (2021) propose the distributional soft actor-critic (DSAC) algorithm, which learns a Gaussian distribution function of state-action returns (i.e., Q Q-values) effectively mitigate Q-value overestimation. However, these methods may face incompatibility issues in O20 RL with offline pretrained models, which usually estimate Q- Q-values in expectation rather than the full distribution (Fujimoto et al.) 2018, Kumar et al., 2020). In this paper, model the heavy- Q bias with Laplace distribution instead of directly modeling the Q-values, thereby capturing the heavy-tailedd characteristics of bias while maintaining expected estimates of Q-values.
Distributional
Extreme Q-learning. Garg et al. (0023) apply the Gumbel distribution to estimate the maximum Q-value. This work builds on the extreme value theorem which states that the maximal values drawn from any expomential-tailed distribution follow a Gumbel distribution (Fisher & Tippett 1928). However, the training objective of extreme Q-learning differs from that of other offline algorithms, which could disrupt well-pretrained Q-value networks during online fine-tuning.
7 CONCLUSION
In this paper, we focus on the Q-value estimation in online data for 020 RL. Through extensive experiments, we reveal for the first time that the Q bias follows heavy-tailed distribution in 020 RL, a phenomenon unexplored in previous research. These heavy-tailed Q biases lead to instability and inefficiencies performance improvement. To mitigate their influence, we propose LAROO for robust Q-value estimation. LAROO captures the heavy-tailed property with Laplace-distributed noise models, and mitigates the overestimation with conservative ensemble-based estimates. LA- ROO alleviates the heavy-tailednnes of Q bias and promotes it into standardized form, improving training stability and performance. Future work could explore more robust techniques to mitigate the heavy-tailed Q bias. We look forward to the continued development of robust 020 RL meth- ods that improve online fine-tuning for pretrained agents. These advancements could significantly facilitate the deployment of well pretrained agents in various real world scenarios.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors would like to thank all the anonymous reviewers for their valuable suggestions. This research is supported by Anhui Provincial Natural Science Foundation (Grant No. 200.28888)))))4 the Fundamental Research Funds for the Central Universities (Grant No. W,220000000000) and the Smart-Grid National Science and Technology Major Project (Grant No. 20252D0805500).
ETHICS STATEMENT
Offline-to-online reinforcement learning methods hold significant promise for facilitating the de- ployment of well-pretrained agents real-world scenarios. Our study shows that existing ap- proaches continue to suffer from training instability and inefficiencies due to heavy-tailed Q-value bias. We propose strategies that effectively mitigate the influence of this bias, leading to more robust Q-value estimation. We believe research does not contain potentially harmful insights and any discrimination issues.
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-basee offline rein- forcement learning with diversified q-ensemble. In Advances in Neural Information Processing Systems 34, pp. 7436-7447, 2021.
Jushan Bai. Least absolute deviation estimation of shift. Econometric Theory, 11(3):403 436, 1995.
Philip J. Ball, Laura M. Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 1577-1594, 2023.
Vasileios Belagiannis, Christian Rupprecht, Gustavo Carneiro, and Nassir Navab. Robust optimiza- tion for deep regression. In Proceedings the pp. 2830-2838, 2015.
the IEEE international conference on computer vision,
Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International conference on machine learning, pp. 449-458. PMLR, 2017.
Mare G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT Press, 2023.
Jacopo Cavazza and Vittorio Murino. Active regression with adaptive huber loss. 0bss/660.01555 015601 2016.
CoRR,
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. LAPO: latent-variable -eeightge-weiggheed policy optimiza- tion for offline reinforcement learning. In Advances in Neural Information Processing Systems 35, 2022.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without model. In 9th International Conference on Learning Represen- tations. OpenReview.net, 2021.
Carlo D'Eramo, Andrea Cini, Alessandro Nuara, Matteo Pirotta, Cesare Alippi, Jan Peters, and Marcello Restelli. Gaussian approximation for bias reduction in q-learning. J. Mach. Learn, Res., 22:277:1-277:51, 2021a.
Carlo D'Eramo, Andrea Cini, Alessandro Nuara, Matteo Pirotta, Cesare Alippi, Jan Peters, and Marcello Restelli. Gaussian approximation for bias reduction in q-learning. Journal of Machine Learning Research, 22(277):1-51, 2021b.
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng, Distributional soft actor-critic: Off Off-policy reinforcement learning for addressing value estimation errors. JEEE transactions on neural networks and learning systems, 33(11):6584 6598, 2021.
11

---

Published as a conference paper at ICLR 2026
S. EI Adlouni, B. Bobée, and T.B.M.. Ouarda. On the tails of extreme event distributions in hydrol- ogy. Journal of Hydrology, (55(1):16-33) 16-3, 2008. ISSN 0022-1694, doi: https://oi..rr//Iu 1016/j. 0hyydrol.000.022.11
Paul Embrechts, Claudia Klüppelberg, and Thomas Mikosch. Modelling extremal events: for insur- ance and finance, volume 33. Springer Science & Business Media, 2013.
Jiaheng Feng, Mingxiao Feng, Haolin Song, Wengang Zhou, and Houqiang Li. Suf: Stabilized unconstrained fine-tuning for offline-to-online reinforcement learning. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp. 1991-1199999 19999, 2024.
Ronald Aylmer Fisher and Leonard Henry Caleb Tippett. Limiting forms of the frequency distribu- tion of the largest or smallest member a sample. In Mathematical proceedings of the Cambridge philosophical society, volume 24, pp. 180-190. Cambridge University Press, 1928.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL:: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems 34, pp. 20132-20145, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations. OpenReview.net, 2023.
Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Adarsh Prasad, J. Zico Kolter, Zachary C. Lipton, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Pradeep Ravikumar. On proximal policy optimization's 's heavy-tailed gradients. In Proceedings of the 38th International Conference on Machine Learning. 2021.
Andrew Gelman, John B Carlin, Hal S Stern, and Donald B Rubin. Bayesian data analysis. Chap- man and Hall/CRC, 1995.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
Thomas Hettmansperger and Joseph W McKean. Robust nonparametric statistical methods. CRC press, 2010.
Hao Hu, Yiqin Yang, Jianing Ye, Chengjie Wu, Ziqing Mai, Yujing Hu, Tangjie Lv, Changjic Fan, Qianchuan Zhao, and Chongjie Zhang. Bayesian design principles for offline-to-online reinforce- ment learning. In Forty-first International Conference on Machine Learning, 2024.
Jiayi Huang, Han Zhong, Liwei Wang, and Lin Yang. Tackling heavy-tailed rewards in reinforce- ment learning with function approximation: Minimax optimal and instance-dependent regret bounds. In Advances in Neural Information Processing Systems 36, 2023.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? Pro- ceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5084-5096 PMLR, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In The Tenth International Conference on Learning Representations. OpenReview.net, 2022.
12

---

Published as a conference paper at ICLR 2026
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1199 2020.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified frame- work for ensemble learning in deep reinforcement learning. In International conference on ma- chine learning, pp. 6131-6141. PMLR, 2021a.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-toonllnee reinforcement learning via balanced replay and pessimistic g-ensemble. In Conference on Robot Learning. Proceedings of Machine Learning Research. PMLR, 2021b.
Kang-Ping Lu and Shao-Ture Chang. Robust switching regressions using the laplace distribution. Mathematics, 10(24):4722, 2022.
Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochas- tically larger than the other. The annals of mathematical statistics, pp. 50-60, 1947.
Kanti V Mardia. Measures of multivariate skewness and kurtosis with applications. Biometrika, 57 (3):519-330, 1970.
Gregory P. Meyer. An alternative probabilistic interpretation of the huber loss. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 5261-5269 Computer Vision Foundation / IEEE, 2021.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.09358 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-qi: Calibrated offline RL pre-training for efficient online fine- tuning. In Advances in Neural Information Processing Systems 36, 2023.
Donald T Searls and Pichai Intarapanich. A note estimator for the variance that utilizes the kurtosis. The American Statistician, 44(4):295-29) 296, 1990.
Han Shao, Xiaotian Yu, Irwin King, and Michael R. Lyu. Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs. In Advances in Neural Information Processing Sys- tems 31, pp. 8430-8439, 2018
Weixing Song, Weixin Yao, and Yanru Xing. Robust mixture regression model fitting by laplace distribution. Comput. Stat. Data Anal., 71:128-137, 2014.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 2012.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforeement learning library. Advances in Neural Infor- mation Processing Systems, 36:30997-31020, 2023.
Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data online reinforcement learn- ing. In International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 35300-35338. PMLR, 2023.
Eshetu Wencheko and Honest W Chipoyera. Estimation of the variance when kurtosis is known. Statistical Papers, 50:455-464 464, 2009.
Thomas Werner and Christian Upper. Time variation in the tail behaviour of bunds futures returns. Available at SSRN 362320, 2002.
Makoto Yamada, Taiji Suzuki. Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Rel- ative density-ratio estimation for robust distribution comparison. Neural Computation, 25(5): 1324-1370, 2013. doi:
00442.
13

---

Published as a conference paper at ICLR 2026
Rui Yang. Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption. In The Twelfth International Conference on Learning Representations. OpenReview.net, 2024.
Tao Yang, Colin M Gallagher, and Christopher S McMahan. m-estimation. Communications in Statistics-Theory and Methods, 48(5):1092-1107, 2019.
robust regression methodology via
Ke-Hai Yuan, Peter M. Bentler, and Wei Zhang. The effect of skewness and kurtosis on mean and covariance structure analysis: The univariate case and its multivariate implication. Sociological Methods & Research, 34(2):240-258, 2005.
Mohamed M Zahra, Mohamed H Essai, and Ali R Abd Ellah. Performance functions alternatives of mse for neural networks learning. International Journal of Engineering Research & Technology, 3(1):967- 9700 2014.
Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforce- ment learning. In The Eleventh International Conference on Learning Representations, 2023,
Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang. Yu Liu, and Wanli Ouyang. A perspective q-value estimation on offline-to-onlin reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 16908-16916, 2024.
Kai Zhao, Jianye Hao, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. ENOTO: improving offline-to-online reinforcement learning with q-ensembles. In Proceedings of the Thirly-Thirl
International Joins Conference Artificial Intelligence, pp. 5563-5571, ijeai.org. 2024.
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline-to Celine-to-online reinforcement learning. In 30th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. 2022.
Weitao Zhou, Bo Zhang, Zhong Cao, Xiang Li, Qian Cheng, Chunyang Liu, Yaqin Zhang, and Diange Yang. Drarl: disengaggement-resso-- reason-augmentted reinforcement learning for efficient im- provement autonomous driving policy. In 2025 JEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pp. 604-611. 1. IEEE, 2025.
Vincent Zhuang and Yanan Sui. No-regret reinforcement learning with heavy-tailed rewards. In The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 3385-3393, PMLR, 2021.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A
THE Q BIAS IN OFFLINE-TO-NNLINE RL
A. THE HEAVY-TAILED DISTRIBUTION
There exist several mathematical definitions of heavy-tailed distribution. These definitions char- acterize "heavy-tailed" respectively based on (i) the convergence rate of the tail (Embrechts et al.) 2013) and (ii) the moment properties of the distribution function (El Adlouni et al. 2008 Werner & Upper 2002). We adopt the following definitions:
Definition A.1 (Embrechts et al. 2013) A non-negative random variable is called heavy-tailed if its tail probability (t) == P(w 2 is asymptotically equivalent to t-a" as t - oo for some positivee number a². Here, x² determines the heavy-tuieddnes and o² called tail index of w. lower tail index indicates heavier tails.
In practice, widely used comentt-based criterion defines heavy tails as follows.
Definition A.2 (El Adlouni et al. 2008 Werner & Upper 2002) X is a random variable, and OX are the mean and the standard deviation of X, then X is called heavy-tailed if
pxx x
>
This criterion (kurtosis > 3) serves as a well-cstablished measure for hcavy-tailednees. this study, explore two estimators as heuristic measures to understand heavy tails and non-Gaussianity of Q bias.
The Kurtosis. We will compute the Kurtosis values of Q bias distribution Definition A.2
The log-log plot. A log-log plot directly visualizes the distribution's tail-index for diagnosing the heavy-tailed behavior. The utility of this plot stems from its ability to transform power- law relationship, defined by a probability density function p(x) & x= into a linear form: log(p(æ)) -a log(æ) (z) + C. Consequently, the slope of the linear segment in the plot's tail gion serves as a direct graphical proxy for the negative tail-index -a, as discussed in Definition A.I
A.2 THE HEAVY-TAILED Q BIAS
In this section, we present the heavy-tailed distribution of Q bias in online samples when fine-tuning pretrained agents with different algorithms on Mujoco tasks. We calculate the kurtosis value to quantify the heavy-tailedness of Q bias. It should be noted that the Q bias is defined as [s(a(s, a)] Q(s, a) in our paper.
We first present the Q bias distribution when fine-tuning with offline algorithms. Specifically, employ the same algorithm in offline pre-training and online fine-tuning stage. The results show that the distribution of Q bias has longer tail than the normal distribution, with notably high biases in the tail.
CUL
EDAC
TD3BC
IQL
1150-
20.0
17.5
12.5-
Kunosis
Moar: 55.01
Vallase: 56535.82
Paspentagg (N6)
15
Kurtosii 5.49
Moun: -126.02
Varianse: 12115.33
Potcestage (5))
20
15
Meun: -11.19
Kemsoa:
Vatiane:: 11545.65
Porcostage (5)
16
14
Mear: -88.7
Kontsci 8.70
Vatiano:: 10014.06
10.0-
7.5-
400-220
200 400 600 500 1000 hits
-300
110
160
0
900
-2101
200
Q
Figure 4: The heavy-tailed Q bias distribution offline RL methods. We present Q bias distributions of four offline RL methods (CQL (Kumar al. 2020), EDAC (An al. 202T), TD3BC (Fujimoto & Gu (02T), IQL (Kostrikov et al. 2022)) for online fine-tuning at the 100k online step in the Walker2d-medium task.
15

---

Published as a conference paper at ICLR 2026
Next, illustrate the distribution of Q bias when fine-tuning with different 020 RL algorithms. We pretrain agents using the corresponding offline backbone algorithms and further fine-tune them online with 020 RL algorithms. Recent work such ENOTO (Zhao et al. 2024), which applies a Q-ensemble strategy to reduce bias, and SO2 (Zhang et al. 2024), which increases the update frequency improve Q-value estimation, still exhibit Q bias distributions with heavy-tailed char- acteristics. In Figure [ and Figure 5 we illustrate the Q bias distribution at 50k, 100k and 150k steps in the Walker2d task, respectively. In Figure we test the distribution of Q bias in more tasks, including hopper and halfcheetah.
For each distribution of Q bias, we also calculate its mean and variance. A positive mean indicates the Q-value overestimation while negative mean implies Q-value underestimation, and low vari- ance benefits stable performance improvement (Chen al.) (02T). We observe that the Q-values tend to be underestimated with offline RI. methods and overestimated with several O20 RL meth- ods during fine-tuning process. Furthermore, they struggle to decrease its high variance, indicating ongoing instability during training.
Based on extensive experiments above, for the first time, we empirically reveal that the Q bias follows heavy-tailed distribution in online data during online fine-tuning. This finding inspires us to identify the influence of heavy-tailed Q bias in 020 RL, and further design our methods to improve fine-tuning performance and stability.
Cal-U welledd eoperiv2
espanig?
502,
502 waw.erd waw.rri-dddmmmoorrraaa 1itk
Pateuruue (5))
35-
Moun: 65.05
Valanao: 11335.49
Percentzee (5))
35
Meas 41.0
Melanve: 12218.5
Poscentage: (N))
25
Kartais 9,17
Meus: 12.78
13534.40 Kunoik 9.14
Moze iI up
12
Percrmaaee
100
260
330
400
500
min
1200 100 2ie 30o de 530
T00 105 335
300 an 500 m0 bis
1200
160
300
300
400 500
Q bias
bas
Q bias
ENDTO wakeraa-metarm-eee opertal 20k
END10 wakertd medumeepooo 1tkk
11.5
Mtaee:
102.24
18.44 Vartano 10144
15.0
Veriice 161271
Nirtance: 5264.5 Kankoii 9,69
Karsoas 9.30
Kanaiis 8.52
Poremtng i
Kartesas 9.11
12.0
10-
Pesenstaes
1.5
5.0
2.5
500
bias
bii.
bias
Figure 5: The heavy-tailed distribution of Q bias wilder choices of online steps with four 020 RL methods (S02 (Zhang et al. 2024), Cal-QL (Nakamoto et al. 2023). ENOTO (Zhao et al. 2024).
PEX (Zhang al. ()
Di-gt baltharabemttimdd Yo0k
16
PEX ballchenah-mriam--
PEX kurpen--etiimm- look
Percontag (N))
1
Meas: 6750,40
Kanneic 4.50
Pececatage
15.
20
Moss
Korsoc K
Variaaz $725.50 50
Percantnge
14
Kunnee: 7.6
23602.79
Peseutoge
12.5
3000
1500
Kartoee
Moar $9,30
10.
7.5
50
2.5-
200
150
230
-150
200
300
402
400
600
biss
binx
hisss
hiss
Figure 6: The heavy-tailed distribution of Q bias wilder tasks.
We illustrate the Log-log plot of Q bias to further validate its heavy-tailedness. As shown in Figure 7 the Q biases in the tail are roughly arranged in a straight line, indicating that the Q bias may follow heavy-tailed distribution. We also compute their tail index.
A.3 THE REASONS CAUSING HEAVY-TAILED Q BIAS
We discuss the underlying reasons causing the heavy-tailed Q bias in this section. In offline- to-online reinforcement learning, a heavy-tailed distribution of Q-value estimation bias (Q bias) is primarily caused by the distributional shift between offline and online data. The Q-function, pre- trained on a static offline dataset, must extrapolate values for out-of-distribution (OOD) samples encountered during online interaction. The heavy-tailed nature of the bias stems from twe rea-
16

---

Published as a conference paper at ICLR 2026
CX-01
PUX
TIOIK:
101
Legp Lud C1.28 211
New 2010
10"
Weider
Alccknn Ros
Alwas Ri
Figure 7: The Log-log plot of Q bias. The blue markers represent the cumulative distribution function (CCDF) for the absolute Q bias. The red dashed line shows a power-law fit applied to the tail of the distribution (the largest 15% of values). These experiments are conduct in the walker2d- medium-expert tasks.
sons: (I) the non-uniformity of this shift: online samples highly dissimilar to the offline data induce large extrapolation errors, which manifest as the extreme values in the distribution's tails. Conversely, samples with smaller shifts result in minor biases, forming the central mass of the dis- tribution. (2) the Q-function with Max operator: Q-learning, Q-value targets are bootstrapped and include maximization step, y + max, Qa 6', d'), the max operator induces systematic selection of overestimated actions (the "winner's s curse"), ) Rare samples with large positive noise are preferentially chosen and then propagated through bootstrapping. which amplifies extremal overes- timation (Fujimoto et al.) 2018 E'amo et al.] 2021a). Thus, (1) and (2) cause the overestimated Q bias yielding heavy right tail in its distribution.
Guided by the analysis above, our methods mainly focus on the latter cause of heavy-tailed Q bias, since the non-uniformity of data shift is hard to control during online exploration. Our method cap- tures the heavy-tailed Q biases with parameterized noise models, and provides theoretical guaran- tee of reduced post-update bias even with Max operator, as shown in Method 4
We conduct experiments to investigate whether the heavy-tailed Q bias partly arises from the non-uniformity of distribution shift. Specifically, we first measure the distance between the online data and the offline dataset, and then compute Spearman's 's rank correlation coefficient to quantify the relationship between the Q bias and the measured distance.
To assess the distance between the online data and the offline dataset, we employ two complementary metries: (1) the k-nearest neighbor distance DANN; (2) the classifier- based density ratio DpR- The DANN calculates the average Euclidean distance to the k nearest offline samples in the state-action space. This metric captures the geometric similarity between the offline and online data.
On the other hand, the classifier-based density ratio DDR is widely adopted to quantify the discrep- ancy between the two distributions (i.e., online and offline data) (Sugiyama et al.] 2012) Yamada (s, a), Por(2) and (2)(2) denote the offline and online state-action distribu- 2) E (0, 1) to distinguish online (label 1) from offline (label 0) by bass-balanced sample. The Bayes-optimal discriminator satisfies
cet al. 2013)). Let tions. We train a discriminator hss (z) minimizing the logistic loss
(2)
(2) (2)
Pon Pon(2) Poff (2)
h (z)
g (z) log
log
(=) + Por(2)
so the logit go (2) of a well-trained classifier is a consistent estimator of the log density ratio (=n log P=)(2) This makes go(2) principled, scalar per-sample distance: larger DDn indicates that online data lies farther away from offline support.
In summary, DANN provides a simple, geometric distance, while DDR offers a distribution-awan measure that reflects how likely sample is to belong to the online versus offline distribution. Taken together, these metrics give a comprehensive characterization of how online samples shift relative to the offline distribution.
Finally, we validate that the non-uniform of distribution shift could contribute to the heavy- tailed Q bias. We present seatter plots illustrating the relationship between Q-bias and two distance metrics. To quantify this relationship, we compute Spearman's 's rank correlation coefficient, which measures the strength of the monotonic association between Q-bias and the distance. As shown in
17

---

Published as a conference paper at ICLR 2026
hopper medium expert-v2
hopper medium expert-v2
120
100
100
Dessiyy
Point Dansty
Q-asa
80
hins
40
20
Speatras 05B
20
Honde Monn Q-kia
(Logit Distance)
Dew (KNN Distunce)
Walker md-.mdium-cccpetttee
Q-bias
100
Qubins
100
Point
Point
Speleppo Tua0
Biaace Xeas Aej
Bizaei Keak Quiin
100
0100
2.5
5.0
100
Dos (Logit Distance)
Danu (KNN Dismnne)
Figure 8: Scatter plots of Q bias and the two distance metrics Dann DpR- We compute the binned mean of Q bias with 95% confidence interval, and spearman correlation coefficient.
Figure there is a clear positive correlation: the Q-bias for state-action pairs tends to increase as their distance from the offline data distribution grows.
A.4 THE INFLUENCE OF HEAVY-TAILED Q BIAS
In this subsection, we validate the influence of the heavy-tailed Q bias in Q-value estimation.
In robust regression research (Song et al. [0144 Huang al. 2023), they have identified that the heavy-tailed сπтогs can lead to instability in the coefficient estimation, particularly when the errors contain significant number of extreme values.
We conduct a simple validation experiment to verify the oscillation in Q-value estimation induced by heavy-tailes Q bias. For approximating the heavy-tailed Q bias, we introduce heavy-tailed noise into the estimated Q-values during the update of the Q-value network. Specifically, we train two agents online based on TD3 (Fujimoto et al., (018)) with ensemble Q-functions. We introduce heavy-tailed Cauchy noise with a mean of O and standard deviation of 10 in one agent, and introduce Gaussian noise with the same mean and standard deviation the other for a fair comparison.
As shown in Figure 9 the results demonstrate that, under heavy-tailed Cauchy noise, the agent exhibits higher and continuously increasing Q bias compared to the agent trained with Gaussian noise. This suggests greater inaccuracy and oscillation in Q-value estimation. Furthermore, the final performance much lower, indicating an inefficient performance improvement due to heavy- biases.
bias
Performance
Cauchy Noise
150
Gaussian Noise
Normalized Return
120
100
Value
100
80-
Cauchy Noise
40
Gaussian Noise
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Timesteps
Le5
Timesteps
le5
Figure 9: Validation of the instability caused by heavy-tailed Q biases. This experiment is conducted in the Walker2d-medium expert task.
18

---

Published as a conference paper at ICLR 2026
B THE SUPPLEMENTARY FOR METHOD
B.1 LAPLACE MODELING FOR Q BIAS
In this section, we analyze the feasibility and rationality of using Laplace distribution to cap- ture the heavy-tailed nature of Q bias in 020 RL. In robust regression research (Song et al. 2014 Huang et al. 2023), the Laplace distribution offers a theoretical foundation for modeling data with many outhers and long tails in robust regression. Specifically, under Laplace errors, the maximum likelihood estimator (MIE) is equivalent to the robust least absolute deviation (LAD) estimator (Bai, 1995), which effectively reduces the influence of non-normal errors by constraining update gradients.
In this paper, we introduce a parameterized Laplace-distributed noise to capture the heavy-tailed Q bias. First, we plot the Laplace distribution against the empirical Q bias in Fig 10 Though the Laplace distribution are significantly better than Gaussian distribution in fitting Q bias, the limita- tions are clear. The Laplace is not heavy-tailed distribution and symmetric, leaving gap to fit Q bias. In practice, it is challenging to identify a distribution that perfectly models Q bias across different baselines. For instance, the distribution of Q bias in ENOTO (Zhao et al. 2024) is highly asymmetric with heavy right tail as shown in Figure I making it difficult to fit.
In this paper, taken into account both the theoretical foundation and practical feasibility, the Laplace distribution for modeling Q bias have the following advantages:
Laplace distribution can model data with heavier tails and outliers compared to the Gaus- sian distribution, and it has solid theoretical foundation in robust regression analysis.
The Kullback-Leiin divergence between Laplace distributions (i.e., Q bias distributions) facilitates the derivation of Equation 6 for Q-value updates. However, for most other com- plex distributions (e-g., generalized Gaussian distribution), obtaining an analytical solution for the KL divergence is infeasible.
laiftecbssedaat-d
L.00
6.04
2015-
811C
8019
AX0S
Q kma
Qb
02 1/:0
bia
Figure 10: The Laplace-based noise model of LAROO. We plot the Laplace distribution against the empirical Q bias. The Laplace assumption better fits the Q bias than the Gaussian.
B.2 DERIVATION OF THE LOSS FUNCTION
We provide the derivation of Equation 5 First, the Kullback- (KL) divergence between a probability distribution q(x) and a reference distribution p(x) is
Dkl(p(=)la(z)) = ((((x).(())) - HH((x))
p(a) log (())dxx+++ / p(x) log p(x)da
(11)
where (p(x), g(x)) is the cross entropy between p(æ) and g(x) and H (p(x)) is the entropy of p(x). If p(x)) follows a Laplace distribution, i.e., p(x) ~ Laplace(44 b). where is the mean parameter and b is the scale parameter, μ R, b Rt. thus the distribution p(z) becomes:
1
[x
14
p(x)
exp
(12)
2h
Next, we derive the KL divergence between two Laplace distributions according to the following Lemma.
19

---

Published as a conference paper at ICLR 2026
Lemma B.1 Both p(z) and q(x) the Laplace distributions, i.e., p(x) Laplace(E1, b1). (x) Laplace(pz, b2), the KL. divergence between p(a) and ((x) is
exp
l lim + +1m
u2l
DRc(o(e)
+ logl
(13)
b2
b
Specifically, in our case, we set 2(s;,))) iI TQo(s; + and مع Qo(s;,a)) p. Thus, we can derive Equation (5)
We clarify the Equation with Bayesian principles for easy understanding. When using non- informative priors, the posterior distribution in Bayesian inference is numerically equivalent to the likelihood function in frequentist statistics (Gelman et al. 1995). In ours, the prior p( Q(s, a)) can be non-informative because the true Q-value always unknown. Therefore, the likelihood of true Q- value pl ((s, a) Qo a)) is also the posterior given the bellman target. The p( Q(s, a) Qo(s, a)) the posterior given the estimated Q- Q-values. Thus, minimizing the KL divergence encourages the posterior p( ((s.a) a) Qu(s, a)) to approximate the posterior p( ( a) TQa(ss,). a)).
environmental return information (i.e., future rewards), it serves as a closer approximation to the Bayes-optimal posterior (i.e., p(2 all environmental evidence)) ). compared with the posterior p( p(2(s, a) (s(saa)).
Since the posterior a)) TQo(s,a)) incorporates
We provide a brief analysis of the advantage of the Equation the robust 1 loss function. The likelihood (s, a) TQa(s, a)) explicitly captures the distributional information of heavy- tailed bias in bellman target with Laplace modelings for Q updates. Instead, if the heavy-tailed Q bias in bellman target is ignored, i.e., the p( ((((s. a) TQo(s, a)) Dirac delta function centered on (s, a). The KL divergence becomes:
pQ((s a) TQu(s, a)) log (Q a)los ())oo( a)) = - log (TQ)(((()))))l))) a))
(14)
Thus, LAROO reduces to the negative log likelihood, which is exactly equivalent to applying L1 loss under a Laplace-noise assumption for TD-errors. Therefore, using 4 or Huber loss corresponds applying a fixed robust regression objective TD errors, which does not explicitly model the heavy-tailed Q bias in bellman target and may discard distributional information of Q bias.
In summary, under heavy-tailed Q bias, for both estimated Q-values and bellman targets, and derive the Equation updates. In contrast, directly using 4 loss for updates may lose the distributional information about heavy-tailed bias.
natural and reasonable choice model the Q-bias
B.3 PROOF OF DIFFERENTIABILITY
In this section, we will derive the first and second derivatives of the loss function Ds(x) and prove their existence at x = O. We follow the proof process in prior work (Meyer) 2021). The first derivative of Dz(z) is shown as Equation 15). sgn(x) the sign function. We can get that the
gradient of Dx(r) is bounded within the range
1
6
exp +)
I >
0
Dx(()
exp
I <<
(15)
(x) (1 exp
(-1-2
And its second derivative is:
exp #
(x)
D exp
T 2
exp
| b
(16)
62
0
Next, to prove the derivatives of D,, and (z)x) at x = 0, we should show that both D3 and D (x())
differentiable at 0. The function (x) is considered differentiable at æ if the limit
20

---

Published as a conference paper at ICLR 2026
exists. The derivative of any function fz) at is defined follows, and the function f(x) is considered differentiable at x = if the limit exists.
[(x) - f(a) x-
f'(a)
lim
(17)
x a
For the function Dz)x), the derivative of Da(x)) at = 0 is defined as:
bexp
[x] + |
b
b
D((0)
lim z-0
br
(18)
b
exp
lim
lim
2-+0
bx
bz
The right limit of (,) is:
1
lim
D2(0)) lim
(- + + =0
(19)
x-0 x-10
The left limit of (0) is:
1 + o(x))
lim Db(0)= lim
z-+0- [5 --
(20)
x-0
Both the left limit and right limit of D,(0) become zero, which means the limit exists and proves Do(x) differentiable at a = O.
For the function (zx, the derivative of D/ is defined as:
sgn(x) (1 - exp (-1²1
DC lim
(21)
hr
The right limit is the following:
exp( (xp((=t)
lim
lim
+ +xx))
(22)
bæ
$2
Similarly, the left limit the following:
1 cxxp()) bor
lim
lim
+ o(x))
(23)
82
Therefore, both the left and the right limits f which implies the limit exists and D/(x;) is differentiable at x = 0.
Finally, we illustrate the function D& (z:) and its first derivative Df(2)) as follows.
Functions
Derivatives
(x(
Dx(x)
3-
L1 loss
LI'
Huber loss
Huber'
-2
-2
2
Figure 11: The curves of D2(x) and its first derivative (x) when b =
21

---

Published as a conference paper at ICLR 2026
B.4 PROOF OF THEOREM
We first provide the prerequisite knowledge for the theorem. Due to page limitations in the main text, we repeat and present a more detailed definition here supplement, following the analysis in DSAC (Duan et al., 2021). which demonstrates the effectiveness of distributional modeling of Q-values by analyzing the reduced estimate bias per training step.
We denote the greedy target R(s, a) + +T-1) [max Qi (s', aa)] as y. Conventionally, the Q- value network Qo is updated by minimizing the mean square loss (y - Qo(s, /2 using gradient descent methods, i.e.,
nnew 0 + B1 - Qo(s, a)) (oQ
(24)
where B is the learning rate and is sufficiently small. Let Osd represents the ideal post-update param- eter obtained based on true target ÿ = R(s, a) + TE, [max, 2(s',a') that is,
02 - B (s( VoQooss
(25)
Since 3 is small enough, the post-epdate Q-value Qe (s, a) can be approximated by linearizing around 9 using Taylor's expansion. Similarly, which represents the most ideal and correct Q value that can be achieved currently in one update. We can approximate:
get the ideal post-update Q-value Qe (s, a),
(s..) = Qo(s, (0 - (o(s.a))
(26)
Qo (8, Qo(s,a) B(y - Qo(s,a))lVoQ(((:0) a)
Then, in expectation, we can define the single-step estimation bias of the l2 udpate, denoted as:
(s, Ezg [Qe... (s, a) Q8.a a)] = (v) Vo a)
where Eg is the Q-value bias in target y. (1, a) represents the estimation bias introduced by the inaccurate target y in one step with lg loss function. Moreover, previous research (Duan al.)
2021 Hasselt 2010) has verified that the Q-value is overestimated due to the max operator and it is clear that:
Ee, [v]
5 2 0,
(s, a) 2 0
(27)
Proof of Theorem 4.4 Building the above analyses, we now repeat the Theorem 4.4 and demon-
strate its proof:
Theorem B.2 When b 1, the single-step estimation bias of post-update Q-value Q0 with function Dz(x) is smaller than that with l2 loss function, i.e., AD. - Ai2 < O.
Proof. When we use Dolx) to update Q-values, the post-update Q-value Qenen (s, can be approx- imated as:
Quals; a) x Qo(s, a) + 3D Qo(s, (o6.. Qu a) ze Qo(s, + 3D - Qo (o a))
(28)
The estimate bias of Qoneo (s, with Dz(x) function in single-step update is: AD. (s, a) = Ezg Qu (s, Qo,(s.a)
BB Dí(yy - Qo(s,a) D' - D0-(0-00000.... (o(ssa)))ll (0(ssa))
and the estimate bias of Qa (8, a) with 12 function in singe-step update is:
(i, = Eeg QQnne (s, Qo (s. a)) x ((E,, [vl - Vo
We define = y - Qu(s, a), 5j - Qo(s, a) and f(x) = D1((z) r. With these definitions, we have:
Ap. - D1, = -BEe, Di(()) - (2) - (= [o
(29)
=Be [f()) f(3) (oQo(s.
For Ee, [y] - > O, we have B.yla] [2] & > 0.
Meanwhile, when b > 1, the f((x) = Dy - 1 = 1/2 expl 144) - < O, which implies f(x) is monotonically decreasing, So we have:
D, DI <0
(30)
22

---

Published as a conference paper at ICLR 2026
B.5 PROOF OF VARIANCE REDUCTION WITH LAPLACE MODELING
In this subsection, we provide additional theoretical analysis of Laplace noise modeling, with a focus on its variance reduction effect. We first introduce the necessary preliminaries in Lemma B.3 and then present the main variance reduction result together with its proof in Theorem B.4
Theorem B.4 can be understood directly, even you choose to skip the derivation of Lemma B3
Lemma B.3 Assume three random variables X Y and Z. Y is independent from X and Z. X and Z are not independent, i.e., Cov(X,2) 2) # O. Then, the variance of X + Y is:
Var(X (orx+V22 = VVa(() + Varl)) + Var(Z) + 2E[Y]Cov(X,, Z)
(31)
Proof. We have the following well-known relationship of two independent variables Y and Z: Var(Y 7) = Var(Y) Var( + (ar(Y) + (er(2)
(32)
Var(Y + Z) Var(Y) Var(Z)
Then, calculate the Cov(X, YZ), have
Cov(X,Y VZ) = E|X YZ - (XX])----VX
(33)
(E| - E[X]E[Z] = C(y)
Therefore, the variance Var(X + YZ) is:
Var(X + +YZ = Var(X) + Var(Y 2) + 2Cov(X, YZ)
Var(X) + Var(Y) Var + Var(Y) 2
(34)
+ Var(()(E(r))) + 2 Z)
Proof is over.
Theorem B.4 With Assumption 4.1 and 4.2
when b > 1, the variance of post-update Q-value Qenen with Dz( function smaller than that with le loss function.
Proof. We first show the variance of Q-value estimation with l2 loss function. Following Equation 26
Qonon (8, a) xo Q0(s.a) + Byy- - Qo(ss.))llV&o((s
With Assumption 4.1
we can have
y Qa(s,a) = (Q(s, (Q(s, -Ee) = Eg Eg
Because the approximation noise Eg and Eg independent of each other, and both are independent of s, a and network 0 under Assumption 4.1 Therefore, the variable Eg Eg is independent from Q0(s,a) a) and ([
Ler X Q), Y -0-Ee E(Y) E(Eo Eg) For simplicity, denote the mean and variance of oo a)l as V and Var V), respectively.
Eg and Z V VoQo(s,
a) The noise so and eg have the same mean.
According the Lemma B.3 above, the variance of post-update Q-values with le function Var(QQnews (8,a)) is as follows, denoted as (Qri(()):
Vari, (Q) = Var(Qu(s, a)) + 32 Vara(E - Eg) Var(V) + +.3 Var(ea - (B)
(35)
Then, demonstrate the variance of Q-value estimation with Dz(x)) loss function. We denote it VarD, (Q). Similarly, let X = Qo(s, a), Y = =D,(Ea -Eg). Z = Vo a)ll According to the
Qinen ()) derived with Do(x), we can get VarD, (Q) as:
Qonen (8 Qo(8, a) (D Qa(s, a)) V (0
Varp, (Q) Var(Qo(s, + 3 Var aa) Var(V) + 3ar (Dr( e))) (36)
23

---

Published as a conference paper at ICLR 2026
Next, with Assumption 4.2 of Laplace-basee noise, the variance of Ea and Egj be approximated with the scale parameter of Laplace distribution, i.e., Var(Ea) Var(ee) 262. We have Var(=e
Eg) = 4b".
On the other hand, the [z(x) < 1/b. the DE - -Eg) E (-1/b,1/b), so the Var(D) EQ)) <
1/1² according to Popoviciu's inequality.
Finally, compare Da (Q) with Vania (Q), we have:
Da (Q) Vari, (Q) 32 Var(D) =Q)) Var(Ey +
< (1/6² - (var(V) + (B[V))))
(37)
<0
Proof done.
The condition keeps the same for satisfying the Theorem 4.4 Theorem B.4 simultaneously. In summary, the estimation bias and variance of Q-value can be reduced using (x(x) function for Q-value updates, compared with using the l2 loss function.
B.6 ALGORITHM FRAMEWORK
We summarize the framework of LAROO Algorithm
Algorithm LAROO
1: Input: Offline dataset Dosf, offline RI. algorithm F flimes value networks (Qo, ) policy
network Tÿ. Online replay buffer Dow: online steps
2: Offine Phase: Training Fof fline using Dofs
3: Online Phase: Remove the constraints in Fofflines named online algorithm Fontine
4: for = 1 to T do
5: Collect an online sample т = (s, o, F, 8') with Tp in online environment, update online replay
buffer D
6: Sample a training batch B from Dosy U Don
7: Update the parameter of Laplace-distributed noise, with Equation 7
8: Update each Q-function Qe, with the loss function g according to Fontine
9: Update policy मे according Fontine
10: end for
B.7 COMPARISON WITH BATCH NORMALIZATION
The widely adopted batch- nonealaormalizationn (BN) technique mitigate the impact of the heavy-tailed biases through explicit normalization for the Q-value losses. For further show the effectiveness of LAROO, We compare it with the batch-normalization (BN) technique detail.
BN: When updating Q-value with batch normalization with MSE, the loss function is defined as:
(BatchNorm(5o(s, a)))
Cr (0) (x..,,,,"'~~
2
where BatchNorm (s, a)) is the normalized TD-errors within a training batch. These TD-errors are normalized by their mean and variance.
LAROO: When updating Q-value with our methods, the loss function is:
1 K
(o.).) = -Ex,a,r.')=BB
Dg (Q (1t)) a) - Ymmia (3. a, a,,,s')
K
(38)
0 (8) .0)
Umin (8,,,r, - max
min ISKSM
a'EA
24

---

Published as a conference paper at ICLR 2026
LAROO also tames heavy-tailed Q bias through distributional transformation: (1) LAROO ap- plies the ensemble models to reduce the mean of Q bias, shifting its mean toward zero. (2) LAROO captures the heavy-tailed nature of O-value biases with a parameterized Laplace-distriuuted noise, thereby alleviating the heavy tailedness of biases. (1) and (2) dynamically promote the asymmetric, heavy-tailed Q bias distribution into more centralized form.
We conduct experiments to compare the performance of these two methods. The agent of BN keeps hyperparameter settings be consistent with LAROO. Experimental results show that LAROO is more effective in improving performance in 020 RL in Table 2
Table 2: Comparison with Batch Normalization (BN).
Tisk
Type
LAROO
BN
medium
120.4 ::
36.2 = 13.3
Walker2d
modiom-rppla
1248
2.0
62.1 7.7
medium-exper
126.7 + 1.0
39.1 + 14.0
medium
1.2
48
balfcheetah
medium-replay
78-4-4
medium-espert
99.5 1.8
92.9 1.6
C EXPERIMENT DETAILS
C.I EXPERIMENT IMPLEMENTATION DETAILS
Implementation details for LAROO. Following the setting of baseline ENOTO (Zhao et al. ) we implement LAROO based on TD3 (Fujimoto et al. 20188) with ensemble Q-functions for Mujoco tasks and implement it based on LAPO (Chen et al. 2022 for Antmaze tasks. We follow the hy- perparameters of offline backbone algorithms. During online fine-tuning, a mini- mini-batch is uniformly sampled for training, with half from the offline buffer and the other half from the online buffer. The scale parameter b is updated according to MBBE at every step. We provide the hyperparameters of LAROO in Table
Table 3: Hyperparameters of LAROO on Mujoco tasks.
Hyperparameter
Value
Optimizer
Adam
Learning rate of critic network
3e -4
Learning rate of actor network
34
Target smoothing coefficient 7
5e -3
Batch size
256
Discount
0.99
Policy noise
0.2
Policy noise clipping
(-0.5,0.5)
Policy update frequency
2
LAROO Parameter
Value
Total environment steps T
10>
Critic UTD
5
Actor UTD
1
Ensemble Q-function size N Subset size M
5
2
Implementation details for baseline methods. We compare LAROO with baseline methods PEX /Zhan et al., 2023). SO2 (Zhang et al., 2024). ENOTO (Zhao et al. 2024), Cal-QL (Nakamoto ct al. 2023). BOORL (Hu et al. (024) and RLPD (Bll et al. (002)) in this paper. We rerun these methods with their official implementations. However, there are some special cases. We observe that SO2 Zhang et al. 2024) fails in Antmaze tasks, primarily because its backbone algorithm (SAC) struggles to handle with sparse reward tasks. Similarly, the BOORL (Hu et al. (024) struggles to perform well due to the backbone algorithm TD3BC. And these two 020 methods do not provide official implements for Antmaze and Adroit tasks.
25

---

Published as a conference paper at ICLR 2026
Therefore, rerun SO2 and BOORL using their official codes only for the Mujoco tasks. For Antmaze and Adroit tasks, in order to ensure a fair comparison, we implement SO2 based on LAPO (Chen al. 2022). aligning with setup, and estimate Q-values according to its methods. As for BOORI, we do not test it on the Antmaze and Adroit tasks due to the lack of provided codes. All experiments conducted on the same experimental setup, which includes eight NVIDIA Tesla V100 GPUs and an Intel Core 17-6700k CPU at 4.00GHz.
C.2 EXPERIMENT RESULTS
C.2.1 MANN-WIITNEYY U TEST FOR Q BIASES AND TARGET Q BIASES
The Mann-Whitner Whitney U test Hhann-Whimee--ee Willcoon test) Mann & Whitney 1947) is parametric statistical method. It can test whether two independent samples come from the same distribution. Meanwhile, it does not require the data to follow normal distribution, making it suitable for handling skewed or non-normally distributed data.
In section 4.1 set b2, which generally states that and Eg follow the same distribution. We conduct the Mann-W MainnW U test to validate it. We sample a number of samples from Eg and Eg at the same training step. The resulting p-values are significantly higher than 0.05, indicating that the noise and Eg could follow the same distribution. Therefore. the setting of bN = by is reasonable. This result can be understood from the fact that the target Q-value network copies the parameters of the Q-value network every few steps.
Table 4: P-values of Mann- Whineey test for Q biases and target Q biases.
Task
Type
P-value
medium
0.902
Halfcheetah
medium-replay
0.720
medium-expert
0.793
medium 0.704
Hopper
medium-replay
0.855
medium-expert
0.914
C.2.2 MAIN PERFORMANCE
We present the training curves of several 020 methods during online fine-tuning. We illustrate the normalized return curves mujoco tasks in Figure 12 and the training curves Antmaze tasks in Figure 13 As discussed in Experiment 5 LAROO outperforms previous state-of-the-an methods in final performance.
We conduct more experiments on the Antmaze and Adroit domains (Fu et al. 2020). Most methods perform poorly in certain scenarios due to the complexity of the Adroit environments. Additionally, previous research studied that the hyperparameters for most offline baseline algo- rithms in the Adroit environment differ from those used in the MuJoCo environment. Based on the CORL (Tarasov et al. 2023). we adjust the parameters of their offline models accordingly.
We conduct statistical significance tests on experimental results further validate the effec- tiveness. Specifically, we conduct mean difference testing using Bootstrap resampling on the final results of different methods. Mean difference testing aims to compare the means two groups and determine if the observed difference in means is statistically significant. The bootstrap resampling used to solve the limited number of results As shown in Table 6 the p-values are less than 0.05 in most scenarios, indicating statistically significant differences in means of LAROO compared to other algorithms. That is, the ad advantage of LAROO is significant compared to others in performance.
26

---

Published as a conference paper at ICLR 2026
halfclteetab--geee----
halichoetah daedrum oxpert-22
walker mediumdddmmmmmmmm expert-22
150
1994
100
100
Vallee
Vilue
0.5
0.0
5,5
2.0
0.2
04
2.5
00
0.2
16
0.6
0.5
Timaolopa
Timestopa
Timealop
Timeslope
hopper expertor2
roplay w2
walkerr2 rop
hopper rpplay
125-
Viloo
Vilue
75-
53-
20-
25-
25-
0.0
0.6
0.5
Timestapa
Tinesiip
163
165
30
halfcheetah moedium
walkerbd medium x2
Iullfehectall mandeicctall-mdotiii2222
hopper candom-??
150
100
Valua
0.0
0.2
0.4
9.6
00
0.2
04
0.6
0.8
Timestops
Timtetepss
Timestope
Timestope
LAROO
ENOTD
BOORL
Otfline
Figure 12: Normalized return curves of several baselines and LAROO 100K time steps. The reference line is the performance of offline pretrained agents. Every experimental result is averaged with five random seeds. Note that RLPD starts training agents from scratch rather than pretrained.
antmazz--arge-diveree..
antmaze-large-play-22
antmuze-mediumddivesee-
Normalzzdd Return
100-
100-
100
Normalized Return
Normalized Return
75
80-
60-
25-
40
0.0
0.2
0.4
0.6
0.8
1.0
1.0
0.4
0.8
Timestops
le5
Timestops
Timestops
antmaze-medium- -2ay--22
antmaze-umane umaze-v2
ammsvee-mssr-divveee-w
Normalized Return
100-
Normalized Return
100
Nornalized Renun
100-
50
0.8
0.4
1.0
0.0
0.6
Timesleps
165
Timesteps
Timesteps
165
LAROO ENOTO
Cal-QL
PEX
S02
BOORL
RLPD
Offline
Figure 13: The training curves of various 020 RL methods in Antmaze tasks.
C.2.3 TRAINING STABILITY
We evaluate and compare the training stability of 020 RL methods in this subsection. In previous research (Feng al. 2024), it defines the Normalized Cumulative Performance Drop (NCD) metric to quantify the degree of performance degradation in 020 RL. Meanwhile, the variance of perfor- mance is widely used evaluate the training stability. smaller NCD variance value indicates
27

---

Published as a conference paper at ICLR 2026
Table 5: Experimental results Antmaze and Adroit tasks of several 020 algorithms.
Task
Type
PEX
S02
Cal-QL
ENOTO
LAROO
umaze-diverse
60.0 - 45.6
66.3
66.3
47,5 - 85.0
63.3 71.
65.0 - 85.5
medium-play
68.8
83.6
Antmaze
medium--diverse
70.0 - 77.6
63.3 71.8
73.5 - 80.2
55.8 80.0
54.5 86.6
large-play
55.8
large diverse
30.0 -+ 24.7
35.0 39.1
36.2 - 42.8
35.5 31.1
36.6 43.0
Aram (0.1M)
58.5
74.1
72.2
105.4
human cloned
0.30 -a 0.20
0.11 0.22
-0.33
0.25
1.03 -0.12
0.11
0.25
0.40 - 0.03
0.06 0.34
-0.31 0.31
0.33 -> 0..3
0.06 0.46
75.50
82,50
pen
cloned
81.00 -+ 88.20
82.90 - 93.70
-2.66 -2.68
-2.76 - -1.28
82.90 97.10
door
human
1.40 - 0.20
4,07
5,90
-0.34
0.33
13.28 - 0.10
4.07 - 5.35
cloned
6.30 0.22
0.32
10.30
-0.33 -> -0.33
-0.33 -0.27
0.32 10.30
human
0.90 - 0.18
3.74 1.56
0.27
-0.30
0.30 3,26
3.74 - 0.55
hammer
cloned
0.21
18.30
0.17
0.56
1.32 18.80
(0.10M
-12.7
44.8
25.4
16.5
52.5
Table 6: Mean difference testing of LAROO against other baselines.
Task
Type
LAROO vs PEX
LAROO S02
LAROO Cal-Qr.
LAROO ENOTO
LAROO BOORI.
random
medium
0.036
0.137
0.011
0.285
0.024
Hopper
medium-replay
0.009
0.061
medium-cxpert
0.046
0.181
0.987
0.130
0.026
expert
0.012
0.023
0.433
0.181
0.107
random
0.041
0.016
0.008
0.065
0.013
Walker2d
medium
0.017
0.346
0.046
0.887
0.326
medium-replay
0.013
0.616
0.006
0.020
0.021
medium expert
0.005
0.017
0.006
0.056
0.022
expert
0.065
0.553
0.481
0.655
0.017
random
0.013
0.017
0.007
0.205
0,104
Halfehectab
median
0.005
0.110
0.010
0.024
medium-replny
0.010
0.090
0.003
0.511
0.018
medium expert
expert
0.026
0.033
0.857
0.175
0.018
medium play
0.023
0.064
0.0230
0.096
Antmaze
modium--iiverse
large-play
0.022
0.064
0.488
0.217
Jarge-diverse.
0.010
0,022
0.059
0.007
better training stability. The NCD metric is defined as follows:
1 - Σ (((()) R())) R(0) R(t) R(0)
(39)
T+1 1=
where R(-) is the performance policy at the training step t. R(O) is the initial pretrained perfor- mance, T is the total online fine-tuning step and [(-) is indicator function that takes the value when the condition is satisfied and otherwise.
Table 7: Variance and NCD values of different 020 RL methods across Walker2d tasks.
Task
PEX
S02
Cal-QI.
ENOTO
BOORI. LAROO
medium
17.9
3.4
1.8
688
6.8
2.9
Varianee
medium-replay
0.8
modium- expert
5.0
0.8
6.3
1.0
medium
0.090
0.059
0.010
0.026
0.037
0.020
NCD
medium replay
0.083
0.025
0.015
0.020
0.013
medium- expert
0.023
0.031
The experimental results in Table that LAROO significantly mitigates the performance degra- dation during fine-tuning. In Figure [4] the standard deviation values of LAROO are noticeably lower than those of most methods. While Cal-QL achieves better stability due to its constraints, LAROO generally demonstrates better performance improvement within limited number of steps. In conclusion, LAROO achieve significant training stability and performance improvement.
28

---

Published as a conference paper at ICLR 2026
30
20-
10
(a) PEX
(b) Cal-QL
(c) SO2
(d) ENOTO
(e) LAROO
Figure 14: The standard deviation values of O20 RL methods on the D4RL datasets are presented,
with each result averaged over seeds. The task environment names abbreviated; for example, 'W-m' refers to "Waker2dd-eddie
C.3 EVALUATION OF COMPATIBILITY OF LAROO
In this study, we propose the Q-value loss function Dz(x)) in Equation 6 for robust Q-value estima- tion. This function can be easily incorporated into previous O20 methods for further performance improvement. To evaluate the compatibility of LAROO with other methods, we select TD3, Cal-QL and ENOTO (Zhao et al. (0224) as these three methods exhibit progressive performance in 020 RI., and then replace the la loss function with the function (6) for Q-value updates.
Table 8: Experimental results of integrating LAROO with other 020 methods.
Task
Type
TD3
TD3+LAROO
Cal-QL
CaiQQL+LAROO
ENOTO
ENOTO+LAROO
medium
9.3
82.1
1: 0.5
110.2 1: 6.8
1: 4.2
Walker2d
medium replay
65.3 1: 6.4
80.5 1: 3.2
80.7 1 5.7
91.2 1 0.9
110.2 1 0.8
119.8 2.0
medium-exper expert
86.3
101.5 3.7
110.2
111.2 2.3
118.0 1
127.5 4:
The experimental results validate that our proposed robust loss function is generally effective and serve as plug-in method further enhance performance over existing methods. It is worth noting that these experiments also demonstrate the compatibility of LAROO with different offline baselines, c.g., Cal-QL based on CQL (Kumar al. 2020), and ENOTO is based on ensemble SAC (Haarnoja et al. 2018), LAROO can integrate with these offline baselines well and be highly compatible with Q-value-basse based algorithms, further indicating its versatility and applicability.
C.4 ABLATION STUDY
We conduct ablation studies for LAROO and provide detailed experimental results in this subsection.
C.4.1 ABLATION FOR LOSS FUNCTION
We conduct ablation studies to validate the effectiveness of L (z) in this subsection. We first eval- uate the performance with and without (x)x) (i.e., using l2 loss function). Meanwhile, considering that previous research (Zahra et al., 2014, Belagiannis et al. 2015, Cavazza & Murino 2016) has demonstrated the effectiveness of various robust loss functions, such the Huber function and Cauchy loss function, in handling outliers in training data, we also compare (x( with these func- tions for Q-value updates during online fine-tuning.
The results show that (x) outperforms others for effectively promoting heavy-tailed Q-value bi- ases into standardized form, improving training stability and performance, as discussed in Sec- tion
C.4.2 ABLATION FOR UTD RATIO AND ENSEMBLE SIZE
We first demonstrate the effectiveness of LAROO with UTD and ensemble size N 1 in Table 10 We compare it with the Cal-QL Nakamoto et al.) 2023) and PEX (Zhang et al. 2023)
Other baselines, such as ENOTO (Zhao et a 2024). SO2 (Zhang et al. 2024). BOORL (Hu et al. 2024). also leverage high UTD or large ensembles, and are concluded from the ablation experiments. As shown in Table 10 LAROO outperforms Cal-QL in 10 out of 13 experiments, demonstrating its effectiveness without the benefits of high UTD ensemble models.
29

---

Published as a conference paper at ICLR 2026
Table 9: Ablation for different loss functions.
Task
Type
LAROO
Huber
Cauchty
MSB
MLE
medium
92.5 1 1.2
91.5 1 4.2
52.4 1. 12.6
89.7 3.2
90.2 1. 1.2
Halfcheerah
medium-replay
= 1,9
81.5
medium--cxport
99.5 = 1.8
100.7
5,3
37.5
1.5+55. 5.0
90.3 = 4.1
92.5
medium
120.4
110.7 = 5.7
107.6 +
88.3 10.1
85.2 11.6
Walker2d
medium replay
124.8
110.4 1 0.9
39.4 1 21.8
104.7 1.11
118.1 1. 0.7
medium-expert
126.7
1104 = 10.5
103.8 + 4.7
105.4
120.5 0.3
medium
106.3
90.4 15.2
1.8
7.4
89.3 16.9
Hopper
medium--repay
109.5 1.8
102.2 = 13.0
19.3 11.1
90.9 1.3
100.2 = 2.6
medium expert
17.6
large-play
3.2
66.5
48.3
5.0
49.2 + 2.1
Antmaze
medium-play
83.6 = 3.6
83.3 = 1.6
+.8
71.6 1.6
78.7 = 2.1
madium--iierrse
86.6 = 3.3
75.0 20 8.3
80.0 + 0.5
78.4 = 11.6
85.2 + 1.0
Table 10: Ablation for UTD and ensemble size. The normalized returns of LAROO with ensemble size N
and UTD 1 for a more fair comparison with Cal-QL and PEX.
Task
Type
LAROO
PEX
Cal-QL
Cal-QL LAROO
medium
80.2 1.6
55.3
52.2 1.9
58.2 0.5 (+ 6.0)
Halfcheetah
medium-replay
82.2 2.3
52.1 2.1
47.9
50.5 L4
medium-expert
93.3 1.6
90.3 + 7.4
91.2
96,8 008:2.1. (+ 5.6)
medium
95.4 + 10.8
42.5 17.9
83.7 0.5
89.4 5.6)
Walker2d
medium-replay
101.3 -: 8.3
40.5 6.8
5.7
9. 9.5) 1112 (+
medium-expert
108.3 -: 6.7
4.6
110.2 0.8
medium
85.5
78.5 14.0
90.8 :: 7.0
5.2
Hopper
medium-replay
104.4 + 2.1
87.5 23.8
92.8 - 4.8
105.3 2.7 12.5)
medium-exp exppet
95.5 +: 7.3
80.2 18.5
108.3 2.8
111.0 1.00+-58( 2.7)
medium play
83.7 2.0
+.3
77.5 2.5
1.1
medium-diverss
86.7 1:
3.3
80.2 2.6
3. 0.2) 47.7 4.7
Antmaze
large-play
50,0 +: 3.3
5.0
47.5 2.5
large diverse
43.3 +:
24.7 2.7
42.8 3.2
46.7 3.3 3.9)
We conduct ablation experiments only for the UTD, as shown in Table [I It has been widely studied that the high UTD promotes the sample efficiency and final performance. Our experiment further validate that higher UTD promotes the performance in offline-to effline-to-onling to-online scenarios.
Table 11: Ablation for UTD ratio. The ensemble size is five, and the updates-to-data ratio (UTD) varies.
Task
Type
UTD
UTD 3
UTD (LAROO)
UTD
medium
88.5 + 1.3
94.7 + 0.3
92.5 1.2
93.2 +
Halfcheetah
medium-replay
80.8 + 2.7
83.6 + 1.8
84.7 1.9
89.6 + 2.1
medium-exppet
92.3 + 6.6
103.1 + 1.6
99.5 + 1.8
98.4 + 0.6
medium
98.2 + 5.2
99.5 + 3.5
106.7 +: 2.6
91.2 + 10.5
Hopper
medium-replay
105.7 + 1.6
107.5 = 1.8
109.5 + 1.8
109.4 + 2.3
medium-expert
101.1 4.8
113.0 + 3.4
112.3 + 0.7
106.4 4.9
C.4.3 ABLATION FOR THE COMPONENTS OF LAROO
LAROO mainly consists of two components: (1) the noise model for capturing the heavy-tailed Q bias, and (2) the ensemble model. To evaluate their effectiveness, we conduct ablation studies under two conditions:
without the noise model and the other without the ensemble model. The results, as shown in Table [2 demonstrate that both components can improve the training stability and the final performance, and the noise model plays
critical role in LAROO.
The distributions of Q bias in Figure 15 provide insights into the effectiveness of each component in mitigating heavy-tailed Q biases. The ensemble models reduce Q-value overestimation bias, shifting the Q bias distribution closer to zero. However, using ensemble models alone still results in high taiy--ta,,,,nes, indicated by the Kurtosis values in Figure (5 and (b), which exceed
30

---

Published as a conference paper at ICLR 2026
10. On the other hand, the noise model effectively alleviates the heavy-tailed nature of the biases, but still suffers from significant overestimation. Finally. LAROO combines the strengths of both components, transforming the heavy-tailed Q-value biases into more standardized form, shown in Figure 15 (c) and (f).
Table 12: Ablation studies for components of LAROO. We demonstrate the normalized returns without the ensemble term and without the noise model respectively.
Task
Type
LAROO w/o ensemble
LAROO w/o noise model
LAROO
medium-replay
105.0 3.8
88.3 + 10.1
124,8 +: 2.0
Walker2d
medium
106.2 2.9
104.7 + 1.7
120.4 +: 1.9
medium-expert
111.8 6.3
105.4 + 3.7
126.7 + 1.0
medium-replay
81.9 + 2.6
83.9 + 2.8
84.7 + 1.9
Halfcheetah
medium
89.5 2.1
87.7 + 1.2
92.5 + 1.2
medium-expert
92.4 + 3.1
90.3 + 4.1
99.5 + 1.8
medium-replay
105.6 1.2
1.3
109.5 1.8
Hopper
medium
6.3
7.4
106.7 2.6
medium-expert
108.3 3.2
99.9 17.6
112.3 0.7
walker2dmmodimm
walker26-meduum-22
walker20
16
THM
and
14
25
Parennage
12
Pereentage %5)
15
15
-200
-100
100
210
300
500
-10
100 bias
200
300
-200
-10
100
200
300
O
bios
bias
(a) bias w/o noise model
(b) Q bias w/o ensemble models
Q bias of LAROO
wwller22-mmdiimm-zpee--
17.5
Vetteee Work Kansra
Vereve 0068 51
23
P
15 30 1..
12.5
50
230 -100
-100
250
Q
Q
(d) Q bias w/o noise model
(e) Q bias w/o ensemble models
(f) Q bias of LAROO
Figure 15: Ablation for the components. Left: the empirical Q bias distribution during fine-tuning w/o noise model. Medium: the empirical bias w/o ensemble models. Right: the empirical Q bias of LAROO.
We present the training curves MuJoCo tasks without the Laplace noise model and without semble models, respectively, as shown in Figure
Table 13: The NCD values of LAROO without ensemble model and without the Laplace noise modeling. lower NCD means the better training stability.
Task
Type
LAROD wil ensemble model
LARO0 wrs noise model
LAR0O
medium
0.123
0.004
0.020
Wilkerdd
meduum-eppa
cun15
0,015
0U013
medium exper
0.034
0.008
0.051
mediun
CLU17
0
0U016
Hopper
meJuum-repäa
CUB14
0,017
CU014
medium expert
0J134
0,336
(U029
31

---

Published as a conference paper at ICLR 2026
hogper-medium--2
walker Milkerrd-mddiimn -eddimm-eeeeer---
1.0-
100
30-
Value
80
Value
60-
LAROO
40-
ensemble
50
noise model
25-
0.0
0.2
L.
0..0
0.4
0.6
0.8
1.0
0.0
0.2
0144
0.5
0.8
1.0
Timesteps
Timesteps
Timesteps
walker2d replayy---r---22222
halfcheetah aedeaaaaaaemmmmmm-mmmm. expert-22
Mulfchectah-mmedinee
100-
100
Value
75-
Valuee
Value
80
50-
40-
60-
20
0.0
0.2
0.5
0.8
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
Timestens
Timesteps
Timestops
Figure 16: Ablation for components of LAROO: the training curves of LAROO without Laplace noise model and without ensemble models in Mojoco tasks, respectively.
C.4.4 ABLATION FOR PARAMETER B
We use parametric approach to model the noise model and use MBBE to estimate the variance of the noise. The parameter b adaptively updated according to Equation demonstrate that adaptive is more effective for better approximating the true Q bias distribution. Meanwhile, we approximate the estimation of parameter by using TD-errors as substitute of Q biases to facilitate the implementation. The curves in Figure 17 (b) show that the estimated b values obtained from Q bias and TD-error are similar.
every online step. We
Nomaltzed Return
120
Value of peuneeter
Q bizses
100
10-
60-
20
b-5
b-10
Tameslpps
Figure 17: (a) Normalized returns with different values of parameter b. (b) Curves of the estimated value of b during online fine-tuning using TD-errors and Q biases respectively. These experiments are conducted the aalker2d-medium task.
C.4.5 ABLATION FOR ASYMMETRIC DISTRIBUTION
Reparameterized Monte Carlo estimation of the KL divergence between asymmetric Laplace distributions. We model the Q-value bias noise using asymmetric Laplace distribution with location parameter μ e R and asymmetric scale parameters bj and b2 > 0. Its probability density function is given by
1
I 1
exp
x <
bi + b2 1
b2
f( n b1)
=
(40)
I - 4
exp
x 2
bi + by
bj
The corresponding cumulative distribution function (CDF) is
bz
r
exp
1 < Hb
bi
b2
by
F(x p, b..11,b) bz)
(41)
b1
I -
exp
x. 2
b
+
bz
b
32

---

Published as a conference paper at ICLR 2026
which admits bosedd-frmm inverse. For E (0, 1), the inverse CDF F-1
is
b1 + b2
b2
1 + b2 log
b2
b + b2
F= (lu1,,,e (..1,
(42)
bi b2
b
11 - bı log
(1 u)
2
b1
b1 + b2
Let ((x) AL(jy bi, b2) and p(x) - AL(up b,, b2) denote two asymmetric Laplace distributions. To obtain gradient estimator that is compatible with backpropagation, apply the reparameteri- zation trick. Instead of sampling I ~ q directly, we first draw u ~ Uniform(C. 1) and then map u via the differentiable transformation equation 42
4(0,1). 1),
x = T(;; Mg. b2) = F= Mag b1, b2).
(43)
Under this reparameterivation, the KI. divergence can be written as an expectation with respect to the fixed base distribution (((0, 1):
KL(g [p) 10,1) log (([(;; May b2)) (T Pa b1, b2))
(44)
In practice we approximate equation 44 by Monte Carlo sampling. Given S i.i.d. samples u(e) (0, 1), compute
KL(a = S Σ log - logg(((())) 214) = Tu' ;g: b1,
(45)
Since the transformation T(:i Pag b1, 62) differentiable with respect to fug (and, if required, with respect to bj and b2), the estimator KL(g p) defines valid pathwise gradient estimator. This enables end-to-end optimization of the parameters of q (c.g., Q-value predictions) using standard gradient-based methods, while retaining flexible asymmetric Laplace likelihood model.
We conduct experiments to compare asymmetric and symmetric Laplace modeling for the Q- bias. Consistent with the training procedure described in the main text, we assume that Eg and Eg follow an asymmetric Laplace distribution with shared scale parameters bj and b2. We esti- mate bn and b2 for each training batch via maximum likelihood estimation, and compute the Kull- back-Leibler divergence DKL. using the Monte Carlo and reparameterization techniques introduced above. All other components the model and hyperparameters are kept identical to those settings in LAROO
We report experimental results on several MuJoCo environments. The results in Table 14 indi- cate that, due the additional computational complexity and approximation errors, the asymmetric Laplace model yields worse performance compared to LAROO.
Table 14: Comparison with asymmetric Laplace modeling and symmetric Laplace modeling.
Task
Type
Laplace modeling (LAROO)
Laplace modeling
WalkerSd
medium
120.4 2.9
36.2 13.3
expert
halfcheetah
medium-espert expert
medium
99.5
166
hopper
medium
medium
112.3
106.7
0.7
1.6
92.9 i 1.6
83.3 4
C.5 THE USE OF LARGE LANGUAGE MODELS
In our study, we use the large language models only for polishing sentences and checking the gram- matical emror..
33

---

Published as a conference paper at ICLR 2026
D THE Q BIAS IN MORE RL DOMAIN
D.1 THE Q BIAS OF SPARSE REWARD DOMAINS O20 RL
In this subsection, we supplement more distribution of Q bias test whether the heavy-tailed Q bias exists across 020 RL settings. We test several existing 020 RL methods in sparse reward environments (Antmaze) and semi-sparse reward environments (Adroit). The experiment settings keep the same as those in Appendix C
We present the Q bias distributions of Antmaze tasks in Figure 18 and these of Adroit tasks in Figure 19 We observe that the Q bias is significantly large in Adroit tasks, because the Q-value network struggles to learn from sparse reward and accurately estimate each state-action pair (Chen)
al. 2022).
The experimental results validate that the heavy-tailed issue of Q bias generally exists across the 020 RL domains. We have discussed the reasons that may contribute the heavy-tailed bias in 020 RL settings in Appendix A.3
Cal-Ql, arrinae lapge-llay-222 5UC
Cal
1MI
Cai-121 ontmaae mediom play
Cel-OU cmneze--nsiiu plays2
Percenaage (5)
Moas: 289,
Vanance 63
Poremlgge:
Vaniance: $237.77
Knrteess 5.63
Pocupaee apo (%)
Mesn: M3s.:
Kurtaiio 4.54
$7.97
Percmtgge
Kontsde 494
Vartance
+
540
100
-000
-500
2000
-100
-10 -40
bizs
bias
brias
CalOC creciun divisle-22
Cal-0l www.aro-mmdmmmmmmmmmmmm..... dww-diieeedddiiii 1UKK
PLX artmano-
PUX 100K
Pacentag (9))
Kustoii:
Vorona:: 2141, 49
Muar: -108
Percentage
Katleso 6.35
Aettnnee 171L5
Persenage (%)
100
Kustosie: 5.78
Moan: -58
Variiaze: 364.48
Percentage
2.
Kurtisie 6,54
Verianne 491
Moan: -2.95
450-400 2w1 201.000020000000000000 2502220--60 1it
350
300
-20
-000
-200
.000
-00
1100
-00
40
40
-00
9 bias
biss
Q
bime
FEX kamlerleooell 150K
catrense medium
PEX ueE play 150C
Mear: -ST 05
17.5
Moan: -7
Percentage (5))
Vartace: 436
15:0
Vettnne::
Vanaace
Kornstt 6.36
Porcettage
Pareannge
12.5
Purcentago
10.0
7.5
2.5
40 higs
5105
-80
-50
-40
-23
0100
Q
bias
Q bius
Q biss
PEX antrraze medium divese 50g
PEX antmnaee medius divene 100g
502 antiaaee large play 50K
502 arimose lugg play LOnK
Mean: -55 52
Mean: -5..7
Moas: 13.23
Moan: 5,37
Porcentgg (%)
Veriece:: 371
Percontagg (%)
Verienee 70
Poruemagg (9))
10-
Vertanee: 355
Percentage 1o
Verbmeo 89
Kirtosic 5.17
Kirteiic: 5.08
Kurtosic
Karmai: 14.88
B
pby
Peroeingg (%))
Kuntstic 4.56
Muiaaee: 166.76
Perorntagg
30
1t.
Kurtoiss: 14,58
89.15
Percentage (%)
Kuntsa: 5.89
Variazee
E 15.0
20.0
17.5
Vallowe 238.11
Kunoiis: 6,0
5.0-
2.5
20
401
60
10
40
10
20
10
-55
75
100
125
0.0
20
0
20
40
80
hiss
Q hids
hiss
Q
hiss
Figure 18: The heavy-tailed distribution of Q bias in sparse reward domain. methods (SO2 (Zhang et al.) 20044) Cal-QL. (Nakamoto et al. 2023), PEX (Zhang al. 2023) in Antmaze tasks (large-play. medium-plas and medium diverse.
We test three 020 RL
34

---

Published as a conference paper at ICLR 2026
PEX pea humes vi S0K
PEX pen bumen 1006
pen-clioeévvi
PEX per-clonedvi 198K
Posceninge (%6
Mean: 6477.34
Kustoss: 2199542.57
Percerage (%6)
Kurtoss: 2,36
Vanance: 1413966 1I
Mean: 6650
Poroentagen (%)
Mallasee: 2281185.33
Kunosss: 14.65
Porcentage (56)
12
10
Maae: $305.27
Kunusie 7.03
Vartana: 24/0512.54
906
990
1009
2000
1066
6000
2000
biss
Q hike
Q hike
Q hiiss
S032 pon-tmmnvei
10CK
25
502 pan pa--lonnd-vl 50K
902 pet-dlreeór) 106K
Peccnage (N0.
B 15 1
Kantusis 22.44
Move
Percintgge (%)
8.
Kartois: 5.15
Variaao: 264,33
Porcenaage (5))
20-
Mear: 439 28
Varancr 64391667
20-
Mear: 453 28
Vanianer 649916.67
Porcenaage
3:00-2500-2000
3:00-2500-0000
1000-500
bits
bias
Figure 19: The heavy-tailed distribution of Q bias in semi-sparse reward domain. We test
SO2 (Zhang et al., (024) and PEX (Zhang et al. 2023 in Adroit tasks (pen-human, pen-cloned).
D.2 THE Q BIAS IN ONLINE RL
In this subsection, we test the distribution of Q-bias in the 020 RL setting. We conduct these tests to gain deeper understanding of the bias distribution in 020 RL. Our research remains primarily focused on 020 settings.
We adopt two effective, widely used online algorithms SAC (Haarnoja al. 2018) and TD3 (Fuji- moto et al., 2018), across several MuJoCo tasks. We choose four fixed checkpoints: 100K, 300K, 600K and 1M (million) steps better understand the evolving distribution of Q bias during training.
The resulting distributions are shown in Figure 20
In early training stages (100K), due the poor estimation ability of Q-value network, Q bias exhibits complex and irregular distributional characteristics, including multi-peak and dispersion, making it difficult to fit with a specific distribution. During training (300K, 600K), we also observed the occurrence of heavy-tailed Q-bias, mainly due to a systematic overestimation problem. In later stages (IM), the Q-bias distribution gradually became concentrated and Gaussian-like, which is reasonable according to the convergence theorem of Q-functions.
In summary, in pure online RL, training data are collected on-policy and evolve more smoothly with the policy, such sharp non-uniform shifts are less severe, and heavy-tailed Q-bias is not consistently sustained.
D.3 THE Q BIAS IN OFFLINE RL
In this subsection, the effective offline algorithms-TDSS TD3BC (Fujimoto & Gul 202I). CQL (Kumar et al. 2020) and IQL (Kostrikov et al.) -arosss several MuJoCo tasks. We also choose four fixed checkpoints: 100K, 300K, 600K and IM (million) steps to better observe the Q bias in offline RL. To obtain ground-truth Q-values for iate-action pairs in the offline datasets, we initialize each pair as the starting point, perform a full-episode rollout, and estimate its true Q-value via Monte Carlo evalua- tion. The corresponding Q-bias distributions are presented
examine the distribution of Q-bias in the offline RL setting. We adopt
21
Experimental results demonstrate that Q bias also show heavy-tailed behavior in offline RI,, exhibit- ing negative skewness with notably high biases in the left tail (i.e., large underestimation). This phenomenon is reasonable because these offline methods usually employ conservative Q-learning approach.
To assess whether LAROO remains effective in offline settings, we integrated the Laplace noise modeling into several offline RL algorithms and evaluated its impact on performance. The ex- perimental results in Table 15 show only modest improvements. This is primarily because offline methods rely on strictly conservative Q-estimation strategies, which make it difficult for LAROO to correct extremely low bias.
35

---

Published as a conference paper at ICLR 2026
SAC bollihetshe2 100K
SAC belichetthhe2 300
SAChediaatabr 603K
32,59
Maan-22111
20,66
Kartods: 2.25
Ventaaa: 502 72
Percestage (10)
Variance:
Percemtage (%)
Kurtode: 5.96
Veriana: 68 36
Perommbage
50
200
33
walker2d--22
2
SAC 1N
Mi
Morr
Meaa:: 76.37
Pareptage (%)
Kieraiie 3.95
Kurtosia 4.17
3831.30
Pereenlage (%g
Kirtosie 3.95
Vietiani:: 1657
Parcstian (N)
Pervatage
-100
100
200-1
150
talisheetahhc22
TBS halleheaaah-a222 SO0K
TD3 halfcheemah-v22
Moxs: 2709.31
Moae 41
Meser
Minnn -67.25
Percentuge: (%)
Kurtoss: 2.29
Kirtoiic: 1.6
Ventanee 458 92
Pereentage (%)
2 2 5
Kartonte:
Varianze:
Penombage
Porcmiage
bcas
Q
TD3 walle26622 Y00K
walkerDd
TD. wolkerrd IM
Perentage (%
Variance: 339,00
Mraan 49
Peromntgge 16
Mariance: 496.01
Percentage (%)
10-
Vailance: 5581.11
Percentage (%6)
Moun
Variaasz: 1342.55
Kurtssts 4,09
Kustosii: 6.76
Kustosis 5.43
-20
130-128
200
Q
Q bizs
bizs
Q
bias
Figure 20: The distribution of Q bias during online training in pure online RL settings. We test
SAC (Haarnoja et al. 2018) and TD3 (Fujimoto et al.) 2018 in MuJoCo tasks.
Table 15: Experimental results of integrating LAROO with several offline methods. Each result is averaged three random seeds.
Task
Type
TD3BC
TD3BC+LARO0
CQL-LAROO
IQL+LAROO
medium
82.7 4.5
6.2
= 3.3
82.4
80.6
Walker2d
medium replay
85.6 + 5.2
89.5 3.2
76.1 E 13.2
88.2 4.9
82.1 +:
79.8 J:: 2.0
medium-ent expert
110.3
111.5
113.5
medium
48.1
Halfcheetah
medium replay
44.8 1. 0.6
50.8 1. 3.2
47.1 1.7
51.2 1 0.9
44.5 1 0.8
46.2 12....0 2.0
medium-expert
91.3 1.33-1.6.
95.9
99.1
Table 16: Normalized returns of integrating LAROO with online RL methods at one million training step. Each result averaged over three random seeds.
Task
TD3
TD3 LAROO
SAC
SAC LAROO
Walker2d-v2
89.3 + 1.3
94.8 + 1.7
114.5 + 2.2
115.2 +: 2.3
Halfcheetah-v2
54.3 +
59.3 + 0.9
70.1 + 4.2
70.3 3.3
Hopper-v2
93.0
99.5 + 1.4
105.5 +
110.2 + 3.5
36

---

Published as a conference paper at ICLR 2026
CCL balfchetta median 200K
COL ballobedihh modians-12 200%
COL helfehectal medium 6268
Moam 68 18
Mear: 89.31
Mexa: $8.06
12-
85.00
Percentage (9)
Vanvase
Poncontage (%)
Kuntsiia
Porcens-gee (%)
Kastoiis
Varianee: 20
10-
8.
Verianee: 18.52
Konteie 10.55
50
90
90
100
70
90
100
30
70
100
50
60
bias
bias
buas
biks
COL meeltere--mermmmmm- 12 100K
Ogg molloott-mmdimmm-
CCB. 2 5008
CQL wellow2-mmelimm-ee IM
Moan: 80.65
Morn: 79, 17
". 12
Meaa: 291 .22
Poroemings (96
Vertaso: 165185
Perorntage 10-
Vorimor 3945,47
Percentign (%)
Vatiama: $19.38
Percentage (%))
Variance: 8593 27
Karusis: 4.16
Kuttoss: 4.53
Kanuss::
Kurtuss
m
RVL
hellchhstabb222 100K
hallcheetah-s2
batfcherrah
Mis::
bo a vt 5
Moar:
Parcentage (%)
Pereetagge (%)
11.73
Purculagg (SQ
Kurtons: 4008
Kuroons:: 4.76
Kurtons: 4,06
Kurtodi:
Parcentage
2.
1 x
100. 2immr 3006
10L. 2nsimm22
100, walkwakkkddd mediumn-d
12
14-
14
Meaa:
Percentnge (%)
K 5000
Percumtagon 19
Kanteis: 6.17
Percentage (1))
Karous
Porcenaage (%)
to
Kurtesic 4.07
4
20
60
100
20
40
63
60
100
20
40
40
100
20
69
90
100
bisss
Q
bins
brns
1 biss
TD hallcherrd mellanonodo-mm----2--- 12 1968
TD4BC (ulfeiecah medina
TD3BC belfcheerab andictt-r2 0028
TD3BC holtt medium 12 1M
Porumtug (6)
Kunteiic
Mtam: 50 1,22
Vertane:: 773.4
Pavestage
12.5
150-
Mean: 419.62
Vastaso: $25.25
Poreataage 2%c)
10
Mdeas: 428
Kurtoois:
Porcentago
to B 12
Meaz: 434,46
Variance: 731.32
Kunois::
10.0
755
b
250
250
400
310
330
450
TD3BC 2olloe2dmmsfuuur2 100
TDSBC osllerti-msiimmm 105K
TDSBC wellerliememmmmmmm
TD4BC walkerriimmmiium-
Mean: 269334
14
14
14.
Peraaagg (6)
Kunooss: 3.57
Vaniaa:: 7621 58
Kamnde 3.47
Poncoestgge (6)
Vaninnz 2717 97
Karmece 5.80
Percentnge (5))
Variane: 7306
Konacas 5.61
Perceetoon
>
0
90
100
180
200
250
300
350
30
100
150
200
250
300
350
30
100
150
200 bizs
250
300
355
50
100
150
250
250
300 350
Qbis
Q bizs
Q bias
Figure 21: The distribution of Q bias during training in pure offline RL settings. We test CQL Ku-
mar al. 2020), TD3BC (Fujimoto & Gu 2021 and IQL (Kostrikov al. 2022 in MuJoCo Tasks.
37


Reviews:
[{"review_id": "650gGE02tO", "reviewer": "Reviewer_2zgx", "summary": "This paper identifies that the Q-value estimation error during offline-to-online (O2O) RL exhibits a heavy-tailed distribution, and proposes to address this using Laplace-distributed noise modeling. Concretely, a new loss function is derived that is less sensitive to outliers. The proposed method achieves state-of-the-art performance on O2O RL benchmarks.", "questions": "- Could the authors also evaluate the proposed method on the D4RL Kitchen benchmark, which, similar to Adroit, is a challenging environment in D4RL?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The investigation of the Q-estimation error distribution is thorough and convincing.\n- The derivation of the Laplace-based robust loss function (D_b) is concise yet effectively mitigates the impact of outlier errors.\n- The experiment results show that the proposed method consistently outperforms the considered baselines.", "weaknesses": "- It would be helpful for the authors to examine whether the heavy-tailed error distribution only arises in the O2O RL setting. Similar distributions may also appear in other RL paradigms, such as purely online or offline RL. If that is the case, applying the proposed Laplace-based modeling to those settings could further validate its generality and effectiveness in handling heavy-tailed errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxVSBpiMJ1", "reviewer": "Reviewer_U4Kt", "summary": "This paper identifies and analyzes a previously unreported phenomenon in offline-to-online reinforcement learning (O2O RL): the Q-value estimation bias (Q-bias) tends to follow a heavy-tailed distribution during online fine-tuning. Such heavy-tailed behavior of Q-bias may introduce instability and impede effective performance improvement in fine-tuning. To address this, they propose Laplace-based Robust Offline-to-Online RL (LAROO), which models the Q-bias using a Laplace-distributed noise and introduces a robust policy evaluation loss derived from KL divergence minimization between Laplace distributions. LAROO further integrates an ensemble-based Q-value re-centering mechanism that shifts the Q-bias mean toward zero. Theoretical analysis shows that the proposed loss yields smaller single-step estimation bias than the widely-used L2 loss. Empirical results show improved stability and sample efficiency.", "questions": "Could you provide further clarification on the weaknesses? In particular, W1 and W3 will have the most significant impact on the overall rating.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. (Empirical observation of heavy-tailed Q-bias)\nIn the MuJoCo domain, this work provides the empirical observation that Q-value biases in O2O RL exhibit heavy-tailed and positively skewed distributions during online fine-tuning (Figures 1, 4, 5). This is a significant observation beyond the mean and variance analysis. Moreover, the experiments (Figure 7) demonstrate that large-magnitude positive Q-biases may correlate with performance degradation.\n\nS2. (Laplace-based policy evaluation)\nThe introduction of Laplace-distributed noise for modeling heavy-tailed bias is well-motivated. The derived loss function $D_b(x)$ effectively suppresses the influence of extreme outlier errors, serving as an empirically effective alternative to the L2 loss (standard Bellman loss).\n\nS3. (Theoretical ground)\nThe theoretical analysis demonstrates that the proposed loss function reduces single-step estimation bias compared to the L2 loss, which plausibly contributes to the observed stability and consistent performance improvements.\n\nS4. (Plug-in compatibility) \nThe proposed method can be used as a plug-in component for existing O2O methods. The experimental results show that replacing the Bellman loss in baselines with the proposed loss consistently improves their performance.", "weaknesses": "W1. (Questionable symmetry assumption of the Laplace model)\nAlthough empirical results (Figures 1, 4, 5) show that Q-bias distributions are typically heavy-tailed and right-skewed (i.e., asymmetric positive bias), LAROO assumes a symmetric Laplace distribution. This modeling choice simplifies the formulation but may fail to accurately capture the empirical asymmetry. Indeed, Figure 8 indicates that LAROO reduces the positive tail but may over-correct by introducing spurious negative bias.\n\nW2. (Limited justification for ensemble-based correction)\nThe proposed method integrates a random subset selection from ensemble Q-functions to re-center Q-bias, but it is somewhat heuristic. This paper lacks an intuitive explanation or theoretical reasoning as to why this mechanism effectively re-centers the bias.\n\nW3. (Q-bias analysis across limited domains)\nThe analysis of Q-bias and its heavy-tailed behavior is restricted to dense-reward environments such as MuJoCo. It remains unclear whether similar heavy-tailed characteristics would emerge in sparse-reward domains (e.g., AntMaze) or semi-sparse domains (e.g., Adroit, OGBench[1]).\n\nW4. (Theoretical and empirical disconnect)\nThe theoretical analysis establishes that the proposed loss reduces single-step estimation bias, but the connection to heavy-tailed variance reduction (the core empirical claim) remains indirect. It would strengthen the contribution to include a theoretical link between the Laplace modeling and variance-bounded Q estimation under heavy-tailed noise.\n\n\n[1] Park, Seohong, et al. \"Ogbench: Benchmarking offline goal-conditioned rl.\" arXiv preprint arXiv:2410.20092 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJbcyKKAmv", "reviewer": "Reviewer_SoSF", "summary": "This paper reveals that in offline-to-online RL, the Q-value biases follow a heavy-tailed distribution. To address this, this paper proposes LAROO, which introduces Laplace -based loss function to replace L2 loss for Q update, and uses ensemble models for Q target estimation to reduce the estimation bias. Theoretical analysis is provided to show that LAROO exhibits a smaller estimation bias than typical L2-based Q updates. Experimental results demonstrate that LAROO outperforms previous baselines on D4RL datasets.", "questions": "Please refer to the weaknesses to address the concerns. I will check the authors' responses to decide whether to revise the rating.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Clarity.** Generally, this paper is well-written and easy to follow. The motivation and observations are clear.\n\n**Novelty.** I think the key finding that the Q bias follows a heavy-tailed distribution is interesting and the method that levearges Laplace-based noise model is reasonable. Also, the authors provide clear theoretical analysis on why the proposed method could reduce the Q bias.\n\n**Significance.** The experimental results show that LAROO outperforms the previous baselines.", "weaknesses": "There are some points that need further clarifications.\n- The motivation that minimizing the KL divergence between $p(\\\\mathcal{Q}|\\\\mathcal{T}Q _ \\\\theta)$ and $q(\\\\mathcal{Q}|Q _ \\\\theta)$ in Line 225 is not clear. It seems just for the derivation of $D _ b(x)$. I wonder why minimizing such KL divergence could deal with the heavy-tailed Q bias issue and why it is reasonable. Could the authors give more clarifications on it?\n- In Line 277, the authors use TD-error as a surrogate for Laplace-based Q bias. That is to say, the TD-error is also assumed to follow a prior Laplace distribution. Then according to MLE, we could directly use L1 loss for Q update, then what is the advantage of using Equation (6)? Since Equation (6) is derived by minimizing the KL divergence, this is also related to the previous issue.\n- The heavy-tailed Q bias issue is not first observed in offline RL. Robust-IQL[1] also observes the heavy-tailed Q bias issue and addresses it with Huber loss, which is mroe easy to implement. This work uses Laplace distribution instead, could you demonstrate whether Huber loss is not used for your work?\n- LAROO seems not designed specifically for the offline-to-online setting, since Laplace-based Q update and ensemble models could also be applied to pure online setting. I wonder if the heavy-tailed Q bias issue also exists in pure online settings when using standard online RL algorithms like TD3 or SAC. If it does, what is the specific advantage of LAROO that makes it suitable for offline-to-online RL? If it does not, why does this issue manifest in the offline-to-online setting but not in pure online RL?\n-  (minor) In line 288, 'They' -> 'It'.\n\n[1] Towards Robust Offline RL Under Diverse Data Corruption. ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pY7ER2cHGt", "reviewer": "Reviewer_qzBg", "summary": "This paper is the first to empirically reveal the phenomenon that Q-value estimation bias in the online fine-tuning process of O2O pervasively follows a heavy-tailed distribution.To address this, the authors propose the LAROO (Laplace-based robust offline-to-online RL) approach, designed to mitigate the heavy-tailedness of Q-value estimation bias, thereby improving training stability and performance. The approach models the Q-value estimation bias using an adaptive Laplace-distributed noise, based on which a robust value loss function is constructed to reduce the variance of the Q-value bias during the learning process. Concurrently, it incorporates ensemble Q-models to shift the mean of the bias towards zero, ultimately achieving more robust and stable value estimation.", "questions": "1.  As noted in Weakness 1, training stability is the primary effect of the noise model, but no training curve plots or related metrics are provided in the ablation experiments. \n2.  The non-novel ensemble method significantly reduces the bias mean, and comparing Figures 13(e) and 13(f)  suggests it also helps reduce variance and kurtosis. As noted in Weakness 2, this raises the question whether the ensemble model might be more critical to performance improvement than the noise model.Could the authors provide ablation results across more environments for further analysis?\n3.  What is the experimental setup for LAROO w/o ensemble in Table 12, and why do the results differ from those in Table 10, where N = 1 and UTD = 1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- First revealed and empirically validated the pervasive heavy-tailed Q-bias phenomenon.\n- Through rigorous theoretical derivation, the LAROO method constructs a robust loss function based on the Laplace distribution, which is insensitive to outliers and reduces the variance of Q-bias.\n- Extensive experimental results demonstrate superior performance over published O2O methods across multiple environments.", "weaknesses": "1.  The paper lacks comprehensive ablation studies validating the specific contribution of the noise model to training stability.\n2.  The contribution of the non-novel ensemble model towards bias correction and final performance improvement is difficult to disentangle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)