Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
Authors: Longxiang He
Deheng Ye
Junbo Tan
Xueqian Wang
Li Shen
Venue: NeurIPS 2025 poster ()
Keywords: Robust RL
Offline-to-Online
Data Corruption
Primary area: reinforcement_learning
TL;DR: We propose RPEX, an Offline-to-Online method that improves the performance of offline pretrained RL policies under a wide range of data corruptions.

Abstract: Pretraining a policy on offline data followed by fine-tuning through online interactions, known as Offline-to-Online Reinforcement Learning (O2O RL), has emerged as a promising paradigm for real-world RL deployment. However, both offline datasets and online interactions in practical environments are often noisy or even maliciously corrupted, severely degrading the performance of O2O RL. Existing works primarily focus on mitigating the conservatism of offline policies via online exploration, while the robustness of O2O RL under data corruption, including states, actions, rewards, and dynamics, is still unexplored. In this work, we observe that data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration. To address this issue, we incorporate Inverse Probability Weighted (IPW) into the online exploration policy to alleviate heavy-tailedness, and propose a novel, simple yet effective method termed $\textbf{RPEX}$: $\textbf{R}$obust $\textbf{P}$olicy $\textbf{EX}$pansion. Extensive experimental results on D4RL datasets demonstrate that RPEX achieves SOTA O2O performance across a wide range of data corruption scenarios.

Robust Policy Expansion for Offline-to-Onlie RL under Diverse Data Corruption
Longxiang He'
Deheng Ye²
Junbo Tan lif Xueqian Wang' Li Shen
TTinngua University Tencent Shenzzen Campus of Sun Yat-sen University *Center for Al Theoretical Foundation and Systems, Shenzhen Loop Area Institute, China longxhe @ geaal..com shenli6@ mail.sysu.edu.cn
g Corresponding authors
Abstract
Pretraining a policy on offline data followed by fine-tuning through online interac- tions, known as Offline-to-Ollnne Reinforcement Learning (O20 RL), has emerged as promising paradigm for real-world RL deployment. However, both offline datasets and online interactions in practical environments are often noisy even maliciously corrupted, severely degrading the performance of O20 RL. Existing works primarily focus mitigating the conservatism of offline policies via online exploration, while the robustness of 020 RL under data corruption, including states, actions, rewards, and dynamics, is still unexplored. In this work, observe that data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration. To address this issue, we incorporate Inverse Probability Weighted (IPW) into the online exploration policy to alleviate heavy-tailedness, and propose novel, simple yet effective method termed RPEX: Robust Policy EXpansion. Extensive experimental results on D4RL datasets demonstrate that RPEX achieves SOTA 020 performance across wide range of data corruption scenarios. Code is available at
Introduction
Deploying reinforcement learning (RL) in real-world environments without relying on simulation has long been central goal in the field and is considered one of the most effective ways realize the full potential of RI. [45). However, directly applying online RL (30 11 14 15 in real-world settings is often expensive and unsafe due to the inherently exploratory nature of RL. To address this issue, Offline Reinforcement Learning (Offline RL), also known as Batch RL, has been proposed to learn optimal policies from pre-collected datasets without further interaction with the environment [ 28) Nonetheless, offline datasets typically lack full coverage of the state-action space, which often leads to suboptimal learned policies. Recently, inspired by the success of pretraining and fine-tuning paradigms [5] 9 (37), Pifline-to-Oniine RL (020 RL), which first trains policy using offline data and then fine-tunes it through online interactions, has emerged a promising paradigm for real-world
learning [56] 36 53].
remain constrained offline datasets derived from simulators, where data collection is relatively accurate. In contrast, real-world collecoleetedd by humans or sensors and generated through online interactions-ofien contain unpredictable noise or even malicious corruption [50 48 51 [54]. For example, during fine-tuning data collection for Vision- Language-A Aattuage-Action Action (VLA) models, human annotators may unintentionally deliberately introduce incorrect trajectories into the dataset. Therefore, robust 020 policy learning that can handle data corruption in both offline and online phases is critical for the practical deployment of O20 RL.
Nevertheless, most 020 RL approaches
36 56 53
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

48 51 54 on Robust RL have primarily focused on the theoretical properties and certification of offline reinforcement learning (RL) under data Corruption. Notably, Yang et al. [50] highlights that existing offline RI. algorithms, such as An et al. 12. suffer significant performance degradation when exposed to corrupted offline data. In contrast, -ty-e methods [25] 50 17 demonstrate greater robustness to such corruption [50]. Nevertheless, as noted in RIQL 50) IQL remains sensitive to dynamics corruption, as data corruption induces a heavy-tailed distribution in Q targets. To address this, RIQL improves the robustness of IQL by incorporating the Huber loss 20 2 and quantile-based Q estimators. Despite these advances, prior efforts have largely focused on robust offline RL. To the best of our knowledge, the challenge of learning robust and efficient 020 policies under both offline and online data corruption remains largely underexplored.
Most previous studies 43 49
38
50
In this paper, we aim to develop a robust and efficient 020 method capable of handling data corruption occurring in both offline and online phases. Our key insight that data corruption not only amplifies the heavy-tailedness of Q targets, as observed by Yang et al. [50), but also exacerbates the heavy-tailedness of the policy, which can severely hinder the efficiency of online exploration. Motivated by this observation, we propose simple yet effective O20 approach, termed RPEX (Robust Policy EXpansion), which integrates Inverse Probability Weighting (42) into Policy Expansion to alleviate policy heavy-tailedness. Experiments on D4RL tasks demonstrate that this simple modification yields an approximate 13.6% improvement in 020 performance over the vanilla PEX implementation.
[
To summarize, our main contributions are as follows:
To the best of our knowledge, we are the first to propose a robust 020 reinforcement learning algorithm, RPEX, which achieves efficient and robust performance improvements under both offline and online data corruption.
We demonstrate that data corruption not only amplifies the heavy-tailedness of Q-targets but also exacerbates the heavy-tailedness of the policy, thereby substantially impairing the efficiency of online exploration. Our theoretical analysis supports the effectiveness of our approach, demonstrating that applying IPW mitigates the heavy-tailee tailedness induced by data corruption and facilitates efficient exploration.
With a simple modification, extensive experimental results show that RPEX yields an additional performance improvement of approximately 13.6% over baseline methods.
2
Related Works
Robust Offline RL. Research on robust offline reinforcement learning (RL) can be broadly cate- gorized into two areas. The first category 4
38 focuses on test-time robustness and sample complexity, aiming to learn from clean data w wile defending against adversarial attacks or focuses on training- tiiiing-time time robustness, which seeks to learn a reliable policy even from corrupted datasets, with evaluation conducted under clean conditions. Yang et al. [50) theoretically establishes the robustness of IQL (25) under various data corruption scenarios and introduces RIQL, which enhances IQL's robustness via Huber loss, observation normalization, and quantile Q estimators. Xu et al. [48] propose the Robust Decision Transformer to improve the robustness of DT under limited and corrupted data by incorporating embedding dropout, mouussian-weigheee learning. and iterative data correction. In contrast to prior work, we primarily focus on the Offline-to-Onlin (020) setting.
49
44
noise during evaluation. The second category [50] 48 51 54
020 RL. 020 reinforcement learning (RL) aims enhance online sample efficiency while lever- aging a pretrained offline policy. A central challenge in 020 RI is mitigating the conservative behavior induced during offline training enable efficient exploration of high-quality samples. Naïvely continuing to train the offline policy with newly collected online data can be suboptimal and may degrade performance due to residual conservativeness from the offline phase. To address this, Nakamoto et al. (36) calibrates the offline value function to mitigate the "unlearning" or "forgetting" phenomenon. Zhou et al. [56] propose WSRL, which eliminates the need to retain offline data during online training. Zhang et al. [53] introduces policy expansion (PEX), which keeps the pretrained offline policy ne frozen and trains a separate learnable policy nœ with independently initialized parameters, decoupled from n.. This design retains the integrity of the offline policy while allowing the online policy to benefit from both prior knowledge and new interactions. Other approaches leverage Q-ensembles [55), data augmentation [31], or online RL exploration techniques [34] to

---

tackle similar issues. Our approach differs in that we explicitly address data corruption and noise during the 020 phase- an important but underexplored aspect current 020 research.
The most relevant related works are Yang et al. [50] and 7.hang et al. 53 However, RIQL [50] focuses robustness against offline data corruption, while PEX [53] focuses addressing the
unlearning phenomenon caused by directly transferring an offline policy to the online phase.
In contrast, our objective is to develop a robust 020 algorithm capable of withstanding various data corruptions in both offline datasets and online transitions.
Preliminaries and Problem Statement
Offline RL. Consider Markov decision process (MDP): M (S, do}. with state space S. action space A, environment dynamics P(s"|s,, [ss a) : S x x A [0, I], reward func- tion R S x A - I, discount factor Y 00 1), policy T(als) S
[0, 1), and ini- tial state distribution du- The value or Q-value of policy क is defined as (T ai)
91421..4 Dj-o (rr(st+) 0[ts) The value function of policy defined as V*((s)
LA (S. (a))))d The goal of RL to get policy maximize the cumulative discounted reward (B) = Ss do(s) (s)ds. dh (s) Li=o 7'Px (8; 8) the state visitation distribution induced by policy æ 45 40 and P- [8: s) the likelihood of the policy being in state s after following ı for timesteps.
In offline setting [12. environmental interaction is not allowed, and static dataset Doftine {(s, Cz, 1) t-1 is used to learn a policy. The key challenge in Offline RL to avoid out-of- distribution (OOD) actions that arise when evaluating the learned policy 12 28].
O20 RL. Given offline-pretrained value functions Qo Va and policy т., the objective Offline- to-Online Reinforcement Learning is to obtain the optimal policy To using the minimal number of online environment interactions.
Data Corruption. Following Yang et al. [50]. Ye et al. [51], data corruption or attacks inject random or adversarial noise into the original states s, actions a, rewards r. and next states s' drawn from t=1'
either the offline buffer Doffine or the online buffer D = {(s), a,, s' } resulting in corrupted N
buffers Dofttine and D {(.,, âr, f, 8(-1) (= For example, in random dynamics attack, random noise is injected into the next states, yielding s' = s' + X - std(s'), where À ~ Uniform [-e, <) dg denotes the dimensionality of the state space, is the corruption scale, and std(s') represents the de dy-dimensional standard deviation of all next states in the dataset. In contrast, adversarial corruption employs Projected Gradient Descent to optimize a noise e for ming Q(8, a) within predefined region, using pretrained value functions. Further details on data corruption are provided in Appendix C.T Throughout this paper, we slightly abuse the terms "corrupted" or "attacked" to refer to datasets or online trajectories that have been subjected to corruption.
Implicit Q-learning (IQL). To avoid OOD actions in offline RL, IQL (26) uses the state conditional upper expectile of action-value function ((8, a) to estimate the value function V(s), which avoid directly querying a Q-function with unseen action. For parameterized critic Q (8, a), target critic Q (8, a), and value network (s) the value objective is learned by
Lv (1b) = B(s,a) 1L a) (x))
where L5(u) =| 1(u 0)lu²
where 1 is the indicator function. Then, the Q-function is learned by minimizing the MSE loss
(x( = (e.,,,,") (r(( a) + (s') ()))).)
For policy extraction, IQL uses AWR by minimizing (3)
41 40 35
which trains the policy through weighted regression
E(s.) (xpo=l/-op(a(()((-550
(s)) log (a(ss))
Robust IQL (RIQL). enhance the robustness of IQI,, particularly its resistance to dynamic attacks, RIQL (50] replaces the quadratic loss in Eq. ( with the Huber loss (-) and derives the following
3

---

Online Fineluning
Kurtosis Values
CLEAN
Pollicy Set
Enviroument
RIQL IQL
Attucker
Policy s
Frozen
barter
Offline Pretrain
Offine Poliey
Office
actions
observations
dynamics
reward
9,a,F.²
Attack Types
Figure 1: (a) Problem Statement. A schematic illustration of the 020 attack, in which both the offline pre-training phase and the online fine-tuning phase targeted. (b) The Kurtosis Values [13] [3] of Policies. CLEAN means IQL trained without attacks. In contrast, RIQL and IQL are trained on the attacked datasets.
loss function.
if
[z|
S
LQ = E(eanne)>D
[ + +V((s) - a))) where (x)
[|2] if
where 5 trades off the resilience to outliers from & loss and the rapid convergence of & loss.
Policy Expansion. To mitigate performance degradation during the transition from offline online learning, PEX [53] utilizes a composite policy set II = [m R and selects actions generated by the policies in IT based their potential utilities (e.g., critic values) in both exploration and policy
learning. Specifically, for each element A = {az ~ (+)) } in the policy set IT, assuming the size of II is K, the probability of selecting a, the final action is
exp(Qo(s,a:)/a) Pw Σ, (s,)Q
Vi Vi [1, K],
where Qu denotes the offline pretrained critic function, and a represents the temperature parameter. In both PEX and our method, the policy is defined IT [T,,, mo) with K = 2, where अy the offline pretrained policy and Ty is the online learnable policy.
Problem Statement. In contrast to test-time robust RL 150 48 51 54], which primarily emphasizes
testing-time robustness-learning from clean data while withstanding attacks during evaluation our setting centers learning from corrupted data and evaluating in clean environment [50] 48] We assume access to an offline pretrained policy N3 and value functions Q(s, a) and (s)), all trained corrupted offline dataset comprising states, actions, rewards, and dynamics. Our objective is to obtain an optimal policy based on Ty through limited number of online interactions. Furthermore, attacks perturbations targeting states, actions, rewards, and dynamics also present during online interactions. We model these attacks as affecting the online replay buffer, i.e., although the agent's interaction with the environment remains clean, data corruption occurs during storage in the buffer, leading to the use of compromised corrupted data for online policy updates. As we discussed in the Introduction, such scenarios are common and critical the application of 020 RL in many real-world contexts [50 48 5T]
The complete problem statement is illustrated in Figure Ia).
4 How does Heavy-tailedness of Policy Affect Performance?
As noted in RIQL [50] IQL exhibits notable robustness to data corruption in states, actions, and rewards, yet it remains vulnerable to dynamics attacks due to the emergence of heavy-tailed Q target, i.e., r(s,a) a) +
(s''.. To address this, RIQI. incorporates the Huber loss into the critic updates (Eq. 44 to mitigate the heavy-tailedness of the Q target. However, we argue that the policy itself also becomes heavy-tailed as result of such attacks.

---

Dataset Transitions
Kurt: -0.24, Len: 18.4
Kurt: 25. Len: 18.4
Kurt: 30.05, Len: 27.6
Kurt: Len:
(a)
(b)
(c)
(c)
(e)
Figure 2: We study the impact of policy heavy-tailedness in the grid-world domain. An offline policy is trained using the dataset shown in Figure a[) and is then used to collect trajectories during the online exploration phase under both corrupted and uncorrupted settings. In Figures ( (b)-ee) the opacity of the green arrows indicates the selection probability. Red arrows denote the most probable trajectory generated by IQL or IQL+IPW under the respective conditions. Specifically, panel (a) illustrates the dataset transitions; panels (b) and (d) show trajectories selected by IQL under clean and corrupted value functions, respectively; panels (c) and (e) show trajectories selected by IQL+IPW under clean and corrupted value functions, respectively.
To illustrate this, we visualize the Kurtosis value the offline policy a pretrained by IQL or RIQL on the hopper-medium-replay task under various types of random attacks. We compute the Kurtosis value using 5000 samples collected after training IQL or RIQL for 10 steps. As shown in [ib)
-induced Seavy-tailednes, is more pronounced in the policy than in the Q target, whose heavy-tailedness primarily arises under dynamics attacks. Notably, the Kurtosis value of the IQL policy without attacks is very low (approximately 0.3), compared to policies trained under corrupted conditions. Furthermore, as illustrated Figure (b), RIQL exhibits more severe policy heavy-tailedness than IQL, suggesting that RIQL alleviates the heavy-tailedness of the Q-function at the cost of increased heavy-tailedness in the policy.
During the offline phase, the policy is evaluated deterministically by selecting the action with the highest probability in the evaluation environments, which limits the influence of policy heavy- tailedness on offline performance. However, in the fine-tuning phase, the offline policy is utilized for exploration to further refine the pretrained policy. In this context, the heavy-tailedness hinders efficient exploration. We analyze this issue in detail through series of simple toy maze experiments presented below.
Figure illustrates the effect of policy heavy-tailedness during the online phase through toy experiment, with experimental details provided Appendix [C.1 As shown in Figure d). IQL samples poor-quality trajectories (i.c., longer in length) during exploration due to corruption-induce heavy-tailedness, as evidenced by elevated Kurtosis values.
Key Observation: Attacks on various components induce heavy-tailed behavior in the policy, leading to inefficient exploration.
Intuitively, the heavy-tailedness resembles common exploration mechanisms in policy learning, such as entropy maximization or injected exploration noise, both assign small probabilities to certain actions. However, in contrast to intentional exploration strategies, the heavy-tailedness arises from malicious attacks and, more critically, introduces stochasticity that is beyond user control. In standard cilline-to-online setting, exploration is encouraged by assigning nonzero probabilities to suboptimal actions, with the degree of exploration regulated by predefined hyperparameters. In the presence of attacks, however, the heavy-tailedness the policy assigns nonzero weights to inferior actions in uncontrolled manner.
5 RPEX: Robust Policy Expansion
This observation motivates the adoption of Inverse Probability Weighting (IPW), technique com- monly used to address heavy-tailedness in classification tasks [42 7 52). Specifically, given the offline pretrained Q. V. and of candidate policies II , Tk]. incorporate IPW into
'The Kurtosis value Kur[XX] = E (x=#) quantifies heavy-tailedness relative to normal distribution.
For detailed introduction, refer to Appendix
5

---

R1OT
RIOI 1PW
Elean
Kurtosls:
Kurtosss:
Ntrtests:
Training Curve
$200
1.0.
12000
20
0023
3000
$200
0000
2000
40-
2300
20-
60
Figure 3: Action distributions generated by the offline pretrained policy under reward attack on the Haffcheetah-MR task. (a) Action distributions of RIQL under attack. (b) Action distributions of RIQL+IPW under attack. (c) Action distributions of IQL. without attack. (d) Comparison of RPEX (with IPW) against RIQL-PEX (without IPW) and RIQL (Vanilla RIQI...
Eq. 55) to derive the following objective:
exp(Qo(ss ai)/a) + kw
Pwlt
vi e [1, K],
(6)
E, exp(Qo(s, + kw
where
1Q- V (a:
ipw = CLIP
MIN, MAX)
(7)
Here, and K are coefficients, and Pw [i] denotes the probability of selecting a, Throughout this paper, we clip the value of with and set |MIN| low-quality actions while mitigating the impact of positive outliers.
10000 > |MAX] |100] to primarily penalize
In what way does Eq. (6) enhance exploration efficiency? Here, we provide an intuitive explanation of the behavior of Eq. (6). For low-quality actions (Q V 0), which generally have low selection probabilities and are widely distributed, the weight in Eq. (6) is predominantly determined by w ipw In such cases, low-quality actions that are rarely chosen offline become even less likely to be selected during online interaction. Conversely, for high-quality actions (Q V > 0), the clipped IPW values cause the weights in Eq. (6) to be primarily influenced by exp(Q(s, az)/a). This analysis indicates that Eq. (6) adaptively allocates weights according action quality and eliminates the need for intricate confidence interval design, thereby mitigating the exploration inefficiency induced by long-tail tail behavior.
As illustrated Figure 3 (a)-(c), data corruption can lead to heavy-tailedness in the policy and degrade 020 performance due to inefficient exploration (Figure 3 (d), Vanilla RIQL). Integrating IPW as presented Eq. (6) can significantly enhance robustness against reward attacks (Figure 3 (d)), as taily-tailedness is more pronounced in reward attacks, as shown in Figure (b). Besides, in we present a theoretical justification for the use of IPW.
Section 6
In practical implementation, following Zhang et al. [53], we freeze the offline policy and set K 2. We also examine the impact of commonly used techniques, such as state normalization [50] and the updates-to-date (UTD) ratio [56] 3. which denotes the number of updates performed per online trajectory, under 020 corruption settings. Notably, we find that state normalization improves the performance of 020 methods, whereas a high UTD ratio degrades performance under 020 corruption,
except in the case of action corruption. Furthermore, policy extraction methods 17
39 16 play
critical role under state attack scenarios, extracting policies using AlignIQI. [17 Bg. 8 yields superior performance under state attacks.
म (als) x p(als) exp -V())
(8)
A detailed ablation study investigating the impact of normalization, UTD, and policy extraction is provided in Sections 7.2 and Appendix C.2
The overall RPEX algorithm is summarized below, with its pseudocode provided in Algorithm [ First, RPEX inherits the offline pretrained policy, obtained via IQL or RIQI. on the attacked offline dataset, and initializes online policy. It then fixes the offline pretrained policy and employs

---

Eq. (6) to collect online samples during the attacked online phase. Finally, RPEX trains the new online policy using buffer that integrates offline and online data via IQL, RIQL, or other offline or online learning methods.
Algorithm RPEX: Robust Policy EXpansion
Input: offline RL algorithm IQL or RIQL O offline Toffinee
La
online
RL algorithm
1 }2 (T 'nnline "online]
Initialize: UTD= M, network parameters Φ, B, 6, corrupted offline replay buffer D Domline Normalize the states in both the environment and the corrupted offline replay buffer Doftine while in offline training phase do
% offline policy training using batches from the corrupted offline replay bufter Dottine & offtine (o), B Ax V offtine (5)
Φ Φ dQ V
end while Policy Expansion: == [ mol: transfer Q while in online training phase do for each environment step do
#(a, (a, according to (Eq. (
8t-11 P88441 [8t, ar),
Attack {(8t, (1,
at, T, 8c+1]}
% Add corrupted transition into online buffer D
D - DU {(S,, f,, 8,+1))
end for for each gradient step do
% online training using batches from both Dollline and D
é - Ø Xa Lo (6), A с 0
- X V L (0) for M times
high UTD for action corruption
end for end while
Theoretical Analysis
Proposition 6.1. Given P aa a2). where ai ~ T3,02 ~ Eg. objective (See proof in Appendix B.I
maximizes the following
Qo - -V ie (a,ass))
Barwze 0g
Σ Pu(iis, (Qols, oi) log Pw (w( kn
Pw (ils,,a,,
Max Rewaar & Entropy
Regularization
(9)
The first term corresponds to the standard objective in maximum entropy RL regularization term, we provide the following justification.
For the
Justification of Regularization. This regularization term be interpreted rectification mechanism. As discussed in Section the heavy-tailed distributions induced by attacks increase the likelihood of assigning nonzero probabilities "bad" actions, characterized by Q(s, Vs) The regularization term penalizes such actions-e.2 e.g.. aj when Q V < Q-by reducing the probability of their selection, i.c., decreasing (w 01, a2). The magnitude of this penalization is modulated by and x, (a,]s); specifically, the smaller (a; (s) is, the stronger the penalization, since low action probability coupled with a poor Q value (Q- V 0) often results from corruption. Conversely, good actions (Q V > 0) are encouraged, but controlled by our clip term in Eq. (7) to avoid placing too many weights on suboptimal actions.
2 L denote the loss functions of the critic and actor, respectively, actor cattor-ritic-baate reinforcement learning. In our implementation. both the offline and online reinforcement learning algorithms are either IQL or RIQL.

---

0.3, the online Table Average normalized Offline-to to- to-nniine score under random data corruption on the Medium- Replay Tasks over 5 random seeds.
Environment
Attack Element
IQL
IQL-PEX
I0L-RPEX (ours)
RIQL
RIOL-PEX
RPEX (ours)
doservation
21.4-22.7
21,4-+21 San.
21.4-21.:218
19.73-21 36432
19.73-20.99as3
19,73-22.55 22.5
Halfcheetah-MR
action
42.9-+18.44
41,9-444,5114
42,9-6559214
419--47.0t07
42.9-692.s0
41.9-52.7:05
43.--49
43.5-4999505
43.6-467.1
43.5-470.2500
43.6-778.6e40
03.5
dynamies
-7.1-13
37.1-3370044
37.1-339941
42.0-45.300 Selv
403 42.0-144.7
47.0-444.4m0
aiservation
87 -17
8.7 -2018-5
K77 -23---3
52. 17231
12.2 -25.632
12.2-2300 7,015 -5.
Walker2d MR
action
64.7 1166.ams
64.7 1105,
77.2-90.1
64.7 1077-1100 4ans
71.2-94,5aa3
85.9 348.8
818--91 Jasa
85.9 100- 24st
818->91-9a1.1
8188-100.5522
85.9
118.99990
dynamics
14.9-66822.5
14.9-4477e11
80.0-189.5a1.
80.0-192.2a14
observation
753 -30, lanli
15.8 (45/4an3
15.8 776.9s
78.3-129 Zana
78.3-373.1M0
HoR
action
93,4-195.0a44
93.4-11022 2an.4
934-116.2.577
75.9-995 Sat
05.9-93322 2a303
25.9 15.9-1112.6as
reward
55,2-197.8112
55.2-199.8
15,2-102.621.
72,9 3e1.9
12.9-490.12 122.
72,9-1100.0
dynamics
08-46.06755
0.8-13,4-09
44.6-451,2436
41.6-45100s
41.6-45.22s1
Average offine soore
44.5
44.3
44.3
38.4
55.4
Average 020 sooee 7
50.3
53.5
59.1
$4,9
66.6
75.16
Average improvement percentage
13.1%
20.3%
32,7%
-6.1%
14.1%
28.7%
Overall, this regularization term supports our explanation in Section regarding how Eq. 6 enhances exploration efficiency.
7 Experiments
In this section, we empirically assess the Offline-to-Online performance of RPEX under various data corruption scenarios.
Experimental Setup. Following RIQL. [50], use the corruption rates c, C2 [0, 1] and the corruption scale to control the overall level of data corruption. The offline pre-trained policy is obtained by injecting random noise into the cornupted elements of Cj fraction the offline dataset. In the online phase, each trajectory is corrupted with probability by either random or adversarial noise. To better reflect real-world conditions, we adopt distinct corruption rates for the offline and online phases. In the main experiments, the offline corruption rate is set to corruption rate to 02 0.5, and the corruption scale Online score in Table ! we report the mean results and standard deviation based on the final three evaluations averaged over five random seeds. Additional experimental details and implementation details are provided in Appendix
for both phases. For the Offline-to-
C
Baselines. For the offline pre-trained policy, we train IQL. 25 and RIQL 50) with 2e6 gradient steps under randomly corrupted elements. We adopt IQL and RIQL as offline pre-trained policies because IQL is one of the most widely used offline RL baselines and has been shown to be robust to various data corruption scenarios [50]. while RIQL is a robust variant of IQL. For the Offline-t-Online algorithms, we compare IQL, PEX [53] IQL+RPEX (ours). RIQL, RIQL+PEX, and RIQL+PREX (referred to as RPEX, ours). IQL and RIQL denote directly applying the respective algorithms in the Offline-to toine-to-Onliee Omline setting without modification. Additional details about the baselines are provided in Appendix
7.1 Main Results
Random Corruption. The main results are obtained on the Medium-Replay tasks 110. which were collected during the training phase of a SAC agent and more closely resemble real-world conditions. Table Pableetsesnnts the average normalized performance of various algorithms. Overall, RPEX yields greater performance improvements compared to Vanilla IQL (RIQL) and IQL+PEX (RIQL+PEX). achieving average score gains of 32.7% over IQL and 28.7% over RIQL. In 11 out of 12 settings involving RIQL-based methods, RPEX achieves the highest performance, particularly under state, action, and reward corruption, where significantly outperforms the baselines.
Additional experiments on mixed attacks, adversarial attacks, and AntMaze tasks are presented the Appendix
7.2
Ablation Study
UTD. The updates-to-data (UTD) ratio refers to the number of updates performed per online trajectory. A high UTD generally accelerates policy learning in uncorrupted Offline-to-nllin scenarios [56).

---

RPEX
RIQL-PX
705
06
60
40 43
10-
UTD
Figure 4: The effect of UTD for 020 methods. The results averaged over the random seeds on the Medium Replay tasks.
However, under corruption, a high UTD primarily improves robustness against action corruption and slightly degrades overall performance. This is likely because the policy update in IQL--ttle methods resembles supervised learning-Fa, Eq. (3) is akin to cross-entropy loss with softmax. Such an update mechanism inherits the robustness properties of supervised learning [8) Consequently, high UTD benefits from more gradient steps and mitigates degradation caused by noise under action corruption. We also conduct additional experiments in Appendix C.2 where high UTD is applied only to the critic while the actor is kept at the standard setting (UTD=1), to further validate this phenomenon.
Policy Type & Normalization & hyperparameter k As noted Yang et al. [50]. IQL or RIQL with deterministic policy and state normalization generally achieves better performance under data corruption. this section, we examine the effects policy type (deterministic or stochastic) and normalization during the 020 period using Hopper MR under action and dynamics attacks.
As shown in Figure unlike in the offline setting, deterministic policy can degrade 020 perfor- mance. This is because exploration in deterministic policies is typically achieved by adding small noise to actions, which may lead inefficient exploration. In contrast, applying state normalization significantly enhances 020 performance. Furthermore, the ablation study on in Figure SaareSSjnniii indicates that RPEX is relatively insensitive to K, although fine-tuning this parameter can still yield marginal improvements. Note that all results in Table I are reported using 0.1 and inverse temperature 3. A detailed ablation study on offline buffer retention, clipping range, and the policy extraction method presented in Appendix C.2 In addition, Appendix [.33 reports the training cost of our method and the evaluation of RPEX under varying levels data corruption.
320
54.2
10.2
Norma Setum
$1.
Narma Senturn
93.3
Normil Serumm
32
20
10
Dete Determinestc
Normaize States
Figure 5: Ablation study the main components of RPEX on Hopper MR task. From left to right: policy type ablation, state normalization ablation, and hyperparameter ablation.
8 Conclusion and Limitations
In this paper, we propose RPEX, method designed enable robust online fine-tuning under 020 data corruption by leveraging importance sampling (IPW). This setting is critical for deploying

---

reinforcement learning in real-worke environments, where noise and adversarial attacks are ubiquitous. RPEX integrates IPW into PEX to mitigate the heavy-tailed behavior of policies induced by corrupted data. Through empirical evaluations under random, adversarial, and mixed attacks, we demonstrate the robustness RPEX against various forms of data corruption during the 020 phase. We hope this work inspires future research addressing data corruption in more realistic scenarios. A limitation of our approach is that IPW in RPEX requires access to the exact action probabilities, which may hinder integration with probability-agnoste models such diffusion models.
Acknowledgments and Disclosure of Funding
This work was supported by the Natural Science Foundation of Shenzhen (NoJJCY1022080711 1000230007116000006 1604008, dation of Guangdong Province No.JCY1202408131 1000101000001, the Natural Science Foun- National Key Research and Development
114111510000 Program of China (No. N0o22022YFB7011400) and Cross-disciplin Fund for Research and Innovation (No. JC2024002) of Tsinghua SIGS, Li Shen is supported by NSFC Grant (No. 62576364), Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (NO, 1CY120241102212430041 CCF-Tencen Rhino-Bire Open Research Fund (NO, CCF-Tencent RAGR202501 14) and Tencent JR2025TEG002.
References
[1] Anurag Ajay. Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. 2022. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations.
[2] Gaon An, Seungyong Moon, Jang- Hyun Kim, and Hyun Oh Song. 2021. Uncertainty- offline reinforcement learning with diversified q-ensemble Preprint, arXiv:2110.01548.
[3] Philip J Ball, Laura Smith, lya Kostrikov, and Sergey Levine. 2023. Efficient online rein- forcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594. PMLR.
[4] Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. 2023. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. Advances in Neural Information Processing Systems, 36:66845-66852
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Hersert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot
learners Preprint, arXiv:2005. 141.5500..1415...
[6] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 15084 15097.
[7] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268-9277.
[8] Wenqi Dang. Zhou Yang, Weisheng Dong, Xin Li, and Guangming Shi. 2024. Inverse weight- balancing for deep long-tailed learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11713-11721.
[9] DeepSeek-AL Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. DDepssek-ri:: In- centivizing reasoning capability in 1lms via reinforcement learning Preprint, aXXiv:2501.12948.
10

---

[10] Justin Fu, Aviral Kumar. Ofrr Nachum, George Tucker, and Sergey Levine. 2020. D4rl: Datasets for deep data-driven reinforcement learning arXiv preprint SxXi::200.0072144
[11] Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning. pages 1587-1596.
PMLR.
[12] Scott Fujimoto, David Meger, and Doina Precup. 2019, Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062.
PMLR.
[13] Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Adarsh Prasad, Zico Kolter, Zachary Lipton, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Pradeep Ravikumar. 2021. On proximal policy optimization's is heavy-tailed gradients. In International Conference on Machine Learning. pages 3610-3619. PMLR.
[14] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR.
[15] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, and Pieter Abbeel. 2018. (Soft actor-critic algo rithms and applications arXiv preprint arXiv: 18/2.0595.
[16] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. 2023. [dql: Implicit q-learning actor-critic method with diffusion policies arXiv preprint ar Xiv: 2304.
[17] Longxiang He, Li Shen, Junbo Tan, and Xueqian Wang. 2024. Aligniql: Policy alignment in implicit g-laarning through constrained optimization Preprint, 18X11:2405.81187
[18] Jifeng Hu, Sili Huang, Li Shen, Zhejian Yang, Shengchao Hu, Shisong Tang, Hechang Chen, Yi Chang, Dacheng Tao, and Lichao Sun. 2024. Solving continual offline through selective weights activation on aligned spaces. arXiv preprint arXiv:2410.15692
[19] Jifeng Hu, Sili Huang, Zhejian Yang, Shengchao Hu, Li Shen, Hechang Chen, Lichao Sun, Yi Chang, and Dacheng Tao. 2025. Analytic energy-guided policy optimization for offline reinforcement learning, arXiv preprint arXiv:2505.01182.
[20] Peter J. Huber. 1973. Robust Regression: Asymptotics, Conjectures and Monte Carlo The Annals of Statisties, 155)7999 821.
[21] Peter Huber. 1992. Robust estimation of location parameter In Samuel Kotz and Norman Johnson, editors, Breakthroughs in Statistics: Methodology and Distribution, pages 492-518. Springer New York.
[22] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. 2022. Planning with difflusion for flexible behavior synthesis arXiv preprint arXiv:2205.0999)
[23) Samir Khan and Johan Ugander. 2021. urXiv:2106.06695.
Adaptive normalization for ipw estimation
Preprint,
[24] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.690.
[25] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. 2021. Offline reinforcement learning with fisher divergence critic regularization. In International Conference Machine Learning, pages 5774-5783. PMLR.
[26] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2021. Offline reinforcement learning with implicit g-laarning. In International Conference on Learning Representations.
[27] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 179- 1191.
11

---

[28] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review and perspectives on open problems arXiv preprint arXiv:2005.01643
[29]
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. 2023. Adaptdiffuser: Diffusion models as adaptive self-evolving planners arXiv preprint 2xXiv2230..11877
[30] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning arXiv preprint wrXiv:1509.0297)
[31] Xu-Hui Liu, Tian-Shuo Liu, Shengyi Jiang, Ruifeng Chen, Zhilong Zhang, Xinwei Chen, and Yang Yu. 2024. Energy-guided diffusion sampling for offline-to-olline reinforcement learning Preprint, arXiv:2407.12448.
[32] Xiaoteng Ma, Yigin Yang, Hao Hu, Qihan Liu, Jun Yang. Chongjie Zhang, Qianchuan Zhao, and Bin Liang. 2021. Offline reinforcement learning with value-based episodic memory. arXiv preprint 1r..v:2110.09796
[33] Kanti V Mardia. 1970. Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3):519-530
[34] Trevor McInroe, Adam Jelley, Stefano V Albrecht, and Amos Storkey. 2023, Planning to go out-of-distribution in offline-to-onling reinforcement learning. arXiv preprint arXiv:2310.05723.
[35] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. 2020. Awac: Accelerating online reinforcement learning with offline datasets) arXiv preprint arXiv:2006.099: 09359.
[36] Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. 2023. Cal-QL: Calibrated offline RL pre- traininnning for efficient online fine-tuning In Workshop on Reincarnating Reinforcement Learning at ICLR 2023.
[37] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Balteseu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. Gpt-4 technical report Preprint, arXiv:2303.08774.
[38] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. 2022. Robust reinforcement learning using offline data. In Advances in Neural Information Processing Systems.
[39] Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. 2024. Is value learning really the main bottleneck in offline rl? Preprint, arXiv:2406.09329.
[40] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. 2019. weighted regression: Simple and scalable off-policy reinforcement learning arXiv preprint arXiv:1910.001
Advantage
[41] Jan Peters, Katharina Mulling, and Yasemin Altun. 2010. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1607-1612.
[42] James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. 1994. Estimation of regression coefficients when some regressors are not always observed, Journal of the American Statistical Association, 89(47):8446 8666.
[43] Laixi Shi and Yuejie Chi. 2024. Distributionally robust madel-based offline reinforcement learn- ing with opar-altimal sample complexity. Journal of Machine Learning Research, 25(200):1-91.
[44] Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, Matthieu Geist, and Yuejie Chi. 2023. The curious price of distributional robustness in reinforcement learning with generative model In seirth- Conference on Neural Information Processing Systems.
[45] Richard Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT press.
12

---

[46] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Coumaaeaa,, Evgeni Burovski, Pearu Peterson, Warren Weekesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, and 16 others. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python Nature Methods, 17:261-272.
[47] Jingkang Wang, Yang Liu, and Bo Li. 2020. Reinforcement learning with perturbed rewards Preprint, arXiv: 181.0100..
[48] Jiawei Xu, Rui Yang. Shuang Qiu, Feng Luo, Meng Fang, Baoxiang Wang, and Lei Han. 2025. Tackling data corruption in offline reinforcement learning via sequence modeling Preprini, arXiv:2407.04285.
[49] Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. 2022. RORL: Robust offline reinforcement learning via conservative smoothing, In Advances in Neural Information Processing Systems.
[50] Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. 2023. Towards robust offline reinforcement learning under diverse data corruption. arXiv preprint arXiv:2310. 12955.
[51] Chenlu Ye, Jiafan He, Quanquan Gu, and Tong Zhang. 2024. Towards robust model-based reinforcement learning against adversarial corruption Preprint, arXiv:2402.08991.
[52] Chongsheng Zhang, George Almpanidis, Gaojuan Fan, Binquan Deng, Yanbo Zhang, Ji Liu, Aouaidjia Kamel, Paolo Soda, and João Gama. 2025. A systematic review on long-tailed learning. IEEE Transactions on Neural Networks and Learning Systems.
[53] Haichao Zhang, Weiwen Xu, and Haonan Yu. 2023. Policy expansion for bridging offline-to- online reinforcement learning ArXiv, abs/2302.0099 00935.
[54] Xuezhou Zhang, Yiding Chen, Jerry Zhu, and Wen Sun. 2021. Corruption-robust offline reinforcement learning Preprinr, arXiv:2106.06630.
[55] Kai Zhao, Jianye Hao, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. 2023. Enoto: improv- ing offline-to-online reinforcement learning with q-ensembles. arXiv preprint arXiv:2306.0687/
[56] Zhiyuan Zhou, Andy Peng, Qiyang Li. Sergey Levine, and Aviral Kumar. 2025. Efficient online reinforcement learning fine-tuning need not retain offline data In The Thirteenth International Conference on Learning Representations.
13

---

Border Impact. Pefline-to-Online reinforcement learning (020 RL) aims to learn policy from fixed offline dataset, followed by fine-tuning through online interaction. Our approach focuses on enhancing the robustness of 020 algorithms under data corruption occurring in both offline and online phases. Such corruption ubiquitous due to noise and potential adversarial attacks. By improving robustness, our method advances the field of 020 RI and facilitates its deployment in real-world applications such as robotic control, without introducing significant ethical or societal concerns.
Index of the Appendix
In the following, we briefly recap the contents of the Appendix.
Appendix A presents further discussion related work and additional background information. Appendix B reports all proofs and derivations.
Appendix reports additional experiments and ablation studies about our methods, along with some implementation details.
Appendix D reports further discussion on the connection between the heavy-tailedness of the Q P-ttaree distribution and that of the learned policy.
A Related Works
IPW in Heavy-tailedness Classification tasks. Inverse probability weighting (IPW) [42 is general methodology survey sampling and causal inference [23], and is also widely employed to mitigate heavy-tailedness in classification tasks 7 52 In addressing heavy-tailed classification problems, IPW is typically applied as a weighting mechanism for the classification loss. For example, Cui et al. 77.. Dang et al. [8] reweight the loss of sample inversely proportional to the number of instances in its class. this study, we incorporate IPW into the O20 reinforcement learning framework to address the heavy-tailed nature of policies under various data corruptions, and we demonstrate its significant effectiveness.
Offline RL. Offline RL algorithms need to avoid OOD actions. Previous methods to mitigate this issue under the model-free offline RI. setting generally fall into three categories: 1) value function-based approaches, which implement pessimistic value estimation by assigning low values avoid the use of out- of-distribution actions 2) sequential modeling approaches, which casts offline RL as sequence generation task with return guidance (6] 22 29 and 3) constrained policy search (CPS) approaches, which regularizes the discrepancy between the learned policy and behavior policy [41 40
to dittr--disttibution actions [27 12], or implicit TD backups (26 32
35
19
Kurtosis Value. For random variable X with mean j and std o, metric Kurtosis
13
33
calculated by Kurt[X] = E (x=x)
41 { which measures the heavy-tailedness compared to
normal distribution. (Small Kurtosis means less heavy-tailedness) The excess kurtosis defined as kurtosis minus 3. High values of Kurt arise in two circumstances: (1) where the probability mass is concentrated around the mean and the data-generating process produces occasional values from the mean, (2) where the probability mass is concentrated in the tails of the distribution. (Note that the kurtosis reported in this paper is computed using SciPy [46), which, in fact, calculates the excess kurtosis.)
AlignIQL. AlignIQL 17 aims to find the implicit policy in the learned value function of IQL. 25 To find the implicit policy, He et al. [17 define such implicit policy finding problem (IPF)
((als)
min E (sdd
(
s.t. <(ass) 20
Vs, Va
(IPF)
/ (aals)da= 1 Vs
E Ea-a(a)a) 1Q(s, V(s) As.
14

---

where V(ss), ((s)) a) is the learned value function, which does not have to be the optimal value function. f()) is regularization function which aims to avoid aut-of-distribution actions. Further He et al. 17 consider the relax formulation of Eq. (IPF)
f m(aas) p(aas)
+ (Q( -) V(s))
min E sndd (a)
n. (e)(e) a~a((
s.t.
< 8) 20,
Vs, Va
(IPF-Soft)
J. n ( s)da Vs.
and derive the formulation of implicit policy (suppose that f(x) log
x (als) p(als) exp (n - v(())))
(10)
B Proofs and Derivations
In this section, we present a detailed derivation and justification of our method, along with the proofs of the main theoretical results.
B.1 Derivation and Theoretical Justification of RPEX
Following Zhang et al. [53] let K 2, the compositional policy स can be rewritten as
a(als) = /
Tp (a (TTT(()ss)) E Pw(ils a1, 02)0(a = a,)da,da2,
(11)
01
where d is the Dirac delta function, Pw is a discrete policy with action dimension of two. For V-value, we can have the following derivations (Zhang et 53 Appendix A.3)
v(s) J ((sls)
(12)
J Tp(ails)To(a)) [
Σ (i),, a;)aa Qo a))a
[ Ts(ails)To(a211>>
Pw (i)s, 02) [/ ō(a a.) da da)dda((4)
Σ
[ Ts(ails)To(
Pw (ils, a1 (s) dd),, Σ
(15)
Ja,02
E 07
Σ Euu((ssanna)00asss..)
(16)
where dj Tg,02 mg.
By incorporating an entropy regularization term to promote online exploration, we obtain
Σ (ils, a1, az) az) (Qo(s, a:) log Pw (1,
Bg a
(17)
By adding regularization term
Qo V Pw (i))s, 11 n, (a.1s)
(18)
where Kn is the coefficient. m((a.(s) is the probability that action a; selected by the corresponding policy ny. denotes the offline policy ny and TO denotes the online policy πe-
15

---

We obtain
Qo -V ((alss)
Bai aa
Σ Pu(ls,, az) (Qo(s,a,) a log Pw (w
R1
Pw (wliss,a
(19)
Solving Pw by setting the gradient of Eq. 19 to 0 with respect to Pw given n3, no, we have
(wliss, a1 02) a exp (s(s, xi/a (a(sss)
Qo )
(20)
Finally, let K k1/a, have (for simplicity, we omit the subscripts and 1b.)
Q-V mita.s)
Pw Σ; ex)((((s.a)))/)
vi [1, ,
(21)
Q-V m
which is the Eq. (6)
C
Implementation Details and Additional Experiments
In this section, introduce the implementation details for reproducing results and some extra experiments to validate our method.
C.1 Implementation Details
Evaluation. Throughout this paper, we report mean scores and standard deviation averaged over the last three evaluations across different random seeds. Each evaluation consists of 10 episodes.
To obtain the offline checkpoints for RIQL and IQL, we rerun the official code released by Yang et al. 50 with default configuration parameters. The offline policy and value function are trained for 2e6 gradient steps, while the 020 update phase consists of leб steps. For O20 results, we use the official code from Zhang al. [53] implement our RIQL+PREX and IQL+RPEX methods. We adopt the codebases of Zhang et al. 153 and Yang et al. [50] to implement the baseline methods. For RIQL- based 020 implementations, following Yang et al. [50), we employ the Huber loss and Q quantile estimators. The policy and value networks for both IQL and RIQL are implemented as multilayer perceptrons (MIPs) with two hidden layers, each comprising 256 units and ReI.U activations. For Q-network in RIQL-based methods, we follow RIQL and adopt the ensemble implementation of EDAC [2]. For RPEX with a deterministic offline policy, we parameterize the online policy within the policy set 1I using stochastic Gaussian policy and assign the IPW weight of the online policy to the offline deterministic policy. These neural networks are updated using the Adam (24). The hyperparameters used to reproduce results in Table are listed in Table C.1
The following section provides details on the baseline implementations.
State Normalization. Following Yang et al. (50], we normalize each state in the corrupted offline dataset and environment using 1oo and o.. which denote the mean and standard deviation of all states and next states in the corrupted offline replay buffer Dotttiee Based on 14o and oo. the states and next-states are normalized as 8i of 2N EN [(s; (2o)2 + (8/ pp
8 = -- where Ho = 2N Σn (s: + s() and
do
RIQL-direct. For RIQL, we directly the offline pretrained policy and critic for online learning. Accordingly, the offline policy and critics continue be updated during the online phase with additional samples collected by the offline policy. We adopt the offline parameters reported by Yang et al. (50] and apply the same settings during online training. For the results shown in Table we sweep over UTD values {1,2, 4} and report the best outcome.
RIQL-PEX. For RIQL-PEX, the critic and policy updates follow the same procedure as RIQL, except that online samples are collected using composited policy (Eq. (5)). Following PEX 53 we freeze the offline policy and train new policy using both offline and online buffers. For evaluation, consistent with Zhang et al. (53), greedily select the action with the highest weight according to
16

---

For the results in Table we sweep over UTD values (1, and a E f3, 10, 100). and report the best performance.
Eq. (5.
IQL. For IQL. we also directly use the offline pretrained policy and critic for online learning. The offline policy and critics are further updated during the online phase with samples collected by the offline policy. As with RIQL, we sweep over UTD values {1, 2, 4} and report the best result presented in Table
IQL-PEX. For IQL-PEX, the critic and policy updates follow the same scheme as RIQL, with online samples collected using the composited policy (Eq. (5).. Following PEX [53], we freeze the offline policy and train a new policy with both offline and online data. For evaluation, we greedily select the highest-weighted action according to Eq. ( consistent with Zhang al. [53] As with RIQL-PEX, we sweep over UTD values 21,2,4 4] and a E (3, 10, 100). and report the best performance.
Data Corruption Details. Following Yang et al. [50], we apply both random and adversarial corruption to the four components: states, actions, rewards, and dynamics (i.e., next states). The overall corruption level is governed by four parameters: C1. C22 E1, and é2, where eı denotes the corruption rate within an offline dataset of size N, C2 represents the corruption probability of online transitions, e1 specifies the offline corruption scale per dimension, and ez specifies the online corruption scale per dimension. Our experiments primarily focus on the imedium-replay" datasets introduced by Fu et al. [10], which are collected during the training of a Soft Actor-Critic (SAC) agent and are thus more representative of real-world scenarios. Unless otherwise specified, we set <1 0.3, 02 0.5, and 61 = 62 1. Below, we describe four types of random data corruption, as well as a mixed corruption strategy.
Random observation attack: For offline attacks, we randomly sample e₁ N transitions (s, a, 8') and corrupt the states as å = 8 + X std(s), where À ~ Uniform(-e,de cjd Here, d, denotes the dimensionality of the state, and std( is the dy-dimensional standard deviation of all states in the offline dataset. The noise is independently added to each dimension and scaled by the corresponding standard deviation. For online observation attacks, we randomly corrupt online s with probability c2. If state normalization is enabled, we set std = 1; otherwise, we use the standard deviation computed from the offline dataset.
Random action attack: For offline attacks, randomly select c₁ -N transitions (s, s') and corrupt the action as à a+ (td) where À ~ Uniformi-e, e] the dimensionality of the action, and std(a) is the dimdiemnso standard deviation of all actions in the offline dataset. For online action attacks, we randomly corrupt with probability c22 using std(a) computed from the offline dataset.
e Here, da denotes
Random reward attack: For offline attacks, we randomly sample e1 -N transitions -ammmm30-m
(s, a, r, 8') from D and corrupt the reward as f ~ Uniform[-3 30 . e, 30 ce. For online
reward attacks, we randomly corrupt with probability 02 using the same corruption method
as in the offline case.
For offline attacks, (s, a, 8') and corrupt the next state as 8' = s' + A std()'), where À ~ Uniform[-e (- e| ejd Here, de denotes the dimensionality of the state, and std(s') the dy-dimensional standard deviation of all next states in the offline dataset. For online observation attacks, we randomly corrupt the online next state s' with probability c2. If state normalization is enabled, we set std(s') = 1; otherwise, we use the standard deviation computed from the offline dataset. Random mixed attack: The offline dataset corrupted using random dynamics attacks. During the online phase, we randomly corrupt and s' with probability c2, applying the same corruption method as described previously.
Random dynamics attack:
randomly sample C₁ N transitions
In addition, we apply online adversarial dynamics attacks:
Adversarial dynamics attack: Following Yang et al. (50], we use pretrained EDAC 2
agent consisting of a set of Qp functions and a policy function Ty. We then randomly corrupt s' with probability Cg and modify the next state as s' = arg migns"ee(ss...)) Qp o. (5 ))
where Ba(s', <) (8' 8 - s'1 S std($ (s')) The Q function in the objective is defined as the average over the Q functions EDAC. We implement the optimization using Projected initialize learnable
Gradient Descent, following prior work [50, [54). Specifically,
vector z E --E, ejdd, and perform two steps of gradient descent with a step size of 0. on
8' = 8' + std(s'). After each update, we clip each dimension of z to the range [-e,e]
17

---

Toy Experiments Details. For the toy maze experiment, we train IQL and IQL+IPW using the offline dataset shown Figure((a) The reward defined as the negative Euclidean distance between the current position and the goal. The action space consists of eight movement directions, and the state is represented by the current coordinates of the agent. In the corrupted setting, random noise is added to the dataset states with a corruption range of 1 and corruption rate of 0.5.
LR (For all networks)
3e-4
Critic Batch Size
256
Actor Batch Size
256
Discount
0.99
Offline Batch Ratio
0.5
Expectiles for IQL or RIQL
0.7 (MuJoCo)
Initial Collection Steps
5000
Target Update Speed
5e-3
IPW Weight x
0.1 (except mixed attack)
IPW Weight
0.3 (mixed attack)
Inverse Temperature -!
(except mixed attack)
Inverse Temperature a -1
10 (mixed attack)
UTD for RPEX
4 for action corruption 1 for others
Policy Extraction for RPEX
Eq.
for state attack
Policy Extraction for RPEX
Eq. 3
(except state attack)
for AlignIQL
3
State Normalization
True
Number of Offline Iterations
2e6
Number of Online Iterations
1e6
Optimizer
Adam 24
C.2
Ablation Study
RIQL
RIQU-PE -EX
RPEX
03
Kommeitos tocd Roturn
Strmatbed
63
600
600
Figure 7: Offline buffer Ablation.
Figure 6: Policy Extraction Ablation.
High UTD on Critic We conduct experiments in which only the critic is trained with high update-to-data (UTD) ratio, while the actor maintains UTD= 1, on the Walker2d-MR tasks under dynamics and action corruption. All other hyperparameters remain the same as those used in our main paper (Table C. Table shows the results of RPEX with such UTD implementation. All results averaged over 10 random seeds.
18

---

Table Results under High UTD for the Critic.
UTD
Dynamics
Action
1
80 - 92.2 #1A
85.9 + 118,9 #10.1
80 > 92.6 210.0
85.9 + 96.7 221.8
As shown in Table [ the issue discussed in Section 7.2 namely, that high UTD primarily improves robustness against action corruption while slightly degrading overall performance can be alleviated. However, observe that although such UTD implementations mance, they also increase overall variance. This phenomenon may be attributed to the fact that data corruption introduces greater Bellman backup errors compared the non-corrupted setting. While high UTD ratio for the critic can accelerate learning, the subsequent Bellman backups may fluctuate significantly due to corrupted data.
sometimes yield better perfor-
Policy Extraction Methods In this section, we investigate the effect of policy extraction in our method. Since IQL-style methods follow an actor-critic decoupled paradigm, different policy extrac- tion methods can significantly affect final performance [17 [3 [6].
As shown in F extracting the policy using AlignIQL 17 significantly improves robustness under state observation corruption, whereas in other scenarios, the vanilla AWR method yields better performance.
Buffer In this part, we investigate the impact of retaining offline datasets during online fine-tuning. As noted Zhou et al. [56). most reinforcement learning fine-tuning methods rely on continued access offline data ensure stability and performance. As shown in Fig, / the presence absence of offline data during the fine-tuning phase has minimal impact on our method, demonstrating its robustness.
Range of Clip For range of clip, we conduct ablation study on the Walker2d-MR environment under dynamics and action corruption. The reason to choose dynamics and action corruption that they represent two modular 8, in the MDP. As shown in RIQL, dynamics corruption is known as difficulty for robust RL. The results of different clip ranges on the Walker2d-MR are shown in the
Table 3
Note that the = 0.1, a 3. corruption range (1) and rate (0.5) are the same as those in
Table
All results are averaged over 5 random seeds.
Table 3: Ablation Study of the Clip Range.
Clip Range
Dynamics
Action
(-10000, 100)
80 + 92.2 a1.4
85.9 + 118.9
(-100, 100)
80 > 90.8 +30.0
85.9 + 108.4 =1.07
(0, IO0)
80 + 62.1 238.6
85.9 - 62.67 244.
(-10000, 10000)
80 + 96.7 21.69
85.9 + 105.53 20.09
As shown in Table the performance differences across various clipping ranges are minor, except for extreme cases such the range (0, 100). Similar к, fine-tuning the clipping range may further improve performance. Throughout our paper, we consistently use the same values for k and the clipping range (-10000, 100), which further demonstrates the robustness of RPEX.
C.3 Additional Experiments
Training Time
We report the average epoch time for the Hopper- task measure of com-
putational cost in Table [T. All experiments were conducted on an NVIDIA RTX 4090 24GB GPU with an Intel(R) Xeon(R) Platinum 8358P CPU. As shown in Table 4 RIQL 50 incurs additional computational overhead compared to IQL due to the Huber loss and quantile Q estimators. PEX (53) also requires more computation than IQL as it involves training an additional online policy. RPEX introduces further overhead relative to PEX owing to the added computation of IPW. Nevertheless, despite achieving performance improvement of approximately 13.6% over PEX, RPEX increases
19

---

Algorithm Performance Across Different Attack Types
120
Algorithm
RPEX
100
RIQL PEX
70.3
Normalized
60
46.5
41-2
mixed
adversarial
Attack Type
Figure 8: Results under random mixed and adversarial attacks averaged over random seeds.
training time by only about 8.1%. These results indicate that our method improves performance without incurring substantial computational costs.
Table 4: Average Training Cost.
IQL
IQL-PEX
IQL-RPEX (ours)
RIQL RIQL-PEX
RIQL-RPEX (ours)
17.420.0
30.9+3.2
31.7216
22.721.3
33.1
37.6-1.
Adversarial Corruption & Mixed Corruption Figure presents the performance of RIQL, RIQL-PEX, and RPEX (ours) under random mixed and adversarial attacks in Hopper-MR tasks. For random mixed attacks, the offline policy trained with random dynamics attack, followed by online random dynamics and reward attacks. For adversarial attacks, the offline policy is trained with a random dynamics attack, followed by online adversarial dynamics attacks. For RIQL, we sweep from UTDE {1,4) and report the best results. For RIQL-PEX,
perform hyperparameter sweep over UTD E 41,4] and inverse temperature a- E {3, 10, 100), and report the best results. For our method, sweep over (0.1,0.3) and a E (3, 10).
As shown in Figure RPEX outperforms other methods under both mixed and adversarial attack settings.
AntMaze Tasks We conducted additional experiments the AntMaze-large and AntMaze-diverse tasks, which are the most challenging tasks in the AntMaze tasks. We primarily focus on dynamics and observation corruption, as these represent the most difficult corruption settings.
Why are observation and dynamics corruptions particularly challenging? As shown in Table all algorithms struggle to improve under observation and dynamics attacks. In particular, under observation attack, RIQL-PEX exhibits a performance drop of approximately task. Although RPEX significantly outperforms RIQL-PEX, it still struggles to improve performance. similar phenomenon is also reported in RIQL [50].
on the Hopper-MR
The primary cause of poor performance under observation attack is its strong disruption to online exploration. The key objective during the online phase is collect optimal trajectories and update both the policy and the value function. When observation attacks occur, they affect action selection, potentially resulting poor even
(OOD) actions. Moreover, observation attacks can distort the Bellman backup, thereby hindering value function learning. According to Theorem 3 in RIQL, such suboptimal or OOD actions, combined with high Bellman backup error, lead to a loose upper bound on the difference between value functions learned under clean and corrupted observations. This explains why improving performance under observation attacks particularly difficult.
20

---

Hesparrftaduum-a Reward
Wediimdeeelllee wedaluumlllplyy.. Dynamics
Hopper
Medium
Obs
Horalord
5
40-
146
Stripe
Hallcheetah- Mel---edd-----m Replay-122
Walfchhetah--Medle Regiliap//2
Hatfichhed www.eatdddddmemmmeeeeemmmm
50
Www 2o1
62
20-
30
10
0.0
0.2
0.4
3.4
1.5
0.0
0.2
0.5
9.35
Waltker Mevilam
Walker deediimm Dynamics
Wolker2D-Melle Mewiiii---edi----eee
WalkerDD Mediume
Reward
Acts
obs
100
Naraa D2d TELuT
Step>
166
104
Figure 9: Training curves of RIQL-direct, RIQL-PEX (RIQL), and RPEX (ours) under random data corruption on the "medium-replay" tasks (Table I.
For dynamics attacks, the limited improvement after the 020 phase primarily stems from Bellman backup errors induced by the attack. Although RIQL reduces Bellman backup error by mitigating the heavy-tailedness of the Q-target, attacks the dynamics during the online phase still distort the Bellman backup, thereby limiting performance improvement.
Given the difficulty of observation and dynamics corruptions, together with the sparse-reward challenge in AntMaze tasks, we adopt corruption range of 0.3 and a corruption rate of 0.2 during offline pretraining, consistent with RIQL. For online corruption, we apply a corruption range of 0.5 and a corruption rate of 0.3. The offline value function is trained using RIQL with the default hyperparameters reported in the original RIQL paper. In the AntMaze experiments, employ the same hyperparameters as those in the main paper (Table C.I to demonstrate the robustness of RPEX. All results averaged over 10 random seeds.
Table 5: Results on AntMaze Large Tasks under Dynamics and Observation Corruption.
Environment
RIQL-PEX
RPEX
Large-play Dynamics
33.3 37.8
33.3
38.9
Large-play Observation
23.3 - 21.1 27.1
23.3 + 35,3 =2.7
Large-diverse Dynamics
23.3 20.0 =5.4
23.3 25.6
Large-diverse Observation
26.7 - 25,6 +8.1
26.7 + 36.7 #5.3
Average
26.6 - 26.1
26.6 - 34.1
As shown in Table RPEX outperforms RIQL-PEX by a large margin of approximately 30.7%.
Training Curves
The complete training curves for RIQL-direct, RIQL-PEX (RIQL), and RPEX
presented in Fig.
21

---

Varying Levels of Data Corruption We conduct experiments with varying online corruption levels (including the non-corre setting) on the Walker2d-M tasks under both dynamics and action corruption. The results are presented in Table 6 We use the same hyperparameters as in the main paper, i.e., & 0.1, a= 3, etc. All results are averaged over random seeds. Across varying corruption levels, RPEX demonstrates strong robustness compared to RIQL-PEX,, outperforming it by an average approximately 16.8%, further validating the effectiveness RPEX.
Table 6: Results under varying levels of online corruption.
(<2,02)
Dynamics (RIQL-PEX)
Action (RIQL-PEX)
Dynamics (RPEX)
Action (RPEX)
(0,0)
80 + 94.8 213.7
85.9 - 101.8 20.2
80 - 98.9 16.9
85.9 - 102.5 14.6
(2,0.3)
80 + 37.6 212.5
85.9 - 69.5 129,3
80 - 77.8 217.5
85.9 + 97.0 2105,3
(1,0.5)
80 -> 89.5 =1.3
85.9 -> 109.2 #15.6
80 -> 92.2 +1.4
85.9 - 118.9 #10.1
Average
74.0
85.9 93.5
80 89.6
85.9 106.1
In the non-corrupted setting, (e2, 02) = (0, 0), when the value function is inaccurate due to data corruption or limited coverage, some promising actions may be mistakenly assigned negative advan- tages, leading RPEX to penalize them. However, although the value functions in RPEX and other algorithms trained corrupted offline datasets, they nonetheless preserve substantial information that is beneficial for online exploration. If the offline value functions are made more robust, as in RIQL, the value estimates become more reliable.
Moreover, as illustrated in Figures 2 when the value functions are inaccurate, most penalized actions correspond to suboptimal actions generated by the heavy-tailed policy. While IPW may also penalize some promising actions, improves overall expleration exploration efficiency (Figure[[e)) The above analysis demonstrates that the IPW constraint in RPEX does not substantially impair O20 exploration efficiency under uncorrupted settings. This viewpoint is further supported in Table 6 where RPEX outperforms RIQL-PEX in both the uncorrupted setting and across varying corruption rates. Moreover, such penalties can be tuned via x or the clipping range. In our experiments, we fix both x and the clipping range demonstrate the inherent robustness of RPEX; fine-tuning these parameters could further improve exploration efficiency and performance.
D Further Discussion
In both RIQL and IQL., the policy learned via Advantage Weighted Regression (AWR), which can be interpreted as a form of supervised learning weighted by the value function. In the non- corrupted setting. well-estimated value function improves the policy by assigning higher weights to actions with positive advantages, without increasing the heavy-tailedness. of the policy, as shown in Figure [(b). However, as also illustrated in Figure [ our experiments demonstrate that this conclusion does not hold under corrupted settings: heavy-tailed Q-targets increase the heavy- tailedness of the policy. Furthermore, merely mitigating the heavy- tailedness of the Q-function, as in RIQL, does not reduce the policy's 's heavy-tailednes.
Such heavy-tailed policies have limited impact during evaluation, as the policy is executed determin- istically by selecting the action with the highest probability in the evaluation environment, which reduces the effect of policy heavy-tailedness. However, these heavy-tailed policies lead ineffi- cient online exploration, which in turn results in the performance degradation of RIQL after online fine-tuning (Table (T)
In summary, data corruption induces heavy-tailes Q-targets, and this heavy is inherited by the policy through AWR, supervised like training paradigm. While the heavy-tailedmm has limited influence during evaluation, it significantly impairs online exploration.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section Introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims
made in the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section Conclusion
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofss
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Section Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
23

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section Appendix C Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily to reproduce that algorithm.
algorithm, the paper should make it clear how
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See Attachments
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
24

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Section Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't Io make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Answer: [Yes]
Justification: Our paper does not involve these issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following unintentional) misuse of the technology.
from (intentional
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)? Answer: [No]
Justification: Our paper does not involve these issues.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
26

---

Datasets that have been scraped from the Internet could pose safety risks. The authors
should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section Experiments Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See Attachments Guidelines:
The answer NA means that the paper does not release assets. Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: Our paper does not involve these issues.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
27

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our paper does not involve these issues.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http::://eerrpp...ccofefeeeeeeee for what should or should not be described.
28


Reviews:
[{"review_id": "ollcNKzhB0", "reviewer": "Reviewer_moaS", "summary": "This paper proposes a method for learning an efficient and robust policy that can handle both offline and online data corruption, aiming to prevent unsafe behavior in real-world environments. The paper prove the effectiveness of their approach through extensive experiments across various data corruption settings on D4RL.", "questions": "Please see Weaknesses part. And in my opinion, it would be helpful to clarify whether the theoretical derivation can provide an explanation for the role of IPW in addressing data corruption.", "limitations": "See Weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n1. The authors demonstrate the effectiveness of their IPW-corrected PEX method under data corruption through extensive experiments. The experimental evaluation is thorough and well-designed across multiple data corruption settings.\n2. The proposed method is conceptually simple yet practically effective.\n\n**Weaknesses**:\nThe proposed method lacks sufficient theoretical justification. The mathematical derivation presented in the paper largely builds upon prior work [48, 51], and the core algorithm shows limited novelty especially when compared with [51]. Specifically, in Appendix B1, while the authors explain how the key Equation (6) is obtained, the subsequent steps from (17) to (19) are not sufficiently justified which weakens the theoretical foundation of the method.", "was_revised": true, "final_justification": "The authors’ explanations regarding the theoretical derivation, as well as their discussion on several issues raised by other reviewers, have addressed part of my concerns. I am therefore considering increasing my score by 1 point. However, I still remain concerned about the novelty of the work, particularly in comparison with PEX.", "rebuttal": "We are grateful to the reviewer for their time and effort in reviewing our paper. Below, we address your concerns one by one. \n\n> limited novelty\n\nFirst, as stated in the problem statement, our motivation is to develop a robust algorithm capable of handling both offline and online data corruption. This motivation differs from that of RIQL [48], which focuses on robust offline learning, and PEX, which aims to develop an efficient offline-to-online (O2O) method. Addressing both offline and online corruption is critical for real-world RL applications. To the best of our knowledge, our method is the first to propose a robust O2O reinforcement learning algorithm. We also conduct extensive ablation studies to analyze the effectiveness of commonly used O2O techniques, such as retaining the offline buffer and increasing the update-to-data (UTD) ratio.\n\nSecond, compared to PEX [51], RPEX (ours) incorporates inverse probability weighting (IPW) into the action selection weighting (Eq. 6). Although simple, our toy experiment (Fig. 2) demonstrates that this modification mitigates the heavy-tailed behavior of the policy, thereby enhancing exploration efficiency. Extensive experiments show that this modification improves performance by approximately 13.6% over vanilla PEX. Beyond integration with RIQL, RPEX is also compatible with IQL and outperforms IQL+PEX by about 12.4% under both offline and online corruption. The simplicity and scalability of RPEX enable seamless integration with various offline RL algorithms, making it a promising approach for future robust O2O research (as also noted by Reviewer REEm).\n\nThird, although IPW is commonly used in classification tasks, to the best of our knowledge, our work is the first to apply IPW in reinforcement learning and demonstrate its effectiveness.\n\nIn summary, while RPEX can be integrated with RIQL, it is also applicable to other offline RL methods such as IQL. Although Eq. 6 resembles that of PEX, the motivation behind RPEX is distinct. Our experiments demonstrate that this simple modification significantly enhances the robustness of PEX. The simplicity and scalability of RPEX are key advantages that support its practicality in real-world applications.\n\n> lack of theoretical justification\n\nAs shown in Proposition 6.1, Eq. (6) is derived from Eq. (9), which seeks to maximize both the expected reward and entropy, while incorporating a regularization term $\\frac{(Q - V)}{\\pi_{i}(a_{i}|s)}P_{w}(i|a_{1}, a_{2})$. This regularization term consists of  IPW multiplied by the composite policy $P_w$. The inclusion of $P_w$ enables the computation of its gradient, thereby mitigating the heavy-tailedness of the composite policy. We further elaborate on the role of this regularization in lines 231–239.\n\nIn essence, the regularization term reduces the value of $P_w$ for suboptimal actions (i.e., those with $Q - V < 0$), particularly when such actions are taken with low probability (as captured by the IPW term). This indicates that these actions are likely produced by the policy due to heavy-tailedness induced by data corruption. The IPW component serves to suppress these corrupted actions while promoting better ones, thereby enhancing exploration efficiency.\n\nThe above analysis illustrates the rationale behind this regularization term and supports Eqs. (17)–(19). The theoretical derivation, along with the intuitive explanation in lines 196–204, clarifies the role of IPW in alleviating policy heavy-tailedness and improving exploration efficiency under data corruption."}, {"review_id": "zCQFIL6UOn", "reviewer": "Reviewer_K7Qh", "summary": "This paper addresses the challenge of robust offline-to-online reinforcement learning (O2O RL) under diverse data corruption, including noise and adversarial attacks on states, actions, rewards, and dynamics. To combat this, they introduce RPEX, which integrates Inverse Probability Weighting (IPW) into the Policy Expansion framework for adaptive action selection and improved exploration robustness. Theoretical justification is provided, and extensive experiments on D4RL benchmarks demonstrate that RPEX consistently outperforms strong baselines.", "questions": "How does RPEX fare in terms of computational overhead, wall-clock time, and scalability?", "limitations": "Please see weakness.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper tackles an important and underexplored problem: robust O2O RL under both offline and online data corruption, which is central for real-world RL deployment.\n\n- This paper has clear motivation, connecting the challenge of heavy-tailedness in policy distributions to degraded learning and exploration, and linking this to the inefficacy of prior robust RL approaches for O2O settings.\n\n**Weaknesses**:\n- The entire work is somewhat incremental, incorporating Inverse Probability Weighting—an established robustness tool from other domains—into PEX, which is designed for offline to online RL.\n\n- Equation (6) is the most crucial part of the algorithm in this paper. However, some hyperparameters such as IPW Weight $\\kappa$) and the range of Clip are not thoroughly discussed and analyzed in terms of their impact on performance. In other words, the ablation experiments are missing.\n\n- The experimental environment is relatively limited, including only three GYM environments. Other more important environments, such as Maze and AntMaze, are not taken into consideration.", "was_revised": true, "final_justification": "My view that the paper is incremental remains unchanged; nevertheless, in light of the substantial rebuttal material the authors have provided, I have opted to raise my score accordingly.", "rebuttal": "We are grateful to the reviewer for their time and effort in reviewing our paper. Below, we address your concerns one by one. \n\n> How does RPEX fare in terms of computational overhead, wall-clock time, and scalability?\n\nWe report the computational overhead in Appendix C.3, Table 2. In summary, RPEX improves performance by approximately 13.6% over PEX, with only an 8.1% increase in training time. These results suggest that our method enhances performance without incurring significant computational costs.\n\nOwing to its simplicity, RPEX can be integrated with a variety of offline RL and O2O RL methods (as also noted by Reviewer REEm). Our experiments demonstrate its integration with both RIQL and IQL, achieving a 13.6% performance improvement over PEX. These findings underscore that RPEX is both simple and effective.\n\n\n> Ablation study of IPW weight $\\kappa$ and the range of clip\n\nThank you for pointing this out.  We have conducted ablation study of IPW weight $\\kappa$ in Fig 5. As shown in subfigure 3 in Fig 5, the performance gap of different $\\kappa$ is relative small, the max gap is about 9%. Note that we use the same $\\kappa=0.1$ and $\\alpha^{-1}=3$ for our main results (Table 1), although further finetuning $\\kappa$ can improve the performance according to the Fig 5. Such results demonstrate that RPEX is relatively insensitive to $\\kappa$. \n\nFor range of clip, we conduct ablation study on the Walker2d-MR environments under dynamics and action corruption. The reason to choose dynamics and action corruption is them represent two modular $s,a$ in the MDP. As shown in RIQL [48], dynamics corruption is known as difficulty for robust RL. The results of different clip range on the Walker2d-MR is shown in the following table. Note that the  $\\kappa=0.1$,  $\\alpha^{-1}=3$, corruption range (1) and rate (0.5) are as same as the Table 1. All results are averaged over 5 random seeds.\n\n| clip range     | dynamics           | action                 |\n| -------------- | ------------------ | ---------------------- |\n| (-10000,100)   | 80->92.2$\\pm$ 1.4  | 85.9->118.9$\\pm$ 10.1  |\n| (-100,100)     | 80->90.8$\\pm$ 30.0 | 85.9->108.4$\\pm$ 1.07  |\n| (0,100)        | 80->62.1$\\pm$ 38.6 | 85.9->62.67$\\pm$ 44.2  |\n| (-10000,10000) | 80->96.7$\\pm$ 1.69 | 85.9->105.53$\\pm$ 0.09 |\n\nAs shown in the table above, the performance differences across various clipping ranges are minor, except for extreme cases such as the range (0, 100). Similar to $\\kappa$, fine-tuning the clipping range may further improve performance. Throughout our paper, we consistently use the same values for $\\kappa$ and the clipping range $(-10000, 100)$, which further demonstrates the robustness of RPEX. We will include these experiments in our paper.\n\n> More experimental environments\n\nThank you for pointing this out. We conducted additional experiments on the AntMaze-Large and AntMaze-Diverse tasks, which are the most challenging tasks in the AntMaze benchmark. We primarily focus on dynamics and observation corruption, as these represent the most difficult corruption settings, as noted in both RIQL and our paper.\n\nFollowing RIQL, we use a corruption range of 0.3 and a corruption rate of 0.2 during offline pretraining. For online corruption, we apply a corruption range of 0.5 and a corruption rate of 0.3. The offline value function is trained using RIQL with the default hyperparameters reported in the original RIQL paper. In the AntMaze experiments, we employ the same hyperparameters as those in the main paper (line 697) to demonstrate the robustness of RPEX. All results are averaged over 10 random seeds.\n\nThe table below presents the performance comparison between RIQL-PEX and RPEX.\n\n|                           | RIQL-PEX            | RPEX                    |\n| ------------------------- | ------------------- | ----------------------- |\n| large-play dynamics       | 33.3->37.8$\\pm$ 4.2 | **33.3->38.9$\\pm$ 1.8** |\n| large-play observation    | 23.3->21.1$\\pm$ 7.1 | **23.3->35.3$\\pm$ 2.7** |\n| large-diverse dynamics    | 23.3->20.0$\\pm$ 5.4 | **23.3->25.6$\\pm$ 4.7** |\n| large-diverse observation | 26.7->25.6$\\pm$ 8.1 | **26.7->36.7$\\pm$ 5.3** |\n| Average                   | 26.6->26.1          | **26.6->34.1**          |\n\nAs shown in the table above, RPEX outperforms RIQL-PEX by a large margin of approximately 30.7%.\nWe will include these experiments in our paper.\n\n>  The entire work is somewhat incremental \n\nAlthough IPW has been widely used in classification tasks, its role in reinforcement learning,  particularly in robust offline-to-online (O2O) RL, remains underexplored. To the best of our knowledge, RPEX is the first method to incorporate IPW into RL as a regularization term.\n\nCompared to PEX, RPEX introduces only minimal modifications yet achieves significantly better performance—improving upon PEX by approximately 13.6% with this simple adjustment. Furthermore, as noted by Reviewer REEm, RPEX is simple yet effective. Its lightweight modification to the PEX framework enables seamless integration with various offline RL algorithms while providing additional performance gains under both offline and online data corruption.\n\nIn summary, the simplicity and scalability of RPEX are key advantages that enhance its practicality for real-world applications."}, {"review_id": "3hDP86i0xG", "reviewer": "Reviewer_1RkU", "summary": "This paper proposes an effective method, Robust Policy EXpansion (RPEX) for offline2online under the data corruption setting. The authors observe that data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration. For this, they incorporate Inverse Probability Weighted (IPW) into the online exploration policy to alleviate heavy-tailedness. Both theoretical and empirical studies show the superiority of RPEX.", "questions": "Apart from the two points mentioned in Weaknesses, there are also some questions\n1. I notice from Table 1 that RIQL, RIQL-PEX, and RPEX get very limited gain even negative gain in observation and dynamics attack. Can you provide some insights on this?\n2. For the UTD ablation study, I think you can use another implementation way: only perform high UTD on critic but maintain standard (UTD=1) for actor. This is because in some literature, like REDQ [1], they only give the critic a high UTD rather than the actor. And in this case, maybe the issue you mentioned in Section 7.2 can be alleviated? I was wondering about the results in this case.\n\n[1] Chen et al., \"RANDOMIZED ENSEMBLED DOUBLE Q-LEARNING: LEARNING FAST WITHOUT A MODE\", ICLR'21.", "limitations": "The authors have discussed the limitations in their paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. It's novel that this work is the first one to propose a robust O2O rl algorithm.\n2. The authors use quite clear toy examples to illustrate the problem of heavy-tailed behavior in the policy and how it influences online exploration.\n3. Though this method introduces extra hyperparameters, the authors show that their method is not sensitive to them and use the same values across all tasks.\n\n**Weaknesses**\n1. I think the main experiment only considers too few tasks (HalfCheetah-MR, Walker2d-MR, Hopper-MR). It would be better to evaluate it on more benchmarks like AntMaze or Adroit.\n2. All results are run on 5 seeds, which I think might not be that confident. From the training curves in the Appendix, it can be seen that on several tasks, the variances are very large. For these tasks, I think more runs can make the conclusion more convincing.", "was_revised": true, "final_justification": "The authors test their methods on more tasks and run more seeds, which solves my major concern. So I decided to raise my score from 4 to 5.", "rebuttal": "We are grateful to the reviewer for their time and effort in reviewing our paper, as well as for the constructive suggestions. Below, we address your concerns one by one. \n\n> experiments on more tasks\n\nThank you for pointing this out. We conducted additional experiments on the AntMaze-Large and AntMaze-Diverse tasks, which are the most challenging tasks in the AntMaze benchmark. We primarily focus on dynamics and observation corruption, as these represent the most difficult corruption settings.\n\nFollowing RIQL, we use a corruption range of 0.3 and a corruption rate of 0.2 during offline pretraining. For online corruption, we apply a corruption range of 0.5 and a corruption rate of 0.3. The offline value function is trained using RIQL with the default hyperparameters reported in the original RIQL paper. In the AntMaze experiments, we employ the same hyperparameters as those in the main paper (line 697) to demonstrate the robustness of RPEX. All results are averaged over 10 random seeds.\n\nThe table below presents the performance comparison between RIQL-PEX and RPEX.\n\n|                           | RIQL-PEX            | RPEX                    |\n| ------------------------- | ------------------- | ----------------------- |\n| large-play dynamics       | 33.3->37.8$\\pm$ 4.2 | **33.3->38.9$\\pm$ 1.8** |\n| large-play observation    | 23.3->21.1$\\pm$ 7.1 | **23.3->35.3$\\pm$ 2.7** |\n| large-diverse dynamics    | 23.3->20.0$\\pm$ 5.4 | **23.3->25.6$\\pm$ 4.7** |\n| large-diverse observation | 26.7->25.6$\\pm$ 8.1 | **26.7->36.7$\\pm$ 5.3** |\n| Average                   | 26.6->26.1          | **26.6->34.1**          |\n\nAs shown in the table above, RPEX outperforms RIQL-PEX by a large margin of approximately 30.7%.\nWe will include these experiments in our paper.\n\n> more random seeds and large variance \n\nThank you for pointing this out. As shown in lines 250–251, we use different corruption rates of 0.5 and 0.3 for the online and offline phases, respectively. Due to the high corruption probability, the training curves of algorithms may exhibit large variance. However, as shown in Table 1, RPEX achieves better performance with smaller variance compared to other methods. A similar high-variance phenomenon was also observed in RIQL [48], Table 1 (Hopper-MR reward: 84.8 $\\pm$ 13.1), where results are averaged over 4 random seeds. To enhance the credibility of our findings, we conducted additional experiments on AntMaze-Large tasks under both observation and dynamics attacks using 10 random seeds. The results and experimental details are provided in the response to the concern titled “experiments on more tasks.”\n\nAs demonstrated by the experimental results on the AntMaze tasks, RPEX achieves superior performance with lower variance compared to RIQL-PEX. These findings are consistent with those from experiments using five random seeds, further confirming the effectiveness of RPEX.\n\n> insights about limited gain in observation and dynamics attack\n\nThank you for pointing this out. This is a very insightful question. As shown in Table 1 of our paper, all algorithms struggle to improve under observation and dynamics attacks. In particular, under observation attack, RIQL-PEX exhibits a performance drop of approximately 41% on the Hopper-MR task. Although RPEX significantly outperforms RIQL-PEX, it still struggles to improve performance. A similar phenomenon is also reported in RIQL [48].\n\nThe primary cause of poor performance under observation attack is its strong disruption to online exploration. The key objective during the online phase is to collect optimal trajectories and update both the policy and the value function. When observation attacks occur, they affect action selection, potentially resulting in poor or even out-of-distribution (OOD) actions. Moreover, observation attacks can distort the Bellman backup, thereby hindering value function learning. According to Theorem 3 in RIQL [48], such suboptimal or OOD actions, combined with high Bellman backup error, lead to a loose upper bound on the difference between value functions learned under clean and corrupted observations. This explains why improving performance under observation attacks is particularly difficult.\n\nFor dynamics attacks, the limited improvement after the O2O phase primarily stems from Bellman backup errors induced by the attack. Although RIQL reduces Bellman backup error by mitigating the heavy-tailedness of the Q-target, attacks on the dynamics during the online phase still distort the Bellman backup, thereby limiting performance improvement. We will include this discussion in the revised version of our paper.\n\n>  UTD only on critic\n\nThank you for pointing this out. We conduct experiments in which only the critic is trained with a high update-to-data (UTD) ratio, while the actor maintains UTD = 1, on the Walker2d-MR tasks under dynamics and action corruption. All other hyperparameters remain the same as those used in our main paper (line 697). Following table shows the results of RPEX with such UTD implementation. The result for UTD=4 is averaged over 10 random seeds.\n\n| UTD | dynamics           | action                |\n| --- | ------------------ | --------------------- |\n| 1   | 80->92.2$\\pm$ 1.4  | 85.9->118.9$\\pm$ 10.1 |\n| 4   | 80->92.6$\\pm$ 10.0 | 85.9->96.7$\\pm$ 21.8  |\n\nAs shown in the table above, the issue discussed in Section 7.2—namely, that high UTD primarily improves robustness against action corruption while slightly degrading overall performance—can be alleviated. However, we observe that although such UTD implementations can sometimes yield better performance, they also increase overall variance. This phenomenon may be attributed to the fact that data corruption introduces greater Bellman backup errors compared to the non-corrupted setting. While a high UTD ratio for the critic can accelerate learning, the subsequent Bellman backups may fluctuate significantly due to corrupted data. We will include these experiments in our revised paper."}, {"review_id": "Hzfij5ogBy", "reviewer": "Reviewer_REEm", "summary": "This paper focuses on offline-to-online (O2O) reinforcement learning under various types of data corruption. The authors observe that such corruption can induce heavy-tailed behavior in the learned policy. To address this, they propose RPEX, a method that incorporates inverse probability weighting to adjust the action probabilities within the PEX framework. Experiments on D4RL Mujoco tasks demonstrate the effectiveness of the proposed approach in corrupted data settings.", "questions": "1. According to Eq. (6), when an action has a negative advantage and a small probability, the adjusted probability $P_w[i]$ could be negative.  In practice, is this value clipped to zero during implementation?\n\n2. In lines 196–202, the text refers to Eq. (5), but based on the context, should this reference instead be to Eq. (6)?", "limitations": "see Weaknesses", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.Investigating O2O RL under data corruption is an important and practically relevant direction, as such scenarios are common in real-world applications.\n\n2.The proposed method is simple yet effective, and its design based on the PEX framework allows for integration with various offline RL algorithms.\n\n3.The paper is clearly written, and the experiments effectively support the existence of the heavy-tailedness issue in the policy trained on corrupted data.\n\nWeaknesses:\n\n1.The paper lacks a thorough discussion on the connection between the heavy-tailedness of the Q-target distribution and that of the learned policy. One confusing aspect is that RIQL, which explicitly addresses heavy-tailed Q-targets, still exhibits higher policy heavy-tailedness as shown in Figure 1(b). Moreover, RIQL—despite having greater policy heavy-tailedness—outperforms IQL in the experiments. This appears to contradict the paper's conclusion that heavy-tailed policies hinder efficient exploration. \n\n2.The paper does not include experiments under non-corrupted data settings. The proposed method constrains the probability of actions with negative advantage estimates; however, when value function estimation is inaccurate, such constraints might prevent exploration of promising actions, potentially leading to slower performance improvement. \n\n3.The experimental section lacks an analysis of performance under varying levels of data corruption. Evaluating the method's robustness across different corruption rates would provide stronger empirical support and clarify the boundary of its effectiveness.", "was_revised": true, "final_justification": "Overall, I am inclined to support this paper. While the proposed method is incremental, it demonstrates clear effectiveness under the diverse data corruption setting, and also shows benefits on non-corrupted data compared with PEX.", "rebuttal": "We are grateful to the reviewer for their time and effort in reviewing our paper, as well as for the constructive suggestions. Below, we address your concerns one by one. \n\n> Why RIQL despite having greater policy heavy-tailedness—outperforms IQL in the experiments.\n\nThank you for pointing this out. This is a very insightful question. RIQL addresses heavy-tailed Q-targets through observation normalization, the Huber loss, and quantile Q estimators. However, as shown in Figure 1(b), RIQL increases the heavy-tailedness of the policy, and such heavy-tailed policies can lead to inefficient exploration (Figure 2(d)). The reason RIQL exhibits greater policy heavy-tailedness yet outperforms IQL is that “the policy is evaluated deterministically by selecting the action with the highest probability in the evaluation environments, which limits the influence of policy heavy-tailedness” (lines 167–169). RIQL learns a better Q-function than IQL in corrupted settings, and the impact of heavy-tailedness is mitigated during evaluation for the reason stated above. This interpretation is supported by the O2O results in Table 1. In Table 1, vanilla RIQL, when trained with additional data during the online phase, experiences a performance degradation of approximately 6.1%—the only algorithm to show a decline in performance following online fine-tuning—whereas IQL improves by about 13.1%. The performance degradation of RIQL during the O2O phase is attributed to poor online exploration caused by policy heavy-tailedness. Therefore, this phenomenon does not contradict our conclusion that heavy-tailed policies hinder efficient exploration.\n\n> lack of discussion on the connection between the heavy-tailedness of the Q-target distribution and that of the learned policy.\n\nThank you for pointing this out. This is a very insightful question. In both RIQL and IQL, the policy is learned via Advantage Weighted Regression (AWR), which can be interpreted as a form of supervised learning weighted by the value function. In the non-corrupted setting, a well-estimated value function improves the policy by assigning higher weights to actions with positive advantages, without increasing the heavy-tailedness of the policy, as shown in Figure 1(b). However, as also illustrated in Figure 1(b), our experiments demonstrate that this conclusion does not hold under corrupted settings: heavy-tailed Q-targets increase the heavy-tailedness of the policy. Furthermore, merely mitigating the heavy-tailedness of the Q-function, as in RIQL, does not reduce the policy’s heavy-tailedness.\n\nSuch heavy-tailed policies have limited impact during evaluation, as the policy is executed deterministically by selecting the action with the highest probability in the evaluation environment, which reduces the effect of policy heavy-tailedness (lines 167–169). However, these heavy-tailed policies lead to inefficient online exploration, which in turn results in the performance degradation of RIQL after online fine-tuning (Table 1).\n\nIn summary, data corruption induces heavy-tailed Q-targets, and this heavy-tailedness is inherited by the policy through AWR, a supervised-like training paradigm. While the heavy-tailedness has limited influence during evaluation, it significantly impairs online exploration. We will incorporate this discussion into the revised version of our paper.\n\n> experiments under non-corrupted data setting and varying levels of data corruption\n\nThank you for pointing this out. We conduct experiments with varying online  corruption levels (including the non-corrupted setting) on the Walker2d-MR tasks under both dynamics and action corruption. The results are presented in the following table. We use the same hyperparameters as in the main paper, i.e., $\\kappa = 0.1$, $\\alpha^{-1} = 3$, etc. All results are averaged over 5 random seeds.\n\n| Online corruption level: (range,rate) | dynamics (RIQL-PEX) | action (RIQL-PEX)     | dynamics (RPEX)        | action (RPEX)             |\n| ------------------------------------- | ------------------- | --------------------- | ---------------------- | ------------------------- |\n| (0,0)                                 | 80->94.8$\\pm$ 13.7  | 85.9->101.8$\\pm$ 0.2  | **80->98.9$\\pm$ 6.9**  | **85.9->102.5$\\pm$ 4.6**  |\n| (2,0.3)                               | 80->37.6$\\pm$ 12.5  | 85.9->69.5$\\pm$ 29.3  | **80->77.8$\\pm$ 17.5** | **85.9->97.0$\\pm$ 105.3** |\n| (1,0.5)                               | 80->89.5$\\pm$ 1.3   | 85.9->109.2$\\pm$ 15.6 | **80->92.2$\\pm$ 1.4**  | **85.9->118.9$\\pm$ 10.1** |\n| Average                               | 80->74.0            | 85.9->93.5            | **80->89.6**           | **85.9->106.1**           |\n\nAs shown in the table above, RPEX outperforms RIQL-PEX in both the non-corrupted setting and across varying corruption rates.\n\nIn the non-corrupted setting, when the value function is inaccurate due to data corruption or limited coverage, some promising actions may be mistakenly assigned negative advantages, leading RPEX to penalize them. Although the value functions in RPEX and other algorithms are trained on corrupted offline datasets, they still retain substantial information that is beneficial for online exploration. If the offline value functions are made more robust, as in RIQL, the value estimates become more reliable. As illustrated in Figures 2(d)–(e), when value functions are inaccurate, most penalized actions are suboptimal actions produced by the heavy-tailed policy. While IPW may also penalize some promising actions, it improves overall exploration efficiency (Figure 2(e)). This conclusion is further supported by RPEX’s superior performance in the non-corrupted setting. Moreover, such penalties can be tuned via $\\kappa$ or the clipping range. In our experiments, we fix both $\\kappa$ and the clipping range to demonstrate the inherent robustness of RPEX; fine-tuning these parameters could further improve exploration efficiency and performance.\n\nAcross varying corruption levels, RPEX demonstrates strong robustness compared to RIQL-PEX, outperforming it by an average of approximately 16.8%, further validating the effectiveness of RPEX. We will include these experimental results in the revised version of our paper.\n\n\n> question regarding Eq. (6): Is $P_{w}$ clipped to zero when it takes negative values?\n\nWe follow the official PEX implementation by using torch.distributions.Categorical with logit as input, which allows negative values, to implement Eq. (6). We do not clip negative values to zero, and our practical implementation strictly adheres to Eq. (6).\n\n> Typos in line 196-202\n\nThank you for pointing this out. The correct reference is Eq. (6), not Eq. (5). We have corrected this typo in the revised version."}]

Author rebuttal:

Meta-review: This paper proposes an effective method, Robust Policy EXpansion (RPEX), for offline-to-online reinforcement learning under the data corruption setting. It observes that data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration. To resolve this, this paper incorporates Inverse Probability Weighted (IPW) into the online exploration policy to alleviate heavy-tailedness. Experiment results have shown the good performance of RPEX.

This paper is borderline and probably more on the acceptance side. For the strengths, to the best of our knowledge, this paper is the first to propose a robust offline-to-online RL algorithm. It also uses clear toy examples to illustrate the problem of heavy-tailed behavior in the policy and discuss how it influences online exploration. The proposed RPEX algorithm is simple and scalable. With the additional experimental results mentioned in the authors' rebuttals, the experimental results of this paper also look strong.

However, on the weakness side, though this paper proposes a novel algorithm for a novel problem, to experts in this field, both the considered problem (robust + offline-to-online RL) and the proposed algorithm (PEX + IPW) might seem a bit incremental. In other words, both the problem formulation and the developed algorithm might not be surprising to experts. This limits the novelty of this paper.

Overall, I think this is a borderline paper, slightly more on the acceptance side. I recommend accepting this paper.
Decision: Accept (poster)