Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Authors: Jeonghye Kim
Yongjae Shin
Whiyoung Jung
Sunghoon Hong
Deunsol Yoon
Youngchul Sung
Kanghoon Lee
Woohyung Lim
Venue: ICML 2025 spotlightposter (2025)
Keywords: Offline-to-Online Reinforcement Learning
Offline Reinforcement Learning
Penalizing Infeasible Actions
Layer Normalization
Reward Scaling​
Primary area: reinforcement_learning
TL;DR: We propose PARS, combining reward scaling with layer normalization and penalizing infeasible actions, achieving SOTA performance in offline and offline-to-online RL. It is the only algorithm to successfully learn Antmaze Ultra in both phases.

Abstract: Reinforcement learning with offline data suffers from Q-value extrapolation errors. To address this issue, we first demonstrate that linear extrapolation of the Q-function beyond the data range is particularly problematic. To mitigate this, we propose guiding the gradual decrease of Q-values outside the data range, which is achieved through reward scaling with layer normalization (RS-LN) and a penalization mechanism for infeasible actions (PA). By combining RS-LN and PA, we develop a new algorithm called PARS. We evaluate PARS across a range of tasks, demonstrating superior performance compared to state-of-the-art algorithms in both offline training and online fine-tuning on the D4RL benchmark, with notable success in the challenging AntMaze Ultra task.

Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Jeonghye Kim 2:17
2im
Yongjac Shin 2:17
Whiyoung Jung' Kanghoon Lee"
Sunghoo Hong' Woohyung Lim
Deunsol Yoon
Youngchul Sung?
Abstract
Offline Scores
Adroit Expert
Adroit Cloned
PARS
MujeCo Random
AntMaze
Prevlous
Reinforcement learning with offline data suffers from Q-value extrapolation errors. To address this issue, first demonstrate that linear extrap- olation of the Q-function beyond the data range is particularly problematic. To mitigate this, we propose guiding the gradual decrease of Q-values outside the data range, which is achieved through reward scaling with layer normalization (RS-LN) and penalization mechanism for infeasible ac- tions (PA). By combining RS-LN and PA, develop new algorithm called PARS. We evalu- ate PARS across a range of tasks, demonstrating superior performance compared to state-of-the- art algorithms in both offline training and online fine-tumie tinettunig on the D4RL benchmark, with notable success in the challenging AntMaze Ultra task.
Ultra
SOTA
MujoCo
Amthraee
Mad
80
Large
MujoCo
60
AntMaze
-Replay
400
Medium
MujoCo Medium
AntMaae
Umnaze
Experr:
AntMaze Umaze
MujoCo
Medium
Replay
AntMaze Mediun um
MujoCo
Mediumm
Online Finetuning
AntMaze
MujoCo
Large
ArtMaae
Adroit
Random
Scores
Figure I: Comparison of PARS and prior SOTA, with scores normalized to each task's ' maximum performance.
mated, limiting the overall performance (Kumar et al., 2020; Kostrikov et al., 2022; Mao et al., 2024; Kim et al., 2024).
1. Introduction
We first demonstrate that a key factor behind extrapola- tion error in RL with offline data is the tendency for linear extrapolation beyond the data range. ReLU-based MLPs often converge to linear functions outside the observed data (Agarap, 2018; Xu et al., 2021; Yue et al., 2024), leading to persistent boundary trends. When this trend is upward, causes Q overestimation for OOD actions. To mitigate this, recent studies (Ball al., 2023; Yue al., 2024) propose applying layer normalization (LN; Ba et al., 2016) to the Q-function. Ball et al. (2023) shows that LN constrains Q-function predictions by bounding them to the norm of weight layers. However, while LN provides this bound. it does not sufficiently control Q-values outside the data range, leaving the problem unresolved without online interaction.
Reinforcement learning (RL) enables agents to develop op- timal decision-making strategies through real-time interac- tions. However, these interactions with real-world envi- ronments can expose the agent to considerable risks. To mitigate these risks, Offline RL, which derives optimal poli- cies from pre-collected data, has emerged as critical area of research (Fujimoto & Gu, 2021; Tarasov et al., 2024). Ad- ditionally, agents trained with offline RL. can be deployed in real-world environments to further acquire knowledge, lead- ing the development (Lee al., 2022; Nakamoto et al., 2024; LEI al., 2024). However, due the limited coverage of offline data, these methods often suffer from extrapolation error, where the Q-values of out-of-distribution (OOD) actions are overesti-
difline-to-onling RL approaches
Therefore, RL with offline data requires methods that not only bound but also effectively reduce Q-values outside the data range. To address this, we propose two approaches: (1) reward scaling with layer normalization (RS-LN) and (2) penalizing infeasible actions (PA). Our analysis shows that increasing the reward scale with LN reduces the perceived similarity by the function approximator between actions within the data range and those outside it. As a
!Work done during internship at LG Al Research. 'LG AI Research, Seoul, Republic of Korea "School of Electrical Engi- necring, KAIST, Daejcon, Republic of Korea. Correspondence to: Woohyung Lim <w.lim @ ggrescarch.ai>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
result, gradient updates for in-distribution (ID) actions have weaker influence on OOD Q-value predictions, leading to reduction in OOD Q-values beyond the data range. In contrast to previous studies that adjust reward with their own manipulation methods (Kumar et al., 2020; Kostrikov et al., 2022; Tarasov et al., 2024), especially in AntMaze, we offer a novel perspective on reward scaling for OOD mitigation and demonstrate the effectiveness. Additionally, to further enforce gradual decline in Q-values beyond the data range, we penalize the Q-values of infeasible actions far from the agent's s feasible action regions.
The NTK determines how updates generalize across inputs in gradient-based learning. A high similarity in gradients implies that the nonlinear function fo perceives the two imputs and x' as similar points (Yue et al., 2024).
Reducing Q function extrapolation error with layer nor- malization. Layer normalization (LN; Ba et al., 2016) stabilizes neural network training by normalizing the outputs of hidden layers. LN recenters and rescales these outputs using the transformation: for the feature vector of the i-th layer h; h, = hi
C 77 + + for some > 0, where Vaf+e
@ denotes the elementwise product, 14 and of are the mean and variance of the elements of h,, respectively, and 7) and B, are learnable rescaling and shifting parameters, respec- tively. In recent RL studies, LN has been shown to enhance training stability, leading to improved final performance (Ball et al., 2023; Yue et al., 2024; Nauman et al., 2024), Ball et al. (2023) demonstrated that LN can constrain the Q- function prediction by bounding it to the norm of the weight layers. Moreover, Yue et al. (2024) provide a theoretical explanation for LN's effectiveness in mitigating Q-function divergence through NTK (Jacot et al., 2018) analysis.
At the end of the discussion, we introduce PARS (Penalizing infeasible Actions and Reward Scaling), built on the min- imalistic TD3+BC framework (Fujimoto & Gu, 2021). PARS is simple implement, requiring only a few extra lines of code. Without excessive conservatism auxiliary models, PARS seamlessly transitions to online fine-tuning. Evaluated across diverse RL tasks, it consistently matches or surpasses prior state-of-the-art performance (Figure 1). Notably, PARS excels in the challenging AntMaze Ultra task, uniquely enabling successful offline-toonnline training, demonstrating its robustness and effectiveness.
3. Extrapolation Error in RL with Offline Data
2. Preliminaries
3.1. Extrapolation Error for OOD Actions Outside the Convex Hull of In-distribution Samples
The RL problem is formulated as a Markov Decision Process (MDP, Puterman, 1990) M = (po, A. P, R. 7), where Po is the initial state distribution, S is the state space, A the action space, P(s++1|st aạ) the transition proba- bility, R(s,, dz) is the reward function, and 7 (0, 1) is the discount factor. In this study, we focus on scenarios with continuous action space. We extend the action space, typically limited to feasible actions, by defining the infeasi- ble action space A, to account for potential extrapolation into infeasible regions by neural networks. Thus, the action space A is given by the union of the feasible action region Ap, typically confined to a compact subset of R", and the infeasible action region defined as A, RR" Ar, which consists of actions the agent cannot perform in any state.
ground truth Q-values
ID action Q-values
learned Q-function
action
action
feasible action region Ay
ID action region Ap
Aouusin
Composition of offline data and comparison be- Figure 2: tween ground truth Q-values and learned Q-function
To gain deeper insight into extrapolation errors in RL with offline data, revisit the nature of offline data D. For given state the associated action set A, C Ap, can form multiple distinct clusters within the feasible action region Ar, as shown in Figure 2. Ar is typically centered at the origin in many RL tasks, e.g., A = [--,, 1]", where n is the action dimension. In this scenario, OOD actions are defined as As) = {a AF a & A.). and a convex hull can be constructed from A, follows:
Neural network extrapolation. amine the behavior of neural networks when extrapolating, specifically focusing on what they learn beyond the scope of the training distribution. Their analysis reveals that ReLU MLPs struggle to effectively extrapolate in most nonlinear tasks due to their linear extrapolation. As predictions move further from the training data, they quickly transition to linear behavior along directions originating from the origin.
Xu et al. (2021) ex-
Conv(A.) =
n
X2a;
A z 0, Σ 1,a
I E A.
Neural tangent kernel (NTK). One factor that can influ- ence network extrapolation is the extent to which a network update at one point affects the update at another. This can be measured using the neural tangent kernel (NTK) (Jacot et al., 2018), defined as K K2 x') = (Vefe (zf)
Subsequently, any OOD action Ason(s) can be catego- rized as:
AooD-i(s), if E Cony(A.),
E
AOOD om()) if g Con)(A.).
2

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
As shown in Figure 2, extrapolation becomes significantly more challenging in AOoD- As discussed in Xu al. (2021); Yue et al. (2024), ReLU-based MLPs tend to be- have linearly outside the data range, generalizing by ex- trapolating increasing or decreasing trends observed at the boundary of the convex hull. Since no training data ex- ists for Aoon-ouns)) the Q-function in AooD-ou (+) remains unconstrained, leading to uncontrollable error growth.
4. Penalizing Infeasible Actions and Reward Scaling
4.1. Reward Scaling Combined with LN (RS-LN)
Consider training Q-function Qo with positive TD tar- get. Since the network is initialized with weights centered around zero, Qa initially produces small outputs, which gradually increase as it learns to match the targets. During training, the network generalizes across inputs, so learning from one input influences the outputs of other inputs deemed similar by Qo (Yue et al., 2024). If Qo considers actions in AOOD-our (s) to be dissimilar from in in-distribution actions Ap. then the gradient updates that push Q-values higher have weaker effect on Aoou-ou(s)) As result, Q-values for AooD on (x) do not increase as much, leading to their natural suppression relative to Ap.
To investigate this, we analyzed how the max- Q action, which maximizes the learned Q-function for a given state, differs from the dataset actions across various D4RL (Fu et al., 2020) datasets, similar to the approach of Yue et al. (2024). We trained Qo using observed state-action pairs from the dataset (offline SARSA; Sutton, 2018; Kumar et al., 2022) and then derived the max-Q policy using the loss function (()
EgeD (o(s.))(s.a)).. a)].
Therefore, our goal is to encourage Qg to better distinguish between AD and AOoD (s), which requires higher feature resolution in the learned repre- sentations. How can this be achieved simply? To build intuition, consider approximating y = x over the inter- val [0. 1] using a piecewise constant function with equal-width bins. The maximum approximation error in this case is 0.2. If we increase the scale of the function to = 5x. the maximum error increases proportionally to 1. Reducing this error requires a finer partition of the input space, i.e., a higher resolution. In other words, larger output scales in- herently demand finer resolution to maintain approximation fidelity.
Table 1:
Mean action norm for dataset actions and max- Q policy actions, with and without LN, normalized by the possible maximum norm, averaged over 5 seeds.
onro
V
NEX
Datasets
Data actions
Max-Q actions (wia LN)
Max-Q -u actions (w/ LN)
hopper medium
0.95
0.94
Figure 4: Error in approximating y = x and y = 5ar
pen clonel
0.99
0.99
antmaze-umaan diazeeeeeeeeee
0,81
0.99
0.99
sntmazz-ullta-dee
0.81
0.98
0.98
As shown in Table 1, the average action norm in the dataset is around 0.6-0.8, while actions generated by the max-Q policy almost always lie at the end of (u) meeting the boundary of Ap. This behavior persists even with LN, indicating that LN alone can fail to address the issue of linear extrapolation. This problem can become increasingly severe as the dimensionality of the action space A increases. If Q-values rise in AooD. (s) along even single action dimension, unintended actions may be selected.
A similar principle applies to neural networks. When the output scale increases, small differences in the input lead to larger differences in the output, encouraging the network to learn more fine-grained and expressive features, as we will see shortly. However, this mechanism does not always yield positive outcomes. It effective primarily when combined with LN. Note that even if we change the function to y 5x. using five bins still results in the same approximation error if the input domain is reduced to [0, 0.2]. Therefore, to necessitate higher resolution for = 5xr the input volume must remain the same in both cases. LN facilitates this by consistently confining the input volume to the unit sphere.
3.2. Extrapolation for RL with Offline Data
"What is the good extrapolation for RL with offline data?"
A neural network cannot know the ground truth trends beyond the data range. Therefore, in of- fline RL, the Q-value in AOoD-mi (s) must be lower than the maximum Q-value within the data range to ensure the selection of optimal actions from within the data. Thus, extrapolation is required to ensure that the curve in AOoD (s) remains below the in-data maximum, flattening or declining from the boundary of the convex hull Co.)(A.).
action Q-values
Imaginary target Q Q-function
action
Figure 3: function the Q-function needs to extrapolate to.
The target Q-
1.1.1. DIDACTIC EXAMPLE
We analyze how reward scaling and LN jointly mitigate OOD Q-value overestimation by adapting a regression ap- proach from Ball et al. (2023). The true Q-uucctoon is de- fined as = [(n, 22) = Creward (V where Grward is the reward scaling factor, and the feasible in- put region is (2)) E [],, We generate dataset
3

---

Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
No LN
LN
LN (Wider Range)
4.1.2. RL EXAMPLE
We extend our analysis of RS-LN to real RL tasks. In various RL contexts, neural network expressivity has been studied especially in online RL (Kumar et al., 2021; Sokar et al., 2023; Kim et al., 2023; Obando Ceron et al., 2024). In particular, Sokar et al. (2023) introduced the concept of dor- mant neurons (neurons with zero activations) in RL, demon- strating that an increase in dormant neurons correlates with the network's underutilization and degraded performance.
Freweed
Comor 100
Apoibusi
Figure 5: Results of training toy dataset using ReLU MLPs with vanilla regression (the 'No LN" column) and with LN (the 'LN' column), varying Greward- The 'LN (Wider Range)' column shows results under the same conditions as 'LN', but with the x and 22 axes extended to the range [-5,5], 5]. For more examples. please refer to Appendix D.
Figure 7 shows the evolution of Q-value, dormant neuron ra- tio, and normalized return Creward increases, with and with- out LN, when training TD3+BC on the AntMaze- medium- diverse dataset with 7 = 0.995. More examples can he found in Appendix G. As shown in the figure, more ReLU neurons are activated up to 50 to increase expressivity as Greward increases with LN, resulting in overall performance improve- ment. (Note that 100 % activation rate corresponds to linear operation and 50 % seems the maximum expressivity ratio for ReLU.) This result indeed aligns with our analysis that increasing Greward enhances feature resolution or expressivity. To the best of our knowledge, we are the first to show that network expressivity, as evaluated by dormant neurons, linked to OOD mitigation in RL with offline data.
{(11, v)) where y is defined only for inputs satisfy- ing of 22 S 0.52. Consequently, AOoD out (8) refers to the remaining region within (I1, x2) E [-1, 11² where +f+12 > 0.5². We fit the data using a 3-layer MLP (256 hidden units, ReI.U activation) for both vanilla regression and regression with LN, varying Grwwadd from 1 to 100.
As illustrated in the 'No LN" column, the Q-values exhibit catastrophic overestimation in AOOD-our (s). Integrating LN helps mitigate the overestimation caused by linear extrapo- lation; however, Aoon. (s) remains overestimated relative to the in-distribution region. As the reward scale increases with LN applied (see the 'LN' column), the predictions in AOOD-ou become noticeably lower. This effect becomes more apparent when the .1 and 22 axes are extended to the range [-5, 5] (see the 'LN (Wider Range)' column). With LN and higher reward scale, the Q-values in AOoD-out (s) remain closer to zero overall, making them appear relatively lower than those in Ap. Note that reward scaling is effective when used together with LN, as expected.
166
velue
Dorment neurons [%]]
Normelized
Grven
100
1000
Gradient stops
Gredient steas
348
Gradlent stoppentsttpps Tes
Figure 7: Plots illustrating how each metric changes during with TD3+BC.
training in AntMaze- AntMaze-medium-
4.2. Penalizing Infeasible Actions (PA)
While RS-LN aims not to increase the Q-values of AooD (s) to positive values from initial zero through the reduced NTK between Ap and AooD-our (s), we additionally impose hard constraint to guide the function's 's behavior at the convex hull boundary trend downward. That is, we introduce penalties in regions far from the feasible region (infeasible action regions, Ar), ensuring that the downward trend persists while minimizing its impact on Q-vauue pre- dictions within Ap.
To investigate this fur- ther, we plot the normal- ized NTK map to analyze the gradient similarity be- tween the data boundary point p (0.32. -0.38) and the entire feasible action region Ap. As shown in Figure 6, when
Deaex1
Figure 6: The normalized NTK map between p= = (0.32, -0.38) and Ap for Greand = 1, 100.
To achieve this, in addi- tion to the standard TD loss, we consider the PA loss to constrain the Q-value in Ar to Quin- Since we want the Q-function in Ap to be sufficiently well estimated with the dataset and not heavily affected by the constraint in Al. we allow some guard interval between the feasible action region and the constraint-impose A,,
Guard interval
Ap
Guard
A
Interval
Geward = 1, high gradient similarity is observed between p and AooD (s). In con- trast, as Creward increases to 100, the similarity decreases, in- dicating that the network perceives the OOD data as less sim- ila to the training data. Consequently, the influence of posi- tive gradient updates from the ID region (xf + 2 < 0.53) on the OOD region is reduced during training, naturally leading to downward extrapolation.
4
41
Figure 8: Ar and A, for
4

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Therefore, we consider the following subset of A₁:
also utilize a critic ensemble with a limited number of crities (4 for AntMaze, 10 for MuloCo, and Adroit). Appendix G discusses its impact. For policy evaluation, we adopt the same approach as in Ball et al. (2023). For policy im- provement, we use the minimum critic value during offlin training, while averaging a subset of critics during online fine-tuning to avoid restricting online exploration.
A = Li
(L)
i=1
where is the dimensions of action. Note that As is defined as Mi=a (fe, u,)), where & and u, denote the lower and up- per bounds of the feasible space in the i-th dimension. Thus, LI, < l and un < U to allow the guard or transition interval. Then, the PA loss is determined as
5. Related Work
Critic regularization in offline RL. To address the ex- trapolation error in offline RL, various critic regularization methods have been proposed. An et al. (2021) showed that increasing the number of crities in an ensemble can pro- vide effective critic regularization. However, relying solely on ensembles for sufficient regularization can require up to 500 ensembles depending on the dataset, and even then, challenging tasks such as AntMaze may not be effectively solved (Tarasov et al., 2022; 2024).
CPA = min BEN,, [Qo(s, a) - Qmm)) }] (2)
-Q
where Qmin can be calculated as Creward "Tmin/ / (1- the minimum reward of the task is unknown, can be estimated from the dataset's minimum reward, suggested in Mao et al. (2024). By combining two losses with weighting factor a, our modified TD loss is given as follows:
min Toxll
E Bea (2o(s,a) -T(s,,,
Additionally, Kumar et al. (2020): Lyu et al. (2022); Mao et al. (2024) introduce a penalty to reduce Q-values for OOD actions that deviate from the behavior policy μ. Unlike our (3) approach, which focuses more on AuoD and addresses critic regularization from more Q-network-centric perspec- tive, prior works do not distinguish between AooD-aur(s) and AOOD-in(s). instead regulating them together by uti- lizing density differences or an auxiliary behavior model to better differentiate OOD actions within the data sup- port. However, these approximation- based methods face increased uncertainty as the action dimension grows or tasks become more complex. In addition, penalizing OOD ac- tions in AooD-in(s) can also impact Q-value predictions for nearby in-distribution actions. Moreover, online fine-tuning introduces the challenge of adapting density differences and auxiliary models learned offline to the online setting.
+ a +a.Bao'nD
BRN''D aEA
(Qo(s,0) Quin)
where s's is defined as Greward a() + (ea==
4.3. PARS Algorithm
Combining RS-LN and PA, we present novel algorithm that prevents Q-value extrapolation error to ensure stable Q-learning across both offline and online fine-tuning phases: Penalizing infeasible actions and reward scaling (PARS). PARS is based on the minimalist offline RI. algorithm TD3+BC, and an extension the in-sample learning meth- ods (Kostrikov et al., 2022; Xu et al., 2023; Garg et al., 2023; Mansen-Esttruch et al., 2023) can be found in Section 6.5. In addition, we provide a theoretical analysis of PARS in Appendix C.
Offline-to-onligg RL methods. The policy trained on offline can be fine-tuned with additional online interaction, but this often causes distributional shifts between offline and online data (Nair et al., 2020; Lee et al., 2022; Uchendu et al., 2023). To tackle this, various approaches have been proposed based on existing offline algorithms (Lee et al., 2022; Nakamoto et al., 2024; Zhang et al., 2023a; Beeson & Montana, 2022). However, traditional offline algorithms are already designed with limited datasets in mind, leading to conservative learning approach that can limit perfor- mance in online fine-tuning. To overcome this, Uni-O4 (LEI et al., 2024) proposes removing conservatism during the of- fline phase to facilitate a smoother transition to the online phase. Recently, Zhao et al. (2024) proposed ENOTO, effee- tively utilizing ensembles for efficient offline-to-onling RL. Compared to existing methods, we do not propose new spe- cialized techniques for online fine-tuning, enabling smooth transitions and superior performance with PARS.
Infeasible action sampling. The expectation over a E A, in eq. (2) is practically realized by sample expectation or sample mean. To sample actions from the support in eq. (I), consider the following uniform distribution for infeasible action sampling:
(4) ag Uniform(L1, (L,, AL, L1s)) for a; S Lr. Uniform(Ur.: UT,5 + Av) for a; 2 Ur.
As we shall see in Section 6.4, the performance does not heavily depend on the values of T.,.;. U1, when these values are set as 100 to 1000 times the boundary of AF.
Critic ensemble. Following prior works (Ghasemipour et al., 2022; Lee et al., 2022; Ball et al., 2023), PARS can
5

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Table 2: Offline PARS evaluation on the AntMaze, Adroit, and MuJoCo domains. We report the final normalized score
averaged over five random seeds, with = representing the standard deviation.
AntMaze
TD3-C
IQL
CQL.
SAC-N
EDAC
MCQ
MSG
SPOT
SAC-RND
ReBRAC
PARS
antmaze-u
78.6
87.5
74.0
0.0
0.0
27.5
97.9
93.5
97.0
97.8
93.8=2.11
antmazo-u-d
71.4
62.2
84.0
0.0
0.0
0.0
79.3
40.7
66.0
88.3
89,9=7,5
antmazo-m-p
10.6
71.2
61.2
0.0
85.9
74.7
38.5
84.0
91.2+#399
antmazo-m-d
3.0
70.0
53.7
0.0
84.6
79.1
74.7
76.3
92.0::2.2
antmaze-I-p
0.2
39.6
15.8
0.0
0.0
64.3
35.3
43.9
60.4
84.8:5599
antmaze-1-d
0.0
47.5
14.9
0.0
0.0
0.0
71.2
36.3
45.7
54.4
83.2:1:.6
untmauz-ulltrapp
0.0
13.3
16.1
0.0
0.0
0.0
0.6
4.4
20.6
22.4
66.41t:44
antmaze-ultra-
0.0
14.2
6.5
0.0
0.0
0.0
1.0
12.0
10.5
0.8
51.4+111.. 1.66
average
20.5
50.7
40.8
0.0
0.0
3.4
60.6
47.0
49.6
60.6
81.6
Adruit
TD34C
101.
OQ1.
SAC-N
EDAC
SVR
SPOT
SAC-RND
ReBRAC
PARS
pen-cloned
61.5
39.2
64.1
107.5+15.8
pen-expert
133.6
107.0
119.9
117.3
152.741.0 1.0
door-cloned
0.1
0.8
0.4
-0.3
9.6
0.2
1.1
0.0
0.2
1.1
4.3:6..1
door-expert
84.6
105.3
101.5
-0.3
106.3
73.0
83.3
0.2
73.6
104.6
106.0:0.22
hammer-cloned
0.8
1.1
2.1
0.2
0.3
5.2
0.5
2.5
0.1
6.7
23.3-+220.8
expeer-experi
117.0
129.6
86.7
25.1
28.5
75.9
103.3
86.6
24.8
133.8
133.5
relocate-eloned
-0.1
0.2
-0.1
0.0
-0.1
0.9
1.2:+007
relocate expert
107.3
106.5
95.0
106.6
110.5
average
64.7
69.3
54.0
22.0
35.4
49.2
54.1
27.8
18.8
75.0
79.9
MuJoCo
ID34BC
10L.
COL
SAC-N
EDAC
MCQ
SVR
SPOT
SAC-RND
ReBRAC
PARS
halfchectab--r
11.0
13.1
17.5
28.0
28.4
28.5
27.2
23.8
29.0
29.5
30.4+00.7
hopper-r
8.5
7.9
7.9
31.3
25.3
31.8
31.0
31.2
31.3
8.1
25.4.111.5
walkerr2d-r
1.6
5.4
5.1
21.7
16.6
17.0
2.2
5.3
21.5
18.4
21.8::0,2
halfcheetah.m
48.3
47.4
44.0
67.5
65.9
64.3
60.5
58.4
66.6
65.6
64.241
hopper-m
59.3
103.5
102.0
104.1:0044
walker22d-m
83.7
72.5
86.4
97.3:1:2.5
mallcheetahh-mr
44.6
44.2
45.5
63.9
61.3
56.8
52.5
52.2
54.9
51.0
57.0:10.6
hoppor-m-r
60.9
94.7
95.0
101.8
101.0
101.6
103.7
100.2
100,5
98.1
103.11006
walksr22d-m-r
81.8
73.9
77.2
78.7
87.1
91.3
95.6
91.6
88.7
77.3
95.8tt 1.4
halfcheetab--m-e
90.7
86.7
91.6
107.1
106.3
87.5
94.2
86.9
107.6
101.1
103,0=2,4
hopper-m-c
98.0
91.5
105.4
110.1
110.7
111.2
111.2
99.3
109.8
107.0
11311:0.3
walker2ddmm-e
110.1
109.6
108.8
116.7
114.7
114.2
109.3
112.0
105.0
111.6
111.820.7
average
58.2
59.9
60.8
76.3
73.6
71.0
6. Experiments
shown in Appendix G. For detailed hyperparameters and implementation, please refer to Appendix F.3.
6.1. Experiment Setup
Benchmark. We use three domains (AntMaze, Adroit, and MuJoCo) with total of 28 datasets from the D4RL benchmark (Fu al., 2020). In the performance comparison table, used the following abbreviations: for umaze, m for medium, I for large, p for play, d for diverse, for random, m-r for medium replay, and e for expert. For a detailed explanation of the benchmark please refer to Appendix F.I.
6.2. Baseline Comparison
Offline training. We evaluate PARS in comparison with
10 prior SOTA baselines: TD3+BC (Fujimoto & Gu, 2021), IQI. (Kostrikov et al., 2022), CQL (Kumar et al., 2020),
SAC-N (An et al., 2021), I), EDAC (An et al., 2021), MSG (Ghasemipour et al., 2022), SPOT (Wu et al., 2022), SVR (Mao et al., 2024), SAC-RND (Nikulin et al., 2023), and ReBRAC (Tarasov al., 2024). The details of the baselines are described in Appendix F.2. We primarily used official scores from the respective papers. If score was unavailable for certain dataset versions, we referenced other benchmark- ing papers or conducted our own experiments, tuning the algorithm with the recommended hyperparameters.
We uniformly set L.,, = I., and U1,1 U, across all action dimensions i. We use Ar = | and A [ where I, and U, scaled by either 100 or 1000 times the boundary of Ap. Given that in the tasks considered, L₁ < 0 and U, > 0, we sample in- feasible actions from the intervals [2L,, Lr] and [Ur, 2Ur,, respectively. Furthermore, we enable the tuning of around 10 sets of hyperparameters, including a and TD3+BC's 3, similar to prior works (Wu et al., 2022; Nikulin al., 2023: Tarasov et al., 2024). For AntMaze, strong performance can be achieved even with single hyperparameter set,
Hyperparameters.
The evaluation results are summarized in Table As shown, while other algorithms often perform well in specific do- mains but falter in others, PARS consistently demonstrates robust performance across diverse range of domains. For
6

---

Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Table 3: PARS evaluation on the AntMaze, Adroit, and MuJoCo domains after fine-tuning with 300k online samples. We report the final normalized score averaged over five random seeds, with + indicating the standard deviation. The corresponding performance graphs are in Appendix A.
Antmaze
CQL'
SPOT
PEX
RLPD
Cal-QL
ReBRAC
PARS
antmaze-u
99.0:2..6
18.4211
15.2
99.4:0.8
90.1=100.8
994:00.9
99.7=+0.88
unimaze-u-e u-d
76.91139.7
$5.2:1:32.8
34,8:30.11
99.211:100
75.2::35.00
97,41:211
97,8:12211
94,41:3.0
91.21:388
83.41233
97.41:14
95.1166.3
96.8119
99.1::
ontmaze-m-d
98, 8::2.5
91.643.5
86.645.0
98.6
96.344488
95.8::3.6
99,4:t
amimaze-I-p
5..
60.4:1:11.5
56.01339
93.0:12.5
75.01114.7
71.41.0.99
ancmaze-l-d
65.3428.3
69.4+23.77
60.44688
90.4+3399
74,4+11.8
89.0+3.4
96,8:22277
antmaze ultra-p
21.3119.0
0.0:100.0
13.3155.8
8.8.1.
6.9.2.77
0.0.1.00
86.5,14.4
antmaze-ulta-d
6.3:66.6
5.8:11.5
26.7+:11.
40.0=:37.0
5.7::1122
1.0+1.7
86,4+t6.55
68.7
59.0
57.1
78.4
64.8
68.9
95.2
Adroit
CQL
Off2On
SPOT
RLPD
Cal-QL
ReBRAC
PARS
pea-cloned
-2.6=0.1
102.5+166.00
117.1
154
-1.61:66
134.1+7..2
155.4#3.1
doar-cloned
-0.34=0.00
-8.020.22
0.05:10.06
110.866.1
-0.34:0.00
53.3=33.33
102. 202.1:226..8
hammer-clonee
0.24:10.03
-7.400.4
90,24223.2
0.21-10088
114.::10.3
relocate-cloned
-0.33-0.011
.1.500.5
0.29::0.04
4.8:2.11
-0.34:0.00
1.2111
average
-0.8
21.4
51.8
102.5
-0.5
75.8
113.2
MuJoCo
CQL
Off2On
PEX
RL.PD
Cal-QL
Uni-O4
PARS
halfcheetah-r
26.51:34
92.71:5.7
60.9.1:500
91.5422
32,9.18.1
6.8.2.999
100.1
hopper-r
10.0:1.5
95.34-9.2
48.5::38.9
90.2+19.11
17.7=:26.0
1244188
109.71:5,3
walker2d-r
12.477.9
27.9+2222
9.8:1:16
87.7=14.11
9,4:::.6
5.7
113,9:::3.9
hullcheetah-m
7.991:3
103.1114
70.4:1:.33
95.5::
77.0:22.2
56.6:10.88
107.0:1:.0
hopper-m
100.910.66
100.7:10.8
104.8122.6
111.510.4
walker2d-m
88.7+04
209.8
91,4+14.3
121 121.6+22.3
97,0+8.2
106.5+3.4
1264422.1
halfchoctah-m-rt
30.34:28.3
95.6+1.77
35.4:5.11
90.1
62.1=1111
53.245.4
98
hopper-m-r
103.94188
101.7=14.88
95.3#17.2
78.9224.5
101 101.4:22.1
103.4:26.6
107.0:1.4
walker2d-m-r
105.4 1.8
120.3199.4
87.2-113.6
119.042..1
98.4113.3
115.5:42.99
130.1144.4
average
64.1
96.4
67.2
96.2
66.3
62.8
111.6
instance, SAC-N (An et al., 2021) shows strong performance MuJoCo, but it struggles to learn effectively in AntMaze. In contrast, ReBRAC excels in Adroit but falls behind other algorithms in both MuJoCo and AntMaze. Specifically, in the challenging AntMaze Large and Ultra datasets, PARS achieves approximately 24% and 280% performance im- provements over existing baselines, respectively. These impressive results, achieved without excessive computa- tional resources (as shown Appendix H) and through straightforward implementation (as shown in Appendix E), have the potential advance the practical application of offline RL.
pendix A. Observing the results, except for two datasets, PARS outperforms all baselines across all datasets. For the online phase, RLPD, which leverages an offline dataset while learning from scratch online without an explicit of- fline training phase, showed strong performance compared to other baselines. This aligns with previous findings sug- gesting that offline-trained policies and critics can some- times hinder online fine-tuning (Zhang et al., 2023b; Kong et al., 2024; Zhang et al., 2024: Zhou al., 2025). How- ever. PARS surpassed RLPD, challenging these findings and highlighting that online fine-tuning can be highly effec- tive framework when proper critic regularization is applied. Notably, in the cases of AntMaze ultra-play. ultra-diverse, and Adroit relocate-cioned, PARS is the only algorithm to demonstrate strong performance in these challenging sce- narios.
Online fine-tuning. After offline training, we conduct online fine-tuning with 300K of online samples and compare its score with prior SOTA baselines: CQL (Kumar al., 2020), Off2On (Lee et al., 2022), SPOT (Wu et al., 2022), RLPD (Ball al., 2023), PEX (Zhang al., 2023a), Cal- QL (Nakamoto al., 2024), Uni-O4 (LEI al., 2024), ReBRAC (Tarasov al., 2024). We reproduced the results using the official implementations for all baseline scores in online fine-tuning, with details provided in Appendix F.2.
PARS enhances network expressivity by increasing the re- ward scale, enabling swift adaptation novel online sam- plcs. Additionally, the progressive reduction of Q-values at AOOD-oul (s) facilitates online exploration by using of- fline data as an anchor for initiating exploration, thereby significantly narrowing the search space. This approach is particularly effective for online fine-tuning, especially in complex tasks like AntMaze Ultra, where online interaction steps are limited.
The experimental results are presented in Table 3, and the corresponding performance graphs can be found in Ap-
'CQL can be fine-tuned with SAC (Haarnoja et al., 2018). as proposed in Nakamoto et al. (2024).

---

Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
6.3. Comparing PARS with Goal-Conditioned Offiine RL in AntMaze
How significant is the infeasible action penalty in online fine-tuning? PA can be applied not only during offline training but also in the online fine-tuning phase. To inves- tigate its benefits in this setting, we conducted additional experiments by varying the use of PA during fine-tuning, fol- lowing offline training with PA applied. The results, shown in Figure 10, indicate that removing PA during online fine- tuning after using in offline training led to instability, In contrast, maintaining PA throughout resulted in more stable learning and significant performance improvements.
We further compare PARS with recent conl-condit of- fline RL baselines tailored for complex, hong-horizon tasks like AntMaze (Kostrikov et al., 2022; Yang et al., 2022;
Hejna et al., 2023; Park et al., 2024b; Zeng et al., 2024), which leverage techniques such as goal relabeling (Yang et al., 2022; Hejna et al., 2023), hierarchical frameworks (Park et al., 2024b), and advanced architectures like trans- formers (Zeng et al., 2024). Details are in Appendix F.2.
Table 4: Performance comparison of PARS on AntMaze with goal-conditioned offline RL baselines.
Antmaze-ultra-play--0
cammazz-utrradiveree...
100
100
GC- IQL
WGCSL
DWSL.
HIQL
GCPC
PARS
50
50-
91.6
90.8
71.2
79.2
71.2
93.8+2.1
u
88.8
55.6
74.6
86.2
71.2
89.917..5
m-p
82.6
63.2
77.6
84.1
70.8
91.2:4:.9
Environment Steps
Environment Steps
76.2
46.0
74.8
86.8
72.2
92.0:12.22
I-p
40.0
0.6
15.2
86.1
78.2
84.8.55.9
Figure 10:
Ablation PA: normalized score over 300k online fine-tuning steps.
1-d
29.8
2.4
19.0
88.2
80.6
83.2:15.6
ultra-p
20.6
0.2
25.2
39.2
56.6
66.4+::444
ultra-d
28.4
0
25.0
52.9
54.6
11.4611.6
avg
57.3
32.4
47.8
75.3
69.4
81.6
How far should infeasible actions be frrm the feasible action region? As described in Section 4.3, sample infeasible actions far from the feasible region. The distance is determined by |L,] and [Uy], and for the benchmark com- parison, we set |Lr] - |Ur]- Figure 11 demonstrates the impact of this configuration. When |Ly] is small, meaning the sampled infeasible actions are near the feasible region, suboptimal performance is noted, indicating that penalizing infeasible actions may influence policy evaluation within the feasible region.
In Table 4, PARS surpasses goal-conditioned baselines not only on the challenging ultra dataset but also in the overall average score across the AntMaze domain. These results highlight that even in sparse. long-horizon tasks, funda- mental off-policy RL with proper regularization can excel without specialized designs or architectures.
6.4. Discussion on the Components of PARS
How does each component of PARS affect offline perfor- mance?
Artmaaeemedduungg divivesse-2
Antmace-ultra-divene
100
To identify the source of PARS's performance
FRAM
untmaaz-uuazzediverseeee
100
100 [L|== [G]
1000
10000
10
100
1000
10000
gains, we evaluate offline performance across differ-
80
[6]= (U1]
60
ent settings: the critic net- work without LN PA (None), with only PA (PA), with only LN (LN), and with both LN and PA (LN & PA), while varying the reward scale Greward- The results are shown in Figure 9, with error bars indicating the standard deviation. As illustrated, PA helps mitigate extrap- olation error, although its effect diminishes at larger reward scales. The presence of LN leads to a consistent trend of improved performance as Goward increases. Furthermore, combining LN with PA results in even higher performance, aligning with the earlier discussion in Section 4.2. Addi- tional ablation results are provided in Appendix G.
Figure 11: Final normalized score of PARS averaged over five random seeds with varying |L]| = =\Ur-
20
100 Conmerd
1000
10000
Figure 9: PARS offline perfor- mance, with varying Gewand and the application of LN and PA.
6.5. Extension to In-Sample Learning-Based Methods
TD3+BC is an actor-critic method that may sample OOD actions during training. Alternatively, in-sample learning- based methods (Kostrikov al., 2022: Xu al., 2023; Garg et al., 2023; Hansen-Estruch et al., 2023) avoid OOD action sampling by implicitly learning the maximum Q- values using only in-sample data. Building on this, we investigated whether RS-LN and PA could also be applied to such methods. Specifically, we conducted experiments with IQL (Kostrikov et al., 2022), a representative in sample learning approach, using a discount factor 7 = 0.995 and
8

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
increasing the temperature parameter to 20. We also varied y and report the best-performing results for IQL.. For RS-LN, the reward scaling factor Creward was set to 1000.
optimal action
In-sample max action
Q(lss
policy m(-ls)
True Q
action
action
(a)
Table 5: Performance improvement after applying RS-LN and PA to IQI.. The scores are the averages of the final evaluations across five random seeds.
Ap
Ap
AooD-in
Agothout
Figure 12: (a) True Q-values at a fixed state for the feasible action region Ay (b) Learned policy obtained using a highly expressive policy.
IQL (reproduce)
TQL with RS-LN
IQL with PA and RS-LN (Ours)
AntMaze
(Ours)
u-d 66,545,5
83.0+6.2
81.1
I-p 45,4+5.8
60.446.2
60.6+t..
also prevents the policy from discovering the globally op- timal action when it lies within AOOD-in. Therefore, prior work (Park et al., 2024a) highlights that allowing a cer- tain degree of deviation from the in-sample region, without straying too far, can be beneficial. Broadening the range of considered actions helps utilize the Q-function over a wider action space and enhances overall performance, as- suming the Q-function is well trained. In addition, when considering online finetuning, assigning near-zero probabil- ity to all OOD actions can severely limit online exploration, restricting performance improvements.
ultra-p
13,3+5,7
13,6+13,3
37.3= 10.7
As shown in Table 5, PA was not particularly effective in this setting, likely due to its nature method that avoids OOD sampling. Conversely, RS-LN demonstrated noticeable ef- fectiveness, possibly because a more expressive Q-function approximator provides better estimates even for in-sample Q-values. However, did not surpass the performance of TD3+BC-based PARS.
Why does the combination of TD3+BC with RS-LN and PA lead to better results? To understand this more deeply, let us assume the existence of highly expressive policy capable of accurately capturing the in-sample actions. Then, consider the following two major policy extraction methods in offline RL:
To overcome the limitation of in-sample policy learning of IQL, we need to go beyond in-sample policy learning. One way is to consider the policy learning, eq. (6). e.g.. TD3-BC in which (-=(--s), (+)+s) = | (s) - al² Here, the policy is allowed to generate OOD action a' from s', but not too far from in-sample actions. Then, the Q-values of OOD actions, especially those in AOOD-oot- matter now, and suppressing the upward trend in AooD becomes crucial. Here, RS-LN and PA play an important role, creating strong synergy with eq. (6), which leads to strong performance in both offline training and online finetuning.
(1) Weighted behavioral cloning (e.g., AWR, IQL)
max E ( (<(((e))) log
log T(a s)}
(5)
with a to control the (inverse) temperature.
(2) Behavior-constraiied policy gradient DDPG+BC, TD3+BC, diffusion-QLL
(e.g.,
7. Conclusion
max E d-))())))) [Q(s, d')l -WDDD (-(3(-))),
T (-(+s))]],
We introduce PARS, aimed at preventing critic extrapola- tion error and enhancing overall performance in both offline training and online fine-tuning of RL with offline data. Our analysis of reward scaling with LN reveals that increas- ing the reward scale reduces the function approximator's perceived similarity between in-range and OOD actions, weakening the influence of gradient updates on OOD Q- values and Icading to their reduction. Additionally, applying penalties to the infeasible action region can impose addi- tional constraints to ensure that OOD Q-values beyond the data range trend downward. PARS demonstrated substantial performance improvements over previous SOTA across di- verse RL tasks in both offline training and online fine-tuning phases. Our findings suggest a new perspective, departing from conventional views, that strong performance across a wide array of RL tasks is achievable with only simple adjust- ments to off-policy algorithms, provided that appropriate regularization for OOD mitigation is applied.
(6)
where E.ED (R(T(-)s), (+ 3-1ss) a behavior regulariza-
tion term. Thus, a more expressive policy can better approxi- mate the target distribution, such as exp (a(Q(s,) V
in Eq. (5), which in turn increases the value of the objective function.
Consider the example illustrated in Figure 12 (a), which contains two disjoint in-sample action regions separated by an intermediate AooD.in region. Suppose we apply IQL's in-sample policy learning using Equation (5), with per- fectly learned Q-function and highly expressive policy wo initialized near zero. The resulting policy (Figure 12 (b))
will match the Q-function within the in-sample regions but assign near-zero probability elsewhere.
As a result, the policy does not generate OOD actions. While this behavior reduces the risk of selecting poor actions, it

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Acknowledgements
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- tional conference machine learning, pp. 1587-1596. PMLR, 2018.
This work was supported by LG Al Research and partly by the National Research Foundation Korea(NRF) grant funded by the Korea goVIrnment(MSIT) (No. RS-2025- 00557589, Generative Model Based Efficient Reinforce- ment Learning Algorithms for Multi-modal Expansion in Generalized Environments).
Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme q-learning Maxent RL without entropy. In The Eleventh International Conference Learning Representations, 2023.
Impact Statement
Ghasemipour, K., Gu, S. S. and Nachum, O. Why so pes- simistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Ad- vances in Neural Information Processing Systems, 35: 18267-18281, 2022.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S.. Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 2018.
References
Agarap, A. F. Deep learning using rectified linear units 2018.
(relu), arXiv preprint
Hansen- P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q q-lernning actor- critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty- based offline reinforcement learning with diversified q- ensemble. Advances in neural information processing systems, 1147436-7447, 2021.
Hejna, J., Gao. J., and Sadigh, D. Distance weighted su- pervised learning for offline interaction data. In Inter- national Conference Machine Learning, pp. 12882- 12906. PMLR, 2023.
Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint
2016.
Ball. P. J., Smith, L., Kostrikov, I,, and Levine, Ef- ficient online reinforcement learning with offline data. In International Conference Machine Learning, pp. 1577-1594. PMLR, 2023.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint 000iv:6606.8880 2016.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances neural information processing systems, 31, 2018.
Beeson, A. and Montana, G. Improving td3-be: Relaxed policy constraint for offline learning and stable online fine-tuning, In 3rd Offline RL Workshop: Offline RL as "Launchpad", 2022.
Kim, J., Lee, S., Kim, W., and Sung, Y. Adaptive g-aid for conditional supervised learning in offline reinforcement learning. Advances in Neural Information Processing Systems, 37:87104 87135, 2024.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Mille S., and Zhang, Q. JAX: composable transformations Python+NumPy programs, 2018. URL http://githun com/ caan///ooll/////oooooooooo jax.
Kim, W., Shin, Y., Park, J., and Sung, Y. Sample-efficient and safe deep reinforcement learning via reset deep en- semble agents. Advances in Neural Information Process- ing Systems, 36:53239-5533 53260, 2023.
Elfwing, S., Uchibe, E., and Doya, K. linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint 22Xiv:2004.0722. 2020.
Kong, R., Wu, C., Gao, C.-X., Zhang, 7., and Li, M. Effi- cient and stable offline-to-on reinforcement learning via continual policy revitalization. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 4317-4325, 2024.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145 2021.
10

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. In International Conference Learning Representations, 2022.
Nikulin, A., Kurenkov, V., Tarasov, D., and Kolesnikov, S. Anti-exploration by random network distillation. In In- ternational Conference Machine Learning, pp. 26228- 26244. PMLR, 2023.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning- Advances in Neural Information Processing Systems, 33; 1179-1191, 2020.
Obando Ceron, J., Bellemare, M., and Castro, P. S. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Im- plicit under- panddr--rrametrizzation inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021.
Park, S., Frans, K., Levine, S., and Kumar, A. Is value learning really the main bottleneck offline RL? The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024a.
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offine goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024b.
Kumar, A., Agarwal, R., Ma, T., Courville, A., Tucker, G., and Levine, S. DR3: Value-based deep reinforce- ment learning requires explicit regularization. In In- ternational Conference on Learning Representations, net/ forum?
2022. URL id=POvMvLi91E.
Puterman, M. L. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Lee, S., Seo, Y., Lee, K.. Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q g-ensebble. In Conference on Robot Learn- ing, pp. 1702-1712. PMLR, 2022.
Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl- Dickstein, J. On the expressive power of deep neural net- works. In International Conference on Machine Learning, pp. 2847-2882 PMLR, 2017.
LEI, K., He, Z., Lu, C., Hu, K., Gao, Y., and Xu, H. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. In International Conference Learning Representations, 2024.
Sokar, G., Agarwal, R., Castro, S., and Evei, U. The dor- mant neuron phenomenon in deep reinforcement learning, In International Conference Machine Learning, pp. 32145-32166 PMLR, 2023.
Lyu, J., Ma, X., Li, X., and Lu, Z. Mildly conservative q- learning for offline reinforcement learning. Advances in Neurul Information Processing Systems, 35:1711-1724, 2022.
Sutton, R. S. Reinforcement learning: An introduction. Bradford Book, 2018.
Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, V., and Kolesnikov, S. CORL: Research-oriented deep offline re- inforcement learning library. In 3rd Offline RI. Workshop: Offline RL as "Launchpad", 2022.
Mao, Y., Zhang, H., Chen, C., Xu, Y., and Ji, X. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the minimalist approach to offline rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accel- erating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09333 2020.
Uchendu, I., Xiao, T., Lu, Y., Zhu, B., Yan, M., Simon, J., Bennice, M., Fu, C., Ma, C., Jiao, J., al. Jump-start reinforcement learning. In International Conference on Machine Learning, pp. 14556-34583 PMLR, 2023.
Nakamoto, M., Zhai, S., Singh, A., Sobol Mark, M., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wu, J., Wu, H., Qiu, 7., Wang, J., and Long, M. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 1277-31291, 2022.
Nauman, M., Bortkiewicz, M., Mitos, P., Trzcinski, T., Os- taszewski, M., and Cygan, M. Overestimation, overfitting, and plasticity actor-critic: the bitter lesson of reinforce- ment learning. Forty-first International Conference on Machine Learning, 2024.
Xu, H., Jiang, L., Li, Yang, Z., Wang, Z., Chan, V. W. K., and Zhan, X. Offline RL. with no OOD actions: In-sample

---

Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
learning via implicit value regularization. In The Eleventh International Conference Learning Representations, 2023.
Xu, K., Zhang, M., Li, J., Du, S. S., Kawarabayashi, K.-I., and Jegelka, S. How neural networks extrapo- late: From feedforward to graph neural networks. In International Conference on Learning Representations, 2021. URL https: httpss///ppeeevveeeeeeee net/ forum? id=UH- emocLCC.
Yang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., Li, X., Han, L., and Zhang, C. Rethinking goal-conditioned supervised learning and its connection to offline RL. In International Conference on Learning Representations, 2022.
Yue, Y., Lu, R., Kang, B., Song, S., and Huang, G. Un- derstanding, predicting and better resolving q-value di- vergence in offline-rl. Advances in Neural Information Processing Systems, 36, 2024.
Zeng, Z., Zhang, C., Wang, S., and Sun, C. Goal- conditioned predictive coding for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Zhang. H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. In International Conference on Learning Representations, 2023a.
Zhang. H., Xu, W., and Yu, Policy expansion for bridging difline-to-onling reinforcement learning. In The Eleventh International Conference Learning Representations, 2023b. URL http:: / /ttpp//wweeww.nettete forum? 1d=-Y34145JR6..
Zhang. Y., Liu, J., Li, C., Niu, Y., Yang, Y., Liu, Y., and Ouyang, W. A perspective of v-luuue estimation on offline-to-onlinn reinforcement learning. In Proceed- ings of the AAAI conference on artificial intelligence, pp. 16908-19660 16916, 2024.
Zhao, K., Hao, J., Ma. Y., Liu, J., Zheng, Y., and Meng, Z. Enoto: Improving offline-to-online reinforcement learn- ing with q-ensembles. In Proceedings of the 23rd Interna- tional Conference on Autonomous Agents and Multiagent Systems, pp. 2609-2611, 2024.
zhengyao jiang, Zhang, T., Janner, M., Li, Y., Rocktäächel, T., Grefenstette, E., and Tian, Y. Efficient planning in compact latent action space. In International Conference on Learning Representations, 2023.
Zhou, 7., Peng, A., Li, Q., Levine, S., and Kumar, A. Effi- cient online reinforcement learning fine-tuning should not retain offline data. The Thirteenth International Confer- ence on Learning Representations, 2025. URL ht tts:: //openreview.oomm wet.rreeieeeeeeeeeeeee htt.forrmmmm..o022AAAA.....
12

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
A. Performance Graphs for Online Fine-Tuning Tuning (300K)
PARS
COL
Off2On
SPOT
PEX
RUPO
Cal- QL
ReBRAC
Uni-04
Antmaze v2
Antmaze-umme umwaze- diverse-2
Antmaze medium-ppey-y-
Antmaze- mediim--dieeeeeee
Normalized Return
100
Normalized Return
100
Normalized Return
300
Narmalized Return
300
80
<0
40
40
20
20
20
20
Stegs
165
Steps
165
Stegs
165
Steps
165
Antmazze-arge-playyy
Antmaaze-lrrg-diiveess--
Artmaze-ultte-playyyo
100
100
100
100
Normalized nettin
Normalized Return
Normalized Return
Normalized Return
80
20
60
40
20
20:
20
0-0
Stegs
le5
Pen-cloned-v1
Door-cloned-v1
Relocate-cloned-y1
Hamimer-cooned-v1
Normelized Return
Normalized Roturn
Hormalzzd Return
Normalized Reetum
100
200
50
22
Steps
Staps
Steps
Steps
Halfcchetthh randomnvv2
Hopper candom
Walkerr2d
Nomollzed Return
100
Normelized Returnn
125
100
00
Nomalized
so
20
Steps
Steps
Steps
Ttalfchettah-mediumm
Hopper-mediunee
Wellker26-mediumm-22
150
Narralzzed Return
Norm Colielizee Return
Normalized sileeeed Return
200
73
50
25
23
Steps
185
Staps
185
Steps
Hopper-mediin-eepllavw
Nalker22-mmeduummeelaaaaz
Normaiized Return
100
Normaiized Return
Normalized Return
150
80
100
60
-33
40
15
3
Steps
185
Steps
165
Steps
165
Figure 13: The performance graph of the online fine-tuning (300k), using five random seeds, corresponds to Table 3. The solid line indicates the mean, while the shaded region represents the standard deviation.
13

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
B. Limitations and Future Research.
Although PARS has achieved superior performance, its approach is most effective when the target Q-value is positive. To naturally mitigate OOD, it is necessary assumee a positive Q-value so that reducing the impact of in-distribution data gradient updates on OOD output leads OOD value reduction. In RL, rewards can include both positive rewards and negative penalties. If the penalties outweigh the rewards, the Q-value can become negative. In such cases, the Q-function can be trained by adjusting the offline data rewards to ensure that the overall reward remains positíve. Moreover, it still benefits from the support of a critic ensemble, particularly in MuJoCo and Adroit, where misestimated Q-values in OOD actions within the convex hull of the data can be further corrected by the ensemble. We expect that further investigation into data fitting challenges in limited datasets, coupled with potential refinements, could lead to robust performance with just a double-critic setup.
As future research direction, exploring the most efficient way apply LN to RL with offline data, both theoretically and empirically, would be valuable. Additionally, designing activation functions in combination with RS-LN PA presents an intriguing research area.
C. Theoretical Analysis
C.1. Theoretical Justification of PARS
Here, we show the convergence of the PARS loss function defined eq. (3). For this, we define the following sets:
{(s, a) | 8(a 8) > 0}
OOD (o)) ID
OOD out (ConvexHul((I)))
Now, we define the operator T as:
Q*)
(s, a) E ID, if (s, a) E 00D
T a(s,)=
Bust a')],
(8)
pars
Qmin:
(s. a) E OOD
where kNN(s,, o; ID) denotes the set of the k-nearest neighbors within ID. We show that T pars is contraction under thel loo norm:
Case 1:
T
pare Q1 T
pars Q2 17 T" S <1Q1 - Q21loo
Case 2:
T pare Q1 7
pars Q2 = [[ANN] [T* ELNN [ Q2]1 S <1111 - Q21la0
(9)
Case 3: (mmi - Qiiil = 0 S 7/|Q: - Q211
Thus, T is contraction, and the Q-function converges accordingly.
In Case 2, computing kNN(s,, a; ID) is computational expensive. Since the Q-function is approximated by neural network, we allow implicitly learn this behavior rather than explicitly incorporating into the loss function. For the same reason, assigning Qmin can unintentionally affect nearby values. To prevent this, introduce "guard interval" that excludes actions just outside Ap from being evaluated as Quin- In practice, guard interval of about 1000 (see Figure 11) is sufficient to avoid influencing Q-updates.
C.2. Comparison of TD3+BC and IQL Critic Updates and the Effectiveness of RS-LN
Following the discussion of applying RS-LN to in-sample learning-base based methods in Section 6.5, we compare the critic update of TD3+BC (Fujimoto & Gu, 2021), as used in PARS, with the critic updates of other in-sample learning-based based methods. We focus on a representative method, IQL (Kostrikov et al., 2022), to analyze in depth whether RS-LN can be broadly applied.
First, the TD3+BC critic update loss function is defined as:
Lo min Eg Eg,..''>
[(ooss,) (Creward T(8, a) + (lo)
(10)
14

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Additionally, the critic update loss function of IQL is expressed as:
2 min S (Qo(s, a) (Crwardd + (Vox))+
(11)
with an additional value function loss that aligns (s(() with Qs a):
Lv = min E8 anD C (Qo(s, a) - Vo(())]
(12)
where expectile regression is defined as (u) = |1| 1(u < 0)|u², with E [0.5, 1), to formulate the asymmetrical loss function for the value network Vy.
Therefore, the critic updates in TD3+BC and IQL can be expressed in unified form. The generalized critic loss as follows:
C min E a' [(oo(s,)) ((,,,,5)))
(13)
where the target (s, a, s') is defined as:
Y(s,a,s)) Grewadd r(s,a) ((8.8))+G(s'))
(14)
The key difference lies in the definition of the next-state value estimate G(6'):
G(d)) = Rol ,'),
(TD3+BC)
(15)
Vi(('),
(IQL)
Although the target (ss a, s') varies between BC and IQI. and is dynamically adjusted throughout the learning process rather than remaining fixed, the core task fitting Qo to Y follows a structurally similar approach in both methods. Each method minimizes the squared error between Qo and the changing Y, providing a common framework for understanding their update mechanisms. In this context, both methods share the characteristic that an increase in Geward leads an increase in the target value Y and that the application of LN helps mitigate catastrophic overestimation. As analyzed in Figures and 7, this can have the same effect regression fitting and network expressivity, suggesting the potential for the general applicability of RS-IN.
D. More Discussion on Didactic Example
Analyzing split data distributions. Figure illustrate a scenario in which data is concentrated in a single region. Additionally, we examined new toy dataset consisting of two slightly separated inverted cones and analyzed its learning characteristics.
LN
LN PA
Grewert
Greword
Creward 100
Ao
Apon
Figure 14: Results of fitting the two inverted, cone-shaped input datasets using an MLP network, with and without LN and PA, at Greward of I or 100.
15

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
where only LN is applied (the 'LN" column) and where both LN and PA applied (the 'LN & PA column). As observed, LN mitigates OOD Q-value overestimation, and when the reward scale increases, the predicted values in AOOD-our (s) even decrease relative to the in on-distribution predictions, similar to the results in Figure 5. Additionally, when both LN and PA are applied. the Q-value in AOOD om (s) is further pushed downward, gradually decreasing as it moves further from the data range. Furthermore, LN enables smooth interpolation between the two split data regions.
Figure 14 demonstrates the data fitting behavior when Grward set to 1 100, comparing
Impact of activation functions. In Section 4.1, we demonstrated that RS-LN's effectiveness is tied the network expressivity, which can be also influenced by the activation function (Raghu et al., 2017). The activation function plays a erucial role in determining whether neurons activated remain inactive during training. Accordingly, examined how fitting characteristics, including OOD mitigation, vary depending on the activation function.
We tested the toy dataset from Figure 5 in a setting with Creward = 100 and LN applied, using GELU (Hendrycks & Gimpel, 2016), Sigmoid, SiLU (Elfwing et al., 2018), as well as cases with no activation function. The results can seen in Figure 15. While Sigmoid prevents OOD Q-values from increasing, does not reduce them as effectively as other activation functions. GELU reduces OOD Q-values, but was overly conservative. Exploring the effect of activation functions on network expressivity could also be an interesting topic for future research. In the absence of activation, OOD Q-values remained high, even with large Creward. This effect extends beyond the didactic setting, influencing real-world RL performance, as detailed in Appendix G.
Activation
Activation Functions
2.0
3.5
sigmaid
0.0
22
2
(b)
(a)
Figure 15: (a) The fitting results of applying various activation functions to the toy dataset from Figure 5 in setting with Grewadd = 100 and LN applied, (b) Plot of various activation functions.
E. Reference Implementation
We provide a JAX-based reference implementation of the critic loss, and the complete training code is available at https://githuub cow/LGAI-Researaeeeeaaaa
def
Leritia_loss
critic_porame
suitic_target_paraet
sctor_largel__aareme, transicion,
ragl:
state, action, aext_state, reward, nos_done transition
Scale the reward by the predefined reward lnn factor reward - relt,
+ reward
Compute the target q-value (implementation details smitted) target_0 ... Placeholder for target Q value computation
Get current Q estimates kimplamentzzinn details onitted) current_0 ... Placeholder for current eate compotation
Compute critic loss = jnp.maan aquare (current_0 target_ali, 0] [None, ....))
Uniform infeasible action sampling
infessible_ = (jax.randan uni formirng, shappe) 2)
16

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
infeaaible_action where (infeasible_scttoo 6, infeasible_acthe infeaaible_action 1) aelf.I
Compute current O-value for infeasible action
curreat_ innfessille selI.erilii.mmdee.pp... parana, state, intessible_action
Set target Q-value infeasible action min_g
target_0_infeasta jrp.onea_ Lanea_like (target_01 self.min_g
Compute infessible loss 0 Q_iosa_infeseibl - unp.mean(jnp httpppppttttoreett.....neeee Larget___inteeeibbee 0J [None, ...]))
Combine the losses
critic_ioas l_ss self.alpha @_less_infeasible
retura critic_lose
Listing 1: An example of reward scaling and infeasible action sampling implementation in JAX.
F. Experiment Details
F1. Benchmark Details
Fu et al., 2020 introduced a variety of datasets designed for different RL tasks, such as AntMaze, Adroit, and MuJoCo. Additionally, zhengyao jiang et al., 2023 expanded the AntMaze domain by proposing an ultra dataset, featuring larger map size than previously proposed, thus increasing the complexity of the task.
For the AntMaze domain, we leverage eight datasets: (umaze-v2, umaze-diverse-v2, medium-play-v2, medium-diverse--v? large-play-v2, bagg-diverse-v2, ultra-play-vll ultra-diverse-vO). These datasets encompass different levels of difficulty based on the maze's size, complexity, and the diversity of start and goal positions.
In the Adroit domain, we utilize four tasks: {pen, door, hammer, relocate), each associated with two dataset qualities: fcloned-vl, expert-v1). The cloned datasets are created by training an imitation policy using demonstration data, executing the policy, and mixing the resulting data with the original demonstrations 50-50 ratio. contrast, expert datasets are derived from fine-tuned RL policy.
For the MuJoCo domain, use three tasks: (halfcheetah, hopper, walker2d), each of which has four dataset qualities: (2a medium-replay-e medium-v2, medium-expert-v22) The random dataset consists of data generated by randomly initialized policies, the medium dataset comes from partially trained policies, the medium-replay dataset includes data from replay buffers during training, and the expert dataset contains demonstrations from well-trained agents performing near-optimal behavior.
F.2. Baselines
Offline training. We primarily relied on the officially reported scores from each paper for datasets benchmarked for comparison. For datasets that were not benchmarked in the original paper, we referred to scores from other works that reported results for those datasets. For datasets without available scores from other sources, we reproduced the results using the respective implementations, tuning hyperparameters according to the recommendations in each paper.
Specifically, for datasets not benchmarked in the original paper, either obtained the scores or conducted experiments as outlined below. In all other cases, we used the scores reported in the original paper. In cases where we conducted the experiments, we reported the final evaluation scores using five random seeds, with the mean and standard deviation provided in Tables and 7.
AntMaze: begin, since the AntMaze Ultra datasets had not been benchmarked in previous studies we compared, we conducted experiments for all prior baselines. For MSG and SAC-RND, we ran the experiments using the official implementations. For TD3+BC, IQL, CQL, SPOT, and ReBRAC, used the CORL library (Tarasov et al., 2022), which provides single-file implementation of state-of-the-art (SOTA) offline RL algorithms. For the remaining datasets, we referenced the SAC-N and EDAC scores from Tarasov et al. (2022). Additionally, we sourced the MSG and SAC-RND scores from Tarasov et al. (2024), as those works benchmarked the v0 and vl versions of the AntMaze datasets, rather than the v2 versions.
Adroit: For Adroit, we obtained the TD3+BC, IQL, CQL, and SAC-RND scores from Tarasov et (2024), and the
17

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
SAC-RND and EDAC scores for expert datasets from Tarasov al. (2022). Moreover, for SPOT and SVR, the experiments using the CORL library and the official SVR implementation, respectively.
MuloCo: For MuloCo, we obtained the IQL and CQL scores for random datasets from Lyu et al. (2022). Moreover, we ran SPOT on random datasets using the CORL library.
The URLs for each implementation are listed below:
CORL
MSG http:///s/..hhcoooooo..eeeerrerr
SSCCADD-.ttt///etttaaaoo////oaaa SVR wtt.tttasss///hhhbb.mooo///oooo0
Table 6: Final normalized evaluation scores averaged over five random seeds for SPOT and SVR. For each dataset, we tuned them as recommended the paper and reported the best scores.
Dataset
SPOT
Dataset
Dataset
MCQ
halfchectah-r
23.8+0.5
pen-cloned
65.6
antmaze-u
27.5=200.6
hopper-r
31.2+0.4
pen-oxpert
119.9:#11.2
antmazz-u-d
0.0:0.00
walker2d-r
5.34993
deor-cloned
1.1+16
antmaze-m--p
0.0:0.0
pen-cloned
18.2
door-expent
83.3114.9
antmare-m-d
0.0:0.0
pen-expert
14931
bammer-cloned
0.5:004
antmaze-I-p
0,0::.00
door-cloned
0.0:0.00
hammer-expert
103.3:16.2
antmaze-l-d
door-expert
0.240.0
relocate-cloned
0.0:00.0
pen-cloned
38.32288.
hammer-eloned
2.533.2
relocate-expert
59.3:1002
pea-expert
121.2:15.9
hummer-expeee
86.61146..3
door-cloned
0.2+0..5
relocate-eloned
=0.1=0.0
door-cxpert
73.0#2222
relocate-esye expert
0.0.10.0
hamimer-clooed
5.2+6.33
hammer-exper expert
75.91130.2
relocate-cloned
-0.1::0.0
relocate-expert
82.547.2
Table 7: Final normalized evaluation scores averaged over five random seeds for the baselines on the AntMaze Ultra datasets. For each dataset, we tuned them as recommended in the paper and reported the best scores.
Dataset
TD3-BC
IQL
CQL
MCQ
MSG
SPOT
SAC-RND
ReBRAC
antmaae- ultmm-
0.0:10.0
13.3
16.118.5
0.0.1...
0.6.10.9
4.4.E1.3
20.6:115.0
22.4.11777
untmaze-ultea-d
0.0:0.00
14.2#66.2
6.5tt:355
0.0::0.0
1.2114
12.0:1444
10.5:18.8
0.8L188
During the online fine-tuning phase, since the official score for 300k online samples is typically unavailable, re-run all the baselines using their corresponding official implementations, except for CQL, which uses the provided code from Cal-QL. The URLs for each implementation are listed below:
Online fine-tuning.
RLPD hLLpoh.ttee///tthhhbooo///ttt/to
Cal-QL G.t.0-/tttees/gghttu...oo//////r
Off2On ott.ooa.tttpss//.ttbb...o////aap
PEX http://ps/ggtttbo.omm////aaaaaaa http:///ht////hhhhooo///////aaha
Uni-O4 bni.ll-nttprr/ghhhhu..co///aaaas
SPOT PPOT-https::/giithbbbcmm /tt
Coal-conditioned offline RL. We obtain the HIQL scores from Park et al., 2024b, and the GC-IQL, WGCSL, DWSL, and GCPC scores from Zeng et al., 2024. For the umaze and umaze-diverse scores for HIQL, we run the experiments using the implementations provided for each respective algorithm the official HIQL repository (https: htt///gitttt com/ seohongpark, /HIQQ.). The resulting scores are as follows: umaze: 79.2+4.2, umaze-diverse: 86.2+t..7.
18

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
E33 PARS
We built our code on the JAX (Bradbury et al., 2018) implementation of TD3 (Fujimoto et al., 2018) (https://githii) com/yi fan12wu/ wttp://ddwjppt td3-jax jax) and made modifications to suit the PARS algorithm, such as adding infeasible action penalty, and reward scaling. For critic ensembles, we referenced the implementation of SAC-N com/ Htth//Howhhhhhh sawuhhhsac-n--aaae jax).
Table 8: Qmin for each task.
Table 9: PARS's general hyperparameters.
AntMaze
Quin
Offiie Hyperparameters
Value
antinaze
opiimizer
Adam (Kingma & Ba, 2015)
antmaze-medium
batch size
256
antmazo-large
learning (all aetworks)
30-4
antmaze-ultra
tau (r)
Adrait
Quin
hidden dim (all networks)
256
pen
-715
gammaa (r)
0.995 on AniMaze, 0.99 on others
door
-42
infeasible region distance
1000 on AntMaze 100 on others
hammer
-348
actor cosine scheduling
True on Adroit, False on others ReLU (Agarap, 2018)
relocale
nonlinearity
MuJoCo
Quin
Online Fine- Tuning Hy Hyperparameters
Fine-Tuning
Value
halfchectah
-366
exploration noise
0.1 MuloCu 0.05 olhers
hopper
-666
learning starts update (UTD) ratio
0
walker2d
-229
AntMaze. For offline training, we tuned Creward to 100, 1000, and 10,000, between 0.005 and 0.01, and between 0.001 and 0.01. During online fine-tuning, we used 50/50 mix of offline and online data. The online / was tuned to 0, 0.001, and 0.01, while was fixed at 0.001, except for ultra-play, which used 0.0001, and Skaw. was set to 1. Thee hyperparameters for each dataset are listed in Table 10.
Adroit. For offline training, we set Geward to 10 and tuned 3 between 0.1 and 0.01, and a to 0.001 and 0.01. During online fine-tuning, a 50/50 mix of offline and online data was used. The online a was set to 0.001, and , was tuned to and 0.01, and S K was set to The specific hyperparameters for each dataset are provided in Table 10.
Table 10: Dataset-specific hyperparameters of PARS for AntMaze and Adroit domains used in offline training and online fine-tuning.
Offiine
Online
Offline
Online
AntMaze
/
Adroit
g
g
antmaze-umaze
10000
0.005
0.001
pen-eloned
0.01
10000
0.005
0.001
docr-cloned
0.01
antmaze-medin maemuae-medium-llle play
hammer-cloned
0.001
antmaze- cemmaze-ediiu--dee diverse
1000
0.01
0.001
relocate-cloned
0.01
0.01
0.01
untmaze-lange play
1000
0.01
0.001
0.01
pen-expert
0.01
0.01
aatmaze-lrrgo-iveen
10000
0.01
0.01
0.01
door-expert
0.1
0.001
untmaz-uulra-play
100
0.01
0.001
0.001
hammer-cxpert
0.01
0.001
antmaze- ultrra divense
10000
0.01
0.01
0.01
relocate 0.1
0.001
MuJoCo. For offline training, we used a Creward of 5 for HalfCheetah and a Creward of 10 for Walker2d and Hopper. The was set to 0, and a was tuned between 0.01, 0.001 and 0.0001. Additionally, for this domain, we found that adjusting policy noise and Skmm provides further benefits, so we varied the policy noise between and 0.2, and Skmm between and 10. During online fine-tuning, used 5% offline data for the HalfCheetah and random datasets, and 50% for the remaining datasets. We also tuned a among 0.1, 0.01, and 0.0001, and the critic sample size for policy improvement between I and 10. The specific hyperparameters for each dataset can be found in Table 11.
19

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Table 11: Dataset-specific hyperparameters of PARS for MuJoCo domain used in offline training and online fine-tuning.
Offine
Online
Mu,ooCo
policy noise
Skyrer
halfchoctah-raaddon
0.0001
0.2
0.0001
halfchootah-medium
0.0001
0.0001
halfcheetahh mediumm replay
0.0001
0
0.0001
10
halfcheetahh mediumn- expert
0.0001
0.2
hopper-
reyper-random random
0.01
0.2
0.01
hopper-medium
0.01
0
0.1
hopper- Bopper-mediun-repee replay
0.01
0.1
hogper-mediun-expe
0.0001
0.2
walker2d random
walker2d-medium
walker2d medium replay
0.01
0.01
walker2d medium expert
0.0001
0.2
G. More Ablation Study
Extended results on the impact of PARS components on offline performance. In addition to Figure 9 in Section 6.4, we conducted additional experiments on various datasets to further explore the impact of the PARS components in offline training. As shown in Figure 16, beyond AntMaze, the application of RS-LN in MuJoCo and Adroit leads to a general improvement performance. Furthermore, incorporating PA results more robust enhancement.
Nore
LN
Wallker2d-random--2
Wa ke::22dmeddimm-v
Pen-claned-vi
Hammer-cioned.vv
20
120
15
80
10
60
40
20
10
01
0
10
10
10
Greandd
Creasn
Greowed
Creawed
Figure 16: Extended ablation results of PARS components beyond Figure 9. We evaluate PARS with varying Greward and the application of LN and PA, averaged over five random seeds. The error bars represent the standard deviation.
Impact of Creward on the dormant neuron ratio. In Section 4.1.2, we demonstrated that RS-LN effectively enhances network expressivity the RL example, contributing to performance improvements. Additionally, we analyzed how the dormant neuron ratio changes with Greward when LN applied across various datasets. Similarly to Section 4.1.2, we examined the dormant neuron ratio for Qg. trained using TD3+BC with 0.995.
Table 12: Dormant neuron ratio across various AntMaze datasets based Greward while applying LN to TD3+BC Q+-
AntMaze
Dormant neurons |% (Grocnt 1)
Dormant neurons [%] (Couant 100)
antmaze umaze-divets. divertse
91:13300
$8:12.11
antmaaze ultra play
93:1.2.1
17:11.5
anmazo-ulra-divens.
94+11.8
5442.3
As shown in Table 12, the trend observed in Figure 7 remains consistent across different datasets. We confirmed that as the reward scale increases, the number of dormant neurons decreases, facilitating the utilization of higher feature resolution and enhancing network expressivity.
AntMaze performance using single hyperparameter setting. We further verified that comparable performance in the AntMaze domain can be achieved with just a single hyperparameter setting. This is particularly significant because many offline algorithms have struggled in the AntMaze domain, especially in the Ultra environment. Achieving significantly better performance than the baseline with a single hyperparameter setting in AntMaze, even though is slightly lower than the PARS score presented in the paper, highlights a key advantage of PARS. The hyperparameters used in the single-hyperparameter experiment are a reward scale of 10,000, an alpha of 0.001, and beta of 0.01.
20

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Table 13: PARS AntMaze performance using single hyperparameter setting. We averaged the scores over five random seeds, with + indicating the standard deviation.
AutMaze
other SOTA
other SOTA (gual-condittned)
PARS
PARS (single hyperparam)
umaze
97.9 (MSG)
91.6 (GC-IQL)
93.8
93.3::3.2
umuze diverse
88.3 (ReBRAC)
88.8 (GC- (GC-IQL) IQL)
89.9
84.7:15.7
medium--pay
85.9 (MSG)
84,1 (HIQL)
91.2
89.4
medium diverse
84.6 (MSG)
86.8 (UTIQL))
92.0
86.2.1..6
large-play
64.3 (MSG)
86.1 (IIIQL)
77.511..8
large-divent
71.2 (MSG)
(HTQL.)
83.54:3.6
ultra-phng
22.4 (ReBRAC)
56.6 (GCPC)
66.4
60.2:17.6
ultra-diverse
(IQL)
54.6 (GCPC)
51.4
50.8:
Sensitivity analysis of hyperparameter. When applying PA, the hyperparameter in eq. (3) is important. It should be set such that assigning a min penalty to infeasible actions does not interfere with the TD update for actions in the dataset. Accordingly, we set a to small value in the range of 0.0001 to 0.001.
Table 14: PARS performance comparison across different values of a. We averaged the scores over five random seeds, with + indicating the standard deviation.
AntMuze
0.0001
0.001
a = 0.01
a 0.1
umazo-diverse
71,6=10.5
89.9+7.5
88.6=5.9
87.5=69
82.3+8.2
medium- diverse
12.5
92.0.1. 2.2
91.7113.
91.21288
40.51125.
large diverse
79.518.2
83.0:1:.9
83.21: 5.6
82.42288
16.111003
ultra-diverse
47.5::15.2
49.5:113.7
45.4:13.11
We additionally conducted sensitivity analysis 0.001 to 0.1. On the other hand, if is too small (e.g., 0.0001), the penalty does not take effect properly, and if it is too large, it starts to interfere with the TD update on the dataset, leading to a decrease in performance.
As shown in Table 14, PARS performs well when is in the range of
Impact the number of critic ensembles. As discussed in Section 4.3, PARS can be used in combination with a critic ensemble. We analyzed the effect of the number of critic ensembles on offline performance in Figure 17. As shown in the figure, while the impact minimal in the Antmaze domain, incorporating an ensemble the MuloCo and Adroit domains enables more stable learning and thus contributes to improved performance.
Antmaze www.aaddeeddee-222
mwtmmaz-meddu-dddeeeee eg Aitmaze-ange-dreesse
Antraaze carmaze-ulttaaae diverse-v0
40
60
60
20
Number of Critics
Number of Critics
Number of Cribics
Number of Critics
H Hallchhetah-rediumzz
Hopper-mediuman
Hemmer-.looeedvi
40
100
10
60 Number Critics
10
Number of Critics
Number of Critics
Figure 17: Final normalized score of PARS for offline training, averaged over five random seeds with varying numbers of critics. Error bars represent the standard deviation.
Which activation function would be most compatible with PARS? In Appendix D, we examine the effect of activation functions on fitting toy data. Additionally, we analyze how activation functions, when combined with PARS, influence performance in real RL tasks.
Figure 18 presents the results of applying each activation function to PARS, showing that the ReLU activation function consistently performs well across various tasks. While other activation functions outperform ReLU in some tasks, they lack robustness across all tasks. The impact of activation functions on RL. tasks, in conjunction with RS-LN and PA, could be an
21

---

Penalizing Infcasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
ReLU
Sigmold
SILU
Activation
Artmazz-umoze-diveee
Antmaze large vlay-ply
Antmaze play.vd
Antmaze- caamaae-utra-dlverreeae
100
500
60
Syradd MLUN
b:l Kctaaa
Talfcheetah--ediinaee
Hopper-medium-22
Walker2d-mediunne
80
80
40
20
Figure 18: Final normalized score of PARS for offline training averaged over five random seeds with varying activation functions. The error bars represent the standard deviation.
interesting topic for future research.
H. Computation Cost
We compared the training time and GPU memory usage of PARS with various offline baselines. The comparison was conducted using single L40S GPU, and the training time was measured over 5000 gradient steps. The baselines were implemented using either PyToorch or JAX. Given that JAX is generally recognized for its speed advantage over Py PoTcch due to optimizations like just-in-time compilation and efficient hardware utilization (Bradbury et al., 2018), distinguished the training time and GPU memory usage for PyTorch and JAX with yellow and blue bars, respectively. PARS, implemented in JAX, is indicated by a red bar.
Showing the comparison results presented Figure 19, PARS has faster training time compared to methods like SAC-N (PyTorch) and MSG (JAX), which use large number of critic ensembles. Additionally, while SAC-RND and ReBRAC have faster training times than PARS, they use significantly more GPU memory. In contrast, PARS efficiently reduces computation costs by using both less training time and less GPU memory.
200
Py T/Tarh
150
PARS (JAX)
125
Training
93.9
75
50
51.0
12.6
4.0
4.2
(JAX)
40000
GPU Memory Usage
35000
PyTorch
34548
34648
30000
JAX
PARS
(MIB)
25000
22423
20000
10000
5000
2220
2230
922
3345
582
TD3
COL
MCQ
SPOT
SAG-N
SAC-RND
PARS
(IAX) (JAX) (JAX)
(JAX)
Figure 19: Comparison of PARS's training time and GPU memory usage with various offline baselines. Yellow bars represent PyTorch implementations, blue bars represent JAX implementations, and the red bar represents PARS implemented JAX.
22


Reviews:
[{"review_id": "H5dfpAUcmV", "reviewer": "Reviewer_rMGd", "summary": "The manuscript introduces penalizing infeasible actions and reward scaling (PARS), a method for discouraging value overestimation caused by extrapolation in offline reinforcement learning (RL). The proposed method uses reward scaling and layer normalization, which are shown to work together to increase the feature resolution. This has the effect of reducing the similarity between gradient updates corresponding to actions found in the dataset and actions that lie outside the convex hull of “feasible actions”. In addition, the authors also propose to penalize out-of-distribution actions that lie far outside the region of feasible actions. Extensive benchmarking experiments demonstrate that the method consistently achieves superior performance compared to many baselines. Analytical experiments provide additional insights on the contribution of each component and the effects of hyperparameters.\n\n## update after rebuttal\nThe authors' detailed rebuttal provided important clarifications and additional experimental results which support the efficacy of the proposed method. I am now more confident in recommending the paper for acceptance and have accordingly raised my score to a 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive evaluation of our contribution and suggestions regarding various prior studies and potential directions for extending PARS.\n### [R1] Comparison of tuning search budget for hyperparameters with baselines\n\nAs noted in our manuscript, we tuned around 8 hyperparameter configurations, varying $\\alpha$, TD3+BC $\\beta$, and reward scale. For context, prior offline RL works we compare against also performed environment-specific tuning: ReBRAC used 25 configurations for MuJoCo and Adroit and 192 configurations (grid over 4 actor betas, 4 critic betas, 4 actor learning rates, and 3 critic learning rates) for AntMaze, SAC-RND used 19 (MuJoCo) and 9 (AntMaze), and SPOT, MSG, and MCQ used 6, 12, and 7, respectively. SAC-N and EDAC used 7 and 12.\n\nAs shown above, most of the prior works **we compare against adopt a similar level (some even more extensive) of hyperparameter tuning, which supports the fairness of our comparisons.** For TD3+BC, IQL, and CQL, we report results from their original papers (1–2 configurations), and ReBRAC [Ref.1] provides extensively tuned results for them—yet PARS still performs strongly.\n\nImportantly, as shown in Appendix G, PARS achieves strong AntMaze performance even with a **single** hyperparameter configuration, outperforming other baselines.\n\n&nbsp;\n### [R2] Motivation for PA\n\nAs discussed in Section 3.2, our goal is to encourage the learned Q-function to exhibit downward extrapolation for OOD actions that lie outside the convex hull of the dataset. To achieve this, RS-LN first increases the expressivity of the neural network, which naturally enables downward extrapolation by preventing gradient updates that would otherwise increase the Q-value positively for OOD actions. In addition, PA makes this process more explicit by penalizing infeasible actions, thereby enforcing downward extrapolation. As demonstrated in the ablation study in Figure 9, RS-LN has a significant effect on its own, but applying PA on top further enforces downward extrapolation, contributing stability and performance of offline RL training.\n\n&nbsp;\n### [R3] Relation to prior work\n\nThank you for pointing out the relevant prior work. We will include the study in the revision. In particular, R-BVE also aims to mitigate Q-function extrapolation errors using SARSA-based in-sample Q-learning and ranking loss, performing policy improvement only once at the end. However, this single-step approach may achieve insufficient performance improvement in complex environments or where rewards are sparse. While the ranking loss aligns Q-values based on high-reward trajectories, it risks performance degradation when applied to low-reward data, requiring additional mechanisms like soft filtering that increase implementation complexity.\n\nIn contrast, PARS directly addresses Q-value extrapolation, especially outside the convex hull of the data, by combining RS-LN and PA. This provides a stable foundation for repeated policy improvement and achieves both simplicity and superior performance.\n\n&nbsp;\n### [R4] Regarding policy churn\n\nThanks for the insightful suggestion. The referenced paper studies policy churn—sudden shifts common in value-based methods like DQN. In contrast, PARS builds on TD3+BC with a behavior cloning term in the actor, which helps suppress such changes. While this likely reduces policy churn, continual critic updates may still cause some churn, potentially aiding exploration. Quantitatively analyzing churn in critic-regularized methods like PARS would be a valuable direction for future work.\n\n&nbsp;\n### [R5] Implications for safe RL\n\nIn our study, we define infeasible actions as those lying outside the set of feasible actions. However, this definition can be extended by incorporating various constraints depending on the objectives of different tasks. In particular, by establishing clear criteria for unsafe infeasible actions and designing the system to penalize such actions accordingly, the framework can be naturally extended to safe RL.\n\n&nbsp;\n### [R6] Where layer normalization is applied\n\nWe applied LN after all linear layers, including the input layer, except for the final output layer.\n\n&nbsp;\n### [R7] Regarding figure 10 (left)\n\nAs the reviewer mentioned in another comment, a key issue during online fine-tuning is that changes in the distribution can lead to unstable learning. Since PA was applied during offline training, we observed that removing PA during online fine-tuning led to instability. On the other hand, maintaining PA throughout resulted in more stable learning and led to significant performance improvements.\n\n&nbsp;\n\n[Ref.1] Tarasov, Denis, et al. \"Revisiting the minimalist approach to offline reinforcement learning.\" NeurIPS 2023.\n\n&nbsp;\n\nWe sincerely appreciate your thoughtful review once again. We hope that our responses have thoroughly addressed all of your comments.", "claims_and_evidence": "* PARS successfully addresses the value overestimation problem caused by extrapolation in offline RL. This is well supported by better performance in experiments on many environments, including the challenging AntMaze Ultra and Adroit relocate-cloned\n* Penalizing infeasible actions during online fine-tuning is effective for stabilizing training. Shown in experiments on two difficult environments (Figure 10)", "methods_and_evaluation_criteria": "The proposed method is mostly well-motivated. The benchmarks show clear performance improvements over competing methods. The authors also present several ablation studies and analytical experiments investigating the effect of hyperparameters such as the reward scale or the distance of infeasible actions from the feasible action region.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "My main concern is whether the comparison with other algorithms is fair in the sense that several hyperparameters of PARS were tuned per environment. Is that the case for every score that you report from other papers and did you have the same search budget for hyperparameters of other methods that you ran yourself?", "relation_to_broader_scientific_literature": "The proposed method is novel and addresses the important problem of value overestimation due to extrapolation error in offline RL that has been studied before. It is the first to make a distinction of OOD actions that are within the convex hull of the actions in the dataset and those outside. Discouraging high Q-values for actions outside this region of “feasible” actions seems to be very effective at mitigating extrapolation errors.", "essential_references_not_discussed": "I think the following work is related:\n* Gulcehre, C., Colmenarejo, S. G., Wang, Z., Sygnowski, J., Paine, T., Zolna, K., ... & de Freitas, N. (2021). Regularized behavior value estimation. arXiv preprint arXiv:2103.09575.", "other_strengths_and_weaknesses": "Strength: Extensive analytical experiments that show efficacy of each contribution or support design choices such as the distance of sampled infeasible actions from the feasible action region (Figure 11)\n\nWeakness: I think the motivation for penalizing infeasible actions is not very strong. I understand that it probably helps to intentionally push down the Q-values at far regions of the action space, but I’d like to know your thoughts on why it improves the performance compared to only using reward scaling + layer normalization.", "other_comments_or_suggestions": "* I think it’d be worthwhile looking at how PARS affects policy churn (Schaul et al., 2022)\n* Can you comment on whether PARS has the potential to improve safe RL in the sense that it reduces sampling of actions that are far from the explored action space?\n\nSchaul, T., Barreto, A., Quan, J., & Ostrovski, G. (2022). The phenomenon of policy churn. Advances in Neural Information Processing Systems, 35, 2537-2549.", "questions_for_authors": "* Is layer norm only applied to the pen-ultimate layer, i.e. the one before the output weights, or to every layer?\n* In Figure 10 (left), the LN online fine-tuning seems to reduce performance. Why is that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rhn3nW7rHs", "reviewer": "Reviewer_snRz", "summary": "The paper introduces a method for applying an OOD  penalty using reward scaling and LN. When combined with a modified TD error loss—incorporating a PA penalty—the proposed approach enhances TD3-BC’s performance, achieving strong results on benchmarks, particularly in maze tasks.\n\nI quite like the paper, as it provides a new perspective by investigating how Q-function evaluation interacts with reward tuning and network design. The insights are intriguing and convincing. However, I noticed that many interesting discussions are deferred to the appendix. I suggest moving some of these discussions—such as the reasoning behind using TD3-BC instead of IQL—into the main text for better accessibility.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the positive feedback on our work and the opportunity to clarify any remaining uncertainties.\n### [R1] Theoretical justification of PARS\nThank you for the suggestions. As theoretical analysis of deep neural networks with nonlinearities is highly challenging, we empirically validated the effectiveness of RS-LN and leave theoretical aspects for future work. We will clarify this point in the limitations. As noted in Appendix B, we used ReLU due to its popularity and its frequent use in prior works, and exploring other activations better suited to RS-LN, especially with PA, is a promising future direction.\n\nWe can show the convergence of PARS, and we will add this to the paper. For this we define the following sets:\n- $\\mathcal{ID} = \\{(s, a) \\mid \\beta(a \\mid s) > 0\\}$\n- $\\mathrm{OOD}_{\\text{in}} = \\mathrm{ConvexHull}(\\mathcal{ID}) \\setminus \\mathcal{ID}$\n- $\\mathrm{OOD}_{\\text{out}} = \\left( \\mathrm{ConvexHull}(\\mathcal{ID}) \\right)^c$\n\nWe define the operator $T^\\pi_{\\mathrm{pars}}$ as:\n\n$\nT^\\pi\\_{\\mathrm{pars}} Q(s,a) =\n\\begin{cases}\nT^\\pi Q(s,a), & \\text{if } (s,a) \\in \\mathcal{ID}, \\\\\\\\\n\\mathbb{E}\\_{(s',a') \\in \\mathrm{kNN}(s,a; \\mathcal{ID})}[T^\\pi Q(s',a')], & \\text{if } (s,a) \\in \\mathrm{OOD}\\_{\\text{in}}, \\\\\\\\\nQ\\_{\\min}, & \\text{if } (s,a) \\in \\mathrm{OOD}_{\\text{out}},\n\\end{cases}\n$\n\nwhere $\\mathrm{kNN}(s,a; \\mathcal{ID})$ denotes the set of the k-nearest neighbors within $\\mathcal{ID}$.\nWe show that $T^\\pi_{\\mathrm{pars}}$ is a contraction under the$\\| \\cdot \\|_\\infty$ norm:\n\n- Case 1:$\n  \\left|T^\\pi_{\\mathrm{pars}} Q_1 - T^\\pi_{\\mathrm{pars}} Q_2\\right| = \\left|T^\\pi Q_1 - T^\\pi Q_2\\right| \\le \\gamma \\| Q_1 - Q_2 \\|_\\infty$\n\n- Case 2:$\\bigl|T^\\pi\\_{\\mathrm{pars}} Q_1 - T^\\pi_{\\mathrm{pars}} Q_2\\bigr| = \\left| \\mathbb{E}\\_{\\mathrm{kNN}}[T^\\pi Q_1] - \\mathbb{E}\\_{\\mathrm{kNN}}[T^\\pi Q_2] \\right| \\le \\gamma | Q_1 - Q_2 |_\\infty$\n\n\n- Case 3:$\\left| Q_{\\min} - Q_{\\min} \\right| = 0 \\le \\gamma \\| Q_1 - Q_2 \\|_\\infty$\n\nThus, $T^\\pi\\_{\\mathrm{pars}}$ is a contraction, and the Q-function converges accordingly. \n\nIn Case 2, computing $\\mathrm{kNN}(s,a; \\mathcal{ID})$ is costly. Since the Q-function is approximated by a neural network, we let it implicitly learn this behavior instead of adding it to the loss. For the same reason, assigning $Q_{\\min}$ can unintentionally affect nearby values. To prevent this, we introduce a **“guard interval”** that excludes actions just outside $\\mathcal{A}_F$ from being evaluated as $Q\\_{\\min}$. In practice, a guard interval of about 1000 (see Figure 11) is sufficient to avoid influencing Q-updates.\n\n&nbsp;\n\n### [R3] Regarding convex hull definition\nPARS aims to be a simple, efficient algorithm. While nonlinear boundaries are certainly possible, we opt for a convex hull to maintain simplicity.\n\n&nbsp;\n\n### [R4] Effectiveness of reward scaling in noisy environments\nAssuming the reward is added by noise, scaling the noisy reward increases both true reward and noise proportionally, leaving the signal-to-noise ratio unchanged. To investigate further, we referred to [Ref.2], which analyzes offline RL under reward noise. Following this, we examined how PARS performs in such noisy reward settings. As shown in **Table A of the link** (https://sites.google.com/view/pars-icml25), in the presence of reward noise **the trend of improved performance with increased reward scale still holds. So the core idea of PARS remains valid.**\n\n&nbsp;\n\n### [R5] Regarding presentation\nIn the revision, we will move Appendix D to the main body, possibly by shortening Fig. 1 and others.\n\n&nbsp;\n\n### [R6] Role of Q-ensemble under the presence of PA penalty\nAs noted in Appendix B, PARS primarily targets extrapolation errors outside the convex hull and does not explicitly address OOD actions within it. These actions may still face approximation errors, which could be mitigated through ensemble-based uncertainty estimation. However, as shown in **Table B (https://sites.google.com/view/pars-icml25)**, PARS outperforms SOTA in AntMaze even without an ensemble.\n\n&nbsp;\n\n### [R7] Dependency on policy expressiveness\nAs the reviewer noted, policy expressiveness is important, and methods like Diffusion-QL use diffusion models to enhance it. But, despite using diffusion, Diffusion-QL underperforms compared to PARS and suffers from the high inference cost of diffusion models. \n\nSince policies are derived from the Q-function, we argue that improving Q-network expressiveness is even more crucial. Our work focuses on this, and notably, PARS achieves strong performance with just a simple MLP policy. Ultimately, we believe that combining efforts to improve both policy and Q-network expressiveness is key to building better offline RL algorithms.\n\n&nbsp;\n\n[Ref.2] Yang, Rui, et al. \"Towards Robust Offline Reinforcement Learning under Diverse Data Corruption.\" ICLR 2024.\n\n&nbsp;\n\nThank you again for the careful review. We hope that our responses have sufficiently addressed the concerns.", "claims_and_evidence": "Most of the claims are supported by explanations and toy tasks. However, I believe the paper would benefit from additional theoretical analysis. Currently, the entire work appears too heuristic, as it lacks theoretical proofs or formally defined mathematical assumptions. Some claims, such as the necessity of a guard interval, are explained but not theoretically justified. \n\nI would be eager to see more theoretical insights if the authors could elaborate on the following points:\n\n1. Why do reward scaling, LN, and ReLU activation lead to good performance, while other combinations do not? Is there any theoretical justification for this choice?\n2. Why is a guard interval necessary, and how should its size be determined?\n3. With the modified TD loss and the designed $Q_{\\min}$, does this modification influence the convergence behavior of Q-value estimation?", "methods_and_evaluation_criteria": "The paper evaluates its method on well-accepted benchmarks.", "theoretical_claims": "No theoretical results are provided in this paper.", "experimental_designs_or_analyses": "I did not find any explicit issues with the experiments or toy examples.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The paper is novel and easy to follow. The visualizations are well-designed and provide valuable support in understanding the concepts presented. The analysis of interactions between reward scaling, LN, and ReLU offers an intriguing and fresh perspective.\n\nFor weaknesses, please refer to my previous section's content and the questions outlined below.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. The definition of OOD relies on the convex hull, but why is it defined this way? I assume the definition is inspired by action interpolation, but is this approach too restrictive? Would it be possible to consider a more flexible, nonlinear boundary for defining OOD actions?\n\n2. I have some concerns regarding reward scaling. In MuJoCo, the rewards are well-designed and provide clean signals. However, in most practical cases, rewards tend to be noisy, and scaling them up to a large magnitude (e.g., $10^4$) could induce **high variance**, potentially leading to training instability or even collapse. Do the authors have any insights on this issue?\n\n3. I find that Figure 1 may not be very informative. Could the authors consider replacing it or, alternatively, moving some discussions from the appendix into the main text? For instance, the discussion on IQL was particularly interesting to me.\n\n4. Q-ensemble is commonly used to mitigate Q overestimation, but is it redundant when using the PA penalty? While I see that the ensemble improves performance, does this imply that the PA penalty does not fully mitigate the overestimation issue? Do the authors have any insights into this?\n\n5. The success of PARS seems to depend on choosing TD3-BC, as it still allows sampling of OOD actions. However, if the policy has high expressive power and can accurately capture all in-sample actions—such as in the case of diffusion policies—would PARS still be effective? If not, does this mean that PARS is mainly beneficial for \"less expressive\" policies and may not help much for \"highly expressive\" policies?\n\nI am happy to raise my score if the authors can address my concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NaDYLs9Uge", "reviewer": "Reviewer_QMiG", "summary": "In this paper authors address Q-value overestimation problem in offline RL when in the presence of infeasible actions. Authors propose to use two diffferent strategies together, to scale reward and penalize infeasible actions. Layer normalization, as proposed in previous research, is also found to be important to use with reward scaling.\n\n## update after rebuttal\n\nI appreciate authors rebuttal, but taking into account rebuttal and other reviewers opinions I have decided to raise my score from 3 to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback and your constructive suggestions that allowed us to strengthen areas we may have initially missed.\n### [R1] Empirical evidence on Q-value overestimation reduction\n\nThanks for the suggestion. Measuring extrapolation error in high-dimensional state-action spaces is challenging, as it requires sampling outside the dataset’s convex hull—a non-trivial task in such high-dimensional settings. Therefore, following the approach in [Ref.3], we visualized the Q-values learned by applying LN, RS-LN, and RS-LN&PA on the Inverted Double Pendulum with a 1D action space. The visualizations can be found at **link (https://sites.google.com/view/pars-icml25). As shown in Fig A of the link**, and consistent with the discussion in Figure 5 of the manuscript, we observe that training the Q-function without LN leads to divergence. Applying LN prevents this divergence. Furthermore, when reward scaling and PA are also applied, the extrapolation curve bends downward while maintaining a similar overall shape within in-sample actions. We also observe that with reward scaling and PA, the average magnitude of Q-values decreases, suggesting a general reduction in overestimation.\n\n&nbsp;\n### [R2] Applicability to discrete action spaces\n\nIn our work, we focused on the problem of Q-function extrapolation error in continuous action spaces. In discrete action spaces, the action set is finite, making the notion of being “outside” the feasible set less clear—and upward extrapolation of the Q-function less relevant. Since many practical tasks, such as robot control, involve continuous control, and many offline RL algorithms target this setting, we also concentrate on continuous control. We believe that achieving strong performance in this setting, compared to other offline RL algorithms that also focus on continuous control, is a substantial contribution.\n\n&nbsp;\n\n### [R3] Sensitivity analysis of $\\alpha$ hyperparameter\n\nAs the reviewer pointed out, when applying PA, the hyperparameter $\\alpha$ in eq.(3) is important. It should be set such that assigning a $Q_\\text{min}$ penalty to infeasible actions does not interfere with the TD update for actions in the dataset. Accordingly, we set $\\alpha$ to a small value in the range of 0.0001 to 0.001. We additionally conducted a sensitivity analysis on $\\alpha$. As shown in **Table C of the link (https://sites.google.com/view/pars-icml25)**, PARS performs well when $\\alpha$ is in the range of 0.001 to 0.1. On the other hand, if $\\alpha$ is too small (e.g., 0.0001), the penalty does not take effect properly, and if it is too large, it starts to interfere with the TD update on the dataset, leading to a decrease in performance.\n\n&nbsp;\n\n### [R4] Missing y-axis labels in figures\n\nSorry for the confusion, and thank you for the helpful comment. We'll make sure to clearly specify the y-axis in the revised version.\n\n&nbsp;\n\n[Ref.3] Kim, et al. \"Adaptive q-aid for conditional supervised learning in offline reinforcement learning.\" NeurIPS 2024.\n\n&nbsp;\n\nOnce again, we sincerely thank you and hope that our responses have adequately addressed all of your concerns.", "claims_and_evidence": "Claims are clear, but as the claim is to show that Q-values are not overestimated after the proposed tricks, I do not see any extensive numerical evaluations where we could see what happens to Q-values with and without the proposed tricks. Final benchmarking results are of course shown, but it would be good to tie these things together more strongly.", "methods_and_evaluation_criteria": "Standard bencmarking datasets and environments are used and evaluation criteria as employed in those is applied. Only thing, as mentioned in the previous box is that the estimated Q-values specifically are not shown.", "theoretical_claims": "none", "experimental_designs_or_analyses": "Analyses provided in numerous Figures are instructive and well designed.", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "Paper is interesting and well written. It proposes a novel but simple tricks to make offline RL work in OOD case. Empirical resutls are also very strong. \n\nI have two main questions: \n1. infeasible set of actions as in Eqs. 1-2 work only for continuous actions. What happens with discrete actions? In more complex games number of discere actions can happen quite rarely. Leading to similar OOD case as in continuous actions. \n2. \\alpha paramerer in Eq. 2 seems to be an important one. But I do not see any experimental, nor theoretical analysis on how to set it.", "other_comments_or_suggestions": "- Many Figures in Section 6 do not have y-axis labels. Please mark them, sometimes it is not self evident whether it is a reward that is recorded.", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B2xxe4he29", "reviewer": "Reviewer_mnXg", "summary": "The paper addresses Q-value extrapolation errors in offline reinforcement learning (RL). It identifies linear extrapolation beyond the data range as a key issue and proposes two methods to mitigate it: (1) reward scaling with layer normalization (RS-LN) and (2) penalizing infeasible actions (PA). These components are integrated into a new algorithm, PARS, which is evaluated on the offline RL benchmark, showing state-of-the-art performance in offline and online fine-tuning, especially in the challenging AntMaze Ultra task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the careful review of our work, the clarification of various components of PARS, and your thoughtful suggestions.\n### [R1] Ablation study separating the effects of RS-LN and PA\n\nBeyond the ablation in Figure 9, we isolated PA in a separate experiment. As shown in **Fig B of the link (https://sites.google.com/view/pars-icml25).**\n\n- None (without LN or PA): Training becomes unstable as the reward scale increases.\n- PA: Helps mitigate extrapolation error, though its effect diminishes at larger reward scales.\n- LN: Performance improves with increasing reward scale.\n- LN and PA: Combining LN with PA further enhances performance by reinforcing downward Q extrapolation.\n\n&nbsp;\n\n### [R2] Hyperparameter justification\n\nAs the reviewer mentioned, $\\alpha$ in Eq. (3), the infeasible action distance, and the reward scale are important factors in PARS. The infeasible action distance is analyzed in Figure 11, reward scale in Figure 9, and $\\alpha$ in our response to Reviewer QMiG [R3].\n\n&nbsp;\n\n### [R3] Generalization beyond D4RL\nThank you for the helpful suggestion. Additionally, **we evaluated PARS on four environments from the NeoRL-2 [Ref.4] benchmark, which includes more complex and realistic problems**, covering a variety of real-world challenges such as delay, external factors, limited data, and safety constraints. As shown in **Table D of the link (https://sites.google.com/view/pars-icml25)**, PARS surpasses previous SOTA even under more realistic benchmark. \n\n&nbsp;\n\n### [R4] Action space dimensionality and PA behavior\nAs shown in Eq.(1), an action is considered infeasible in dimension $i$ if it falls outside predefined thresholds. Thus, $\\mathcal{A}_I$ is defined as the union of these infeasible regions across all dimensions. From this union, we sample the same number of infeasible actions as dataset actions per gradient update and assign them a penalty of $Q\\_{\\text{min}}$.\n\n&nbsp;\n\n### [R5] Computational efficiency of PARS\nIn Appendix H, we compared the computational cost of various offline RL algorithms, showing that PARS is more efficient in training time and GPU memory usage.\n\n&nbsp;\n\n### [R6] Comparison with uncertainty-based approach\nThanks for the insightful point. Both distributional and model-based RL incorporate forms of uncertainty—distributional RL models the distribution over Q-values, while model-based RL estimates uncertainty in learned environment dynamics. These uncertainties help with exploration and OOD detection, albeit often at the cost of added complexity and computation. PARS offers a simple and efficient solution built on TD3+BC, without complex uncertainty modeling, yet still achieves strong performance.\n\nWe see promise in combining PARS with uncertainty-based methods. In offline RL, penalties may need to differ between interpolation and extrapolation regions, as the latter, lacking clear bounds, can lead to more severe value errors. Penalizing OOD while encouraging downward extrapolation could improve robustness. Similarly, uncertainty-guided exploration during fine-tuning, as reviewers suggested, may boost performance. We'll incorporate these insights in the revision.\n\n&nbsp;\n\n### [R7] Applicability to stochastic environments\n\nEven in stochastic settings, actions are generally confined within a certain min and max range. Thus, the span defined by these bounds can be regarded as the feasible region. Moreover, instead of sampling infeasible actions that lie just outside the feasible action set, we sample from a sufficiently distant region, using a guard interval. This allows PARS to function without precise knowledge of the feasible-infeasible boundary, supporting robust performance under stochastic dynamics.\n\n&nbsp;\n\n### [R8] Extension to policy extrapolation\n\nWe're not entirely sure what the reviewer means by policy extrapolation, but we interpret it as the policy incorrectly predicting actions for given states. Since the policy is trained via the Q-function, reducing Q-function errors may help mitigate policy errors. However, as Q-function extrapolation and OOD action selection are distinct, directly applying PARS to improve policy accuracy is not straightforward. If we have misunderstood, we would be grateful if the reviewer could kindly clarify.\n\n&nbsp;\n\n### [R9] Effectiveness of PARS under sparse or varying reward scales\n\nAmong our evaluated domains, AntMaze meets the reviewer’s criteria with its sparse reward setting. As shown in Figure 9, performance improves with larger reward scales, highlighting a clear advantage over other baselines. We also analyzed the case where noise is added to the reward; for details, please refer to our response to reviewer snRz [R4].\n\n&nbsp;\n\n[Ref.4] Gao, Songyi, et al. \"NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios.\" arXiv preprint (2025).\n\n&nbsp;\n\nThank you once again for your thoughtful review. We hope that our responses have thoroughly addressed your concerns.", "claims_and_evidence": "The paper claims that linear extrapolation in Q-functions leads to overestimation and that RS-LN and PA can effectively mitigate this. Empirical results on the D4RL benchmark support these claims, showing that PARS consistently outperforms baselines.\n\nHowever, the evidence could be strengthened by additional ablation studies isolating the effects of RS-LN and PA.", "methods_and_evaluation_criteria": "The methodology is well-grounded, leveraging standard RL benchmarks and evaluation metrics.\n\nHowever, the choice of hyperparameters for PA and RS-LN could be better justified. Additionally, the comparison to other offline-to-online RL methods would benefit from a discussion on computational efficiency.", "theoretical_claims": "The theoretical analysis of Q-function extrapolation is sound, particularly the discussion of layer normalization’s role in bounding Q-values.", "experimental_designs_or_analyses": "The experiments are extensive and cover a broad range of RL tasks. \nBut it would be beneficial to explore whether PARS maintains its advantages when applied to more complex tasks beyond D4RL.", "relation_to_broader_scientific_literature": "The paper effectively situates its contributions within the literature on offline RL and critic regularization.", "essential_references_not_discussed": "The paper does not discuss recent advancements in distributional RL, such as IQN or QR-DQN, which could offer alternative solutions to Q-value extrapolation. Additionally, references to prior works on offline-to-online RL transitions, such as model-based fine-tuning strategies, would be benificical.", "other_strengths_and_weaknesses": "Strengths:\n\n* A well-motivated approach to mitigating Q-function extrapolation errors.\n\n* Strong empirical performance across diverse RL tasks.\n\n* Simple yet effective implementation with minimal computational overhead.\n\nWeaknesses:\n\n* Lack of ablation study of RS-LN and PA’s effectiveness.\n\n* Limited discussion on computational complexity.\n\n* No exploration of PARS’s applicability to other offline RL benchmarks or real-world tasks.", "other_comments_or_suggestions": "Clarify how the infeasible action penalty interacts with different action space dimensionalities.", "questions_for_authors": "* How does the computational cost of PARS compare to other offline RL methods in terms of training time and memory usage?\n\n* Can RS-LN and PA be effectively combined with model-based offline RL approaches?\n\n* How does PARS perform on tasks with highly stochastic dynamics, where infeasible actions are less clearly defined?\n\n* Could the proposed approach be extended to address policy extrapolation errors, in addition to Q-function extrapolation?\n\n* How robust is PARS to changes in the reward scale, especially when dealing with environments that contain highly varying reward magnitudes or sparse returns?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces PARS, an algorithm that aims to bridge the gap between offline reinforcement learning and online finetuning. Concretely, the authors introduce a method that is a strong offline algorithm which is also amenable to online finetuning, and show that this approach greatly outperforms prior work in a large number of domains with minimal hyperparameters. The work achieves this by building on existing work using layer norm that penalizes OOD actions (e.g., RLPD), adding reward scaling and explicit action penalization in infeasible regions.

The reviewers were very positive on this work, and highlighted its very strong empirical results and interesting and novel insights into value learning in RL more broadly. The authors also had a strong rebuttal, which helped eliminate concerns that the reviewers had around experiments and theoretical insights.

As a result, this is clearly a strong work that pushes the SoTA on an important topic on the RL community, and therefore should be accepted into the program.
Decision: Accept (spotlight poster)