Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
Authors: Yixiu Mao
Cheems Wang
Chen Chen
Yun Qu
Xiangyang Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: offline reinforcement learning
Primary area: reinforcement_learning
TL;DR: This work systematically analyzes the underexplored OOD state issue in offline RL and proposes a simple yet effective approach unifying (value-aware) OOD state correction and OOD action suppression.

Abstract: In offline reinforcement learning (RL), addressing the out-of-distribution (OOD) action issue has been a focus, but we argue that there exists an OOD state issue that also impairs performance yet has been underexplored. Such an issue describes the scenario when the agent encounters states out of the offline dataset during the test phase, leading to uncontrolled behavior and performance degradation. To this end, we propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL. Technically, SCAS achieves value-aware OOD state correction, capable of correcting the agent from OOD states to high-value in-distribution states. Theoretical and empirical results show that SCAS also exhibits the effect of suppressing OOD actions. On standard offline RL benchmarks, SCAS achieves excellent performance without additional hyperparameter tuning. Moreover, benefiting from its OOD state correction feature, SCAS demonstrates enhanced robustness against environmental perturbations.

Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
Yixiu Mao', Qi Wang', Chen Chen', Yun Qu', Xiangyang Ji'
DDepartment of Automation, Tsinghua University myx210mails wtt.::mmmmss....nnniiiem tsinghua. edu. en, xyji@tsinghua edu, cn
Abstract
In offline reinforcement learning (RL), addressing the out-of-distribution (OOD) action issue has been focus, but argue that there exists OOD state issue that also impairs performance yet has been underexplored. Such an issue describes the scenario when the agent encounters states of the offline dataset during the test phase, leading to uncontrolled behavior and performance degradation. To this end, we propose SCAS, simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL. Technically. SCAS achieves value-aware OOD state correction, capable of correcting the agent from OOD states to high-value in-distribution states. Theoretical and empirical results show that SCAS also exhibits the effect of suppressing OOD actions. On standard offline RL benchmarks, SCAS achieves excellent performance without additional hyperparameter tuning. Moreover, benefiting from its OOD state correction feature, SCAS demonstrates enhanced robustness against environmental perturbations.
Introduction
Deep reinforcement learning (RL) shows promise in solving sequential mabsiinn-making problems, gaining increasing interest for real-worke applications [42, 57, 63, 53, 7]. However, deploying RL algorithms in extensive scenarios poses persistent challenges, such as risk-sensitive exploration [13] and time-consuming episode collection [27]. Recent advances view offline RL as a hopeful solution to these challenges [34]. Offline RL aims to learn policy from a fixed dataset without further interactions [32]. It can tap into existing large-scale datasets for safe and efficient learning [23, 37, 50].
In offfine RL research, a well-known concern the ootoo--distribution (OOD) action issue: the evaluation of OOD actions causes extrapolation error [12], which can be exacerbated by bootstrapping and result in severe value overestimation [34]. To address this issue, a large body of work has emerged directly indirectly suppress OOD actions during training, employing various techniques such as policy constraint [12, 30, 10]. value penalization [3 2, 6]. and in-sample learning [29, 14, 71].
Distinguished from most previous works, this paper argues that, apart from the OOD action issue, there exists an OOD state issue that also impairs performance yet has received limited attention in the field. Such an issue refers the agent encountering states of the offline dataset during the policy deployment phase (i.e., test phase). The occurrence of OOD states can be attributed to OOD actions, stochastic environments, and real-world perturbations. Since typical offline RL algorithms do not involve policy training in OOD states, the agent tends to behave in an uncontrolled manner entering OOD states in the test phase. This can further exacerbate the state deviation from the offline dataset and lead to severe degradation in performance [34, 75].
In mitigating this OOD state issue, existing limited work attempts train the policy to correct the agent from OOD states to in-distribution (ID) states [75, 22]. Technically, Zhang et al. [75] construct dynamics model and state transition model and align them to guide the agent to ID regions, while
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

State Distribution
State Distribution
State Distribution
Optimal Value of Each State
(a) CQL
(b) TD3BC
(c) SCAS (Ours)
(d) Optimal Value
Figure I: The resulting state distributions of offline RL algorithms and optimal values of states. (a,b,c) The state distributions generated by the learned policies of various algorithms compared with that of the offline dataset on Sifcchetaa-medium-eee
(d) The corresponding optimal value of each state, which is obtained by running TD3 online convergence. SCAS- state distribution almost entirely within the support of the offline distribution and avoids the low-value areas, while CQL and TD3BC tend to produce OOD states with extremely low values.
Jiang et al. [22] resort to an inverse dynamics model for policy constraint. However, they deal with the OOD state and OOD action issues separately, requiring extra OOD action suppression components and complex distribution modeling, which sacrifices computational efficiency and algorithmic simplicity, Moreover, correcting the agent to all ID states impartially could be problematic, especially when the dataset contains substantial suboptimal states. As a result, the performance of prior methods also leaves considerable room for improvement.
In this paper, we aim to address these two fundamental OOD issues simultancously by proposing simple yet effective approach for offline RI. We term method SCAS due to its integration of OOD State Correction and OOD Action Suppression. We start with solving an analytical form of a value-aware state transition distribution, which is within the dataset support but skewed toward high- value states. Then, we align it with the dynamics induced by the trained policy on perturbed states via KL divergence. This operation intends to correct the agent from OOD states to high-value ID states, concept we refer to as value-aware OOD state correction. Through some derivations, it also eliminates the necessity of training a multi-modal state transition model. Furthermore, show theoretically and empirically that, while designed for OOD state correction, SCAS regularization also exhibits the effect of OOD action suppression. We evaluate SCAS on the offline RL benchmarks including D4RL [9] and NeoRL [49]. SCAS achieves excellent performance with consistent hyperparameters without additional tuning. Moreover, benefiting from its OOD state correction ability, SCAS demonstrates improved robustness against environmental perturbations.
To summarize, the main contributions of this work are:
We systematically analyze the underexplored OOD state issue in offline RL and propose simple yet effective approach SCAS unifying OOD state correction and OOD action suppression.
Our approach achieves value-aware OOD state correction, which circumvents modeling complex distributions and significantly improves performance over vanilla OOD state correction methods.
Empirically', our approach demonstrates superior performance on standard offline RL benchmarks and enhanced robustness in perturbed environments without additional hyperparameter tuning.
2 Preliminaries
In reinforcement learning, we generally characterize the environment Markov Decision Pro- cess (MDP) M = (S. R, 7, do), with state space S, action space A, transition dynamics P:S x A + (SS), reward function R : S A -+ IR, discount factor 7 E [0, 1), and initial state distribution do [61]. The agent interacts with the environment and seeks a policy л S - A(A) to maximize the expected discounted return (7(:)
(T(T) Beg (ot.
* R(st, az)
(1)
'Our code is available hutps://gthubbcoommmmyyyxxuuuua

---

For any policy т, we define the value function as VT(s) = Ex 2 R(s,, ai) and the state-action value function (Q-value function) as Q" (s, a) = Ee Er= 70 R(s,, az)] 80 = 8, 00 a).
Offline RL. In offline RL, the agent can only access a static dataset D = {s] aj. s4+1: We denote the empirical behavior policy of D by 3(a)s) and the empirical dynamics model by (s' a). both of which depict the conditional distributions observed the dataset [12]. Typical actor-critic algorithms [56, 18] evaluate policy π by minimizing Bellman loss:
(0) ('s,a,''>>'ll ((Q))))__ssssseoeee) a) R(s, a) -Bo x 20 (s',aa))))
where Te and Qe are the parameterized policy and Q function, and Qo is a target network whose parameters updated via Polyak averaging [42].
Simultaneously, policy improvement in policy iteration is achieved via maximizing the Q-value:
La (6) = -BawD,awTy (Qa (s, a)].
OOD action issue. In offline RI., OOD actions refer to actions outside the support of the behavior policy ((-ss) at a specific state DD Since the Q-values of OOD actions can be poorly estimated and the policy improvement is towards maximizing the estimated Qe. the resulting policy tends to prioritize the OOD actions with overestimated values, leading to poor performance [12].
3 OOD State Correction
The following focuses on the OOD state issue and OOD state correction in offline RL. In Section 3.1, we systematically analyze the OOD state issue, introduce the concept of OOD state correction, and point out limitations of prior methods. Then we present the proposed approach SCAS in Section 3.2.
3.1 OOD State Issue in Offline RL
In offline RL, OOD states refer to states not in the offline dataset. The OOD state issue (Definition 1) pertains to scenarios where the agent enters OOD states during the test phase, potentially resulting in catastrophic failure [34]. However, such topic is rarely investigated in the literature, and existing studies lack deep insights. We mathematically formulate the OOD state issue as follows.
and dp(s) = 0, where MT is the MDP of the test environment, is any learned policy, d'Mr is the state probability density induced by x in My, and dp the state probability density in the offline dataset.
Definition 1 (OOD state issue). There exists S, such that xx
Origins and consequence of OOD states. During the test phase, the OOD states occur primarily in three scenarios: (i) OOD actions: the learned policy, not perfectly constrained within the support of the behavior policy, executes unreliable OOD actions, leading to OOD states. (ii) Stochastic environment: the initial state of the actual environment may fall outside the offline dataset. In addition, stochastic dynamics can also lead to states outside the dataset, even when taking ID actions in ID states. (iii) Perturbations: commonly seen real-world robot applications, some unexpected perturbations can propel the agent into OOD states (e.g., wind, human interference).
During offline training, the typical Bellman updates involve only ID states, and the policies in OOD states are not trained. As a result, when encountering OOD states in the test phase, the agent would exhibit uncontrolled behavior, and the state deviation from the offline dataset can be further exacerbated over time steps, severely degrading performance [34].
OOD state correction. To mitigate this OOD state issue, intuitive solution is to train a policy capable of correcting the agent from OOD states to ID states, a concept known as OOD state correction [75]. Specifically, during offline training, can perturb the original state in the dataset into & to generate substantial OOD states. Then consider the scenario where the agent starts from s. follows the trained policy π. and transitions to the next state 8'. To reduce state deviation, &' is expected to be close to the offline dataset. Thus we can align the distribution of 3' with an ID state distribution regularize the policy and achieve OOD state correction.
Continuing the above train of thought, SDC [75] generates the state distribution by feeding the original state into trained state transition model N
(s'(s) (s|ss of the dataset. This model characterizes
3

---

the conditional state transition distribution in the dataset and is implemented by conditional variational auto-encoder (CVAE) [58]. After pretraining a dynamics model (s'sss (s, a) and the state transition model (s)))s). SDC introduces the following policy regularizer for OOD state correction:
min E
E (m))(
[MMD(M(118, (-11)), (-1+)))
BnDD
where & a Gaussian noise perturbed version of the original state 8, a is the standard deviation of the Gaussian, M-16,, =(-18)) is shorthand for (a (-15,, ()) and MMD is the maximum mean discrepancy measure. More recently, OSR [22] directly aligns the trained policy distribution at the perturbed state with CVAE inverse dynamies model constrain the policy in OOD states.
Limitations. However, the regularizers of prior methods are only designed to deal with this OOD state issue. To mitigate OOD actions, they require additional conservative Q learning (CQL) term [31] in value estimation to penalize Q-values of OOD actions. In addition, the state transition distribution and the inverse dynamics distribution multi-modal in many scenarios [43]. The neces- sity of extra OOD action suppression components and complex distribution modeling compromises their computational efficiency and algorithmic simplicity. Moreover, correcting the agent to all II) states impartially could be problematic, particularly when the offline dataset contains large portion of suboptimal states. In such cases, vanilla OOD state correction can lead to suboptimal behaviors. Consequently, there is also significant potential for improvement in the performance of prior methods.
For more comprehensive discussion of related work, please refer to Appendix A.
3.2 Value-aware OOD State Correction
The objective of this work is to formulate a simple yet effective policy regularizer for offline RL that unifies OOD state correction and OOD action suppression. Moreover, we aim to achieve value-aware OOD state correction, involving the correction of the agent from OOD states high-value ID states.
Value-aware state transition. For the ID state distribution to which the agent is corrected, we expect value-aware state transition distribution (-Is) that lies within the support of the dataset state transition distribution (-1ss) but is skewed toward high-value states s'. To ensure stability and, more importantly, to enable subsequently designed algorithm to circumvent modeling complex distributions, we seek a soft optimal version of it. To this end, we consider the following problem²:
max N
E a
E (11>-(1))
V(8') DKL (N*sS))
(5)
where a is a hyperparameter to balance the two terms.
The optimization problem above has a closed-fom form solution:
(8'1))) 7(5) exp (aV (s'))
(6)
where Z(s)= = = E. exp (aV (s')) (s*ls) a normalization factor. can be seen from Eq. (6) that supp(N (-Is) S (upp( Note that is a key hyperparameter that controls the significance of the values of next states in SCAS's ' OOD state correction. As a increases, N (-Is)) becomes more skewed toward the optimal s' in the support of (-1s))
OOD state correction. In order to produce substantial OOD states, we perturb each state s E D with Gaussian noise (00 o²), resulting in perturbed state s. It is worth noting that the dataset used for RL training remains unchanged. We perturb the states solely to formulate the regularizer.
We anticipate the following value-aware OOD state correction scenario, where the agent starts from OOD state 8, follows the trained policy æ, and transitions to the high-value ID state s' in the distribution of (-Is).. To this end, we train the policy to align the dynamics induced by π on the perturbed state å with the value-aware state transition distribution the original state s via KL divergence. That is, we regularize π by minimizing:
min E SDD (o)))(
( (-18,,((181))))) (((1))).
(7)
NNote that the regularizer DKL (N' (-Is)] can constrain the support of N (-Is) within that of N (-Is). because if supp(NV" (-ps()" & Supp(N (- (-Iss) at some state s, then DKL(N (N (-Is))] cc.

---

By substituting the analytical solution of N from Eq. (6) into the KL divergence, we have
exp (aV (s))
argmin (Ms = arginax
E
log (5
of ((1)())))
7(8)
Note that N is the state transition distribution in the dataset, and s' (-1s) is equivalent to (8, 8') ~ "D. Thus minimizing Eq. (7) is equivalent to maximizing following regularizer:
exp (aV (s'))
R(x)
E (5.5" (s.5")~ (
EE
log M ()),)
(8)
((s)
As a result, R(x)) effectively eliminates the need for pre-trained multi-modal state transition model (N or N and enables direct sampling from the dataset for optimization.
However, the normalization factor (s)) in R(T) can be challenging to compute. We note that the regularizer R(T)) is derived from the minimization of the KL divergence in Eq. (7). Since we aim to minimize this KI. at every state 8 in D and (s) only affects the relative weights at different 8, matters less precisely restore the correct state weights in by computing (s), which is empirically hard to estimate and may bring more instability. Thus, we replace ((s) in R(T)) with an empirical normalizer exp(aV(s)) for computational stability:
exp (aV (s')) Bí (T) (s,s')~D (s) exp (a.V (x)) log M M(5'(8
We provide further rationale behind this choice of the empirical normalizer in Appendix C.I.
Tractable optimization. Now we shift focus the optimization of Rı (T)). The expectation with respect to can be moved outside the logarithm by Jensen's 's inequality:
exp (aV (s')) exp (aV (s))
(1 2
E
E
E
log M ((s
(10)
(8,8") No (-18(-18)
where the equality holds when ग is deterministie. In general, it is convenient to maximize the lower bound in Eq. (10) using the reparameterization trick. However, ensure the equality case Eq. (10), we opt to train deterministic policy In this case, we can directly maximize Rí (T) by computing the gradient of æ using automatic differentiation [46].
In contrast to model-based RL methods that typically use the learned dynamics model to roll out multi-step trajectories for policy training [20, 73], our algorithm utilizes the dynamics model to propagate the gradient of policy and regularize policy training, resulting in significantly enhanced computational efficiency. Moreover, the nature of one-step dynamics prediction in our method is advantageous for maintaining relatively high prediction accuracy.
4 Analysis of OOD Action Suppression
This section focuses on the OOD action issue and shows that the proposed regularizer also exhibits the effect of OOD action suppression. In other words, can also prevent the policy from taking OOD actions, thereby simultaneously addressing the fundamental OOD action issue in offline RL. In offline RL, OOD actions are exclusively defined on ID states. This is because actor-critic training is limited to ID states, and any actions OOD states would not affect training and cause the OOD action issue mentioned in Section 2. Consequently, for the analysis of OOD actions, is essential to R and R2 can be regarded as special cases of R and R1, when sampled from A o²) is equal to s:
consider ID states. We define It R₁ the ID state version of R, R2, where
cxp (aV (s'))
R(T) = E (~D
log M (s'ss, ((|s))
(11)
Z(s)
exp (aV (s'))
RT(T) = E
log M(s'ls, (-
(12)
(~D
exp (aV (s))
5

---

The proposed regularizer functions as follows: when the agent encounters OOD states, it drives the agent to choose actions leading to ID states, as discussed in Section 3.2. When the agent in ID states, the II) state part of it comes into play. In the following, we show that it helps circumvent taking OOD actions by analyzing the maximizer of R, R₁ in tabular MDPs.
Proposition 1. Suppose that the environment dynamics is deterministic, then both R((x)) and Rí (x) achieve their global maximum at the policy x², where"
' (als)
Z(s) exp (aV (M (M))) B(als)
(13)
The support of πt" is within that of the behavior policy 3: supp(#* (-1s)) S supp(3(-1s)), Ys D
(14)
and x² makes the following equation hold:
(+1s)) M(-|s, T" (+Is), Vs D
(15)
Under the deterministic dynamics condition, Proposition 1 shows that n² is constrained within the support of the behavior policy. Thus, regularizer helps to keep the policy from taking OOD actions. Moreover, x" is able to exactly align (-Iss, x (-11)) with (+]s), indicating the guidance of the agent to the high-value IDD state distributions.
Furthermore, we show in Proposition that even under stochastic dynamics, the optimization of R and R, still yields policies constrained within the support of B. Hence, SCAS also exhibits the effect of OOD action suppression this more general scenario.
Proposition 2.
When the dynamics is stochastic, the maximizers of both R(x)) and Rí (T) are constrained within the support of the behavior policy:
supp(x* E supp(3(-1s)), Vs D (-pp(=]((1s)) C supp(3(11)), As D
(16)
(17)
5
Implementation Details
SCAS is easy implement and we design the practical algorithm to be as simple as possible, retaining algorithmic simplicity and improv- ing computational efficiency.
Algorithm SCAS
1: Initialize dynamics model Ma., policy network
Th. Q-network Qo, and target Q-network Qo
// Dynamics Model Training
Dynamics model. We employ determinis- tic dynamics model M.. The loss for training the model is
3: for each gradient step do
4: Update by minimizing (w) in Eq. (18)
5: end for
// Policy Training
[m(w)=
E (s,0,8") (s, (18)
7: for each gradient step do
8: Update by minimizing LQ (8) in Eq. (2)
Policy improvement. With a deterministic model, we replace the log-likelihood in Rg (r) with mean squared error. It is a common ap- proach in RL algorithms to convert a maxi- mum likelihood estimation problem into gression problem when dealing with Gaussians with fixed variance [10]. As discussed in Section 3.2, also adopt deterministic policy model MT. Thus, we have the following policy regularizer:
9: Update & by maximizing (Ø) in Eq, (00)
10: Update target network: 0' +- (1 +)0 + 70
11: end for
exp (aVV (s'))
R2(mØ) = I. E (s.5')~ N.)
exp (aV, (s)) [M (5)) s'113
(19)
where (s(s) Qo(s, (s)) and 7 means Ts with detached gradients. Using deterministic policy also simplifies the training process without learning V-function. Combining R2(TT) with the standard policy improvement objective, we update the policy by maximizing:
(Ø) = (1 A)E,~D [Qo (s, T (s)) + AR2(TT))
(20)
'Here for clarity, we use the notation M with slightly different meanings in different cases: in the stochastic setting, M S x A + A(S): in the deterministic setting, M S x A + S.

---

where À is hyperparameter to balance the two terms. Additionally, following TD3+BC [10], we also normalize Qo in the first term each mini-bated maintain balanced scale across tasks. Overall algorithm. Putting everything together, we present our final algorithm in Algorithm
6
Experiments
In this section, we conduct several experiments to examine the performance and properties of SCAS. Please refer to Appendices D and E for experimental details and additional results.
6.1 Empirical Evidence of OOD State Correction and OOD Action Suppression
OOD state correction. To examine the OOD state correction ability, we compare the state distri- butions generated by the learned policies of different algorithms with the state distribution of the offiine dataset. In detail, first train SCAS, CQL [31], and TD3+BC [10], and then collect 50,000 samples by running the trained policies separately. We also sample 50,000 states randomly from the offline dataset for comparison. Figures ((a) I(c) plot the state distributions in halfcheetah-mediumm-
expert [9] with t-SNE [62], and Figure (dd) visualizes the optimal value of each state. We access these values from the learned value function obtained by running TD3 [11] online convergence.
In Figures 1(a) and 1(b), we observe that the policies learned by CQL and TD3+BC tend to produce OOD states. As depicted in Figure 1(d), these OOD states have extremely low values. entering them can be detrimental to performance. In contrast, the state distribution induced by SCAS is almost entirely within the support of the offline distribution, demonstrating the OOD state correction ability of SCAS. Moreover, we also note that in the low-value area of the offline state distribution (the grey circle in Figure (d)), SCAS exhibits very low state density, which could be attributed to SCAS's value-aware OOD state correction. We refer the reader to Appendix E.2 for additional experiments validating the OOD state correction effects.
OOD action suppression. We empir- ically evaluate the OOD action sup- pression effects through the lens of value estimates. We compare SCAS with three baselines: (1) ordinary off- policy RL which is SCAS with = 0 (all other implementations are the same); (2) SDC [75] without additional
hopper-medium-v2
10
homper-meduum-oppette
10
SCAS: MC return
SCAS: Q
OfF-policy RL: Q
10
SDC wio CQL: Q
10
OSR win CQL:
10"
CQL [31] term suppress OOD ac- tions; (3) OSR [22] without additional CQL term. We conduct experiments on D4RL datasets [9]. Since value over-estimation (divergence) is the main consequence and evidence of OOD ac- tions [12), we plot the learned Q-values of SCAS and the baselines in Figure 2.
0.1 Grudient Steps (x10*)
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4 0.5
Gradient Steps (x 10%)
Figure 2: Oracle Q-values of SCAS (estimated by MC return) and learned Q-values of SCAS and other algo- rithms across optimization steps. Only SCAS's OOD state correction term can achieve OOD action suppression and prevent value over-estimation (divergence).
We also include the oracle Q-values of SCAS by rollouting the trained policy for 1, 000 episodes and evaluating the Monte-Cand Carlo return. Additional results are provided in Appendix E.
The results show that the learned Q-values of ordinary off-policy RI, SDC without CQL, and OSR without CQL. diverge at early learning stages, suggesting that the algorithms suffer from severe OOD actions. By contrast, the learned Q-values of SCAS stay close to the oracle Q-values. This indicates that SCAS regularization alone is able to suppress OOD actions.
6.2 Comparisons Offline RL Benchmarks
Tasks. We evaluate SCAS on D4RL [9] and NeoRL [49] benchmarks. In D4RL, we conduct experiments on Gym locomotion tasks and much more challenging AntMaze tasks. Due to the space limit, the results on NeoRL are deferred to Table 4 in Appendix E.3.
7

---

Table 1: Averaged normalized scores on Gym locomotion and AntMaze tasks over five random seeds.
Dataset (v2)
BC
MОPO
OneStep
TD3BC
CQL
IQL
OSR
SDC
SCAS (Ours)
halfcheetah-med
42.0
73.1
48.3
47.0
47.4
45.1+-088
45.9++0.5
46.6--0.2
hopper-med
56.2
38.3
87.5
59.3
53.0
66.2
62.0+3.6
64.7+33.5
102.5+0.3
walker2d-med
71.0
41.2
84.8
83.7
73.3
78.3
80.1+1.8
82.7+119
82.3+3.0
halfcheetah--med-ry
36.4
69.2
42.7
44.6
45.5
44.2
43.31:0.2
45.1:0.5
44.0-10.3
hopper-med-rep
21.8
32.7
98.5
60.9
88.7
94.7
42.1112.3
94.8.16.5
101.6110
walker2d-mee--rep
24.9
73.7
61.7
81.8
81.8
73.8
78.11188
78.5.6.00
78.114.55
srrrchectah-med-ooe
59.6
70.3
75.1
90.7
75.6
86.7
63.7.11.55
76.3#5.2
91.7=2.77
hopper-med-exp
51.7
60.6
108.6
98.0
105.6
91.5
78.9+16.4
99,9
109.7=3.5
walker2d-med-exp
101.2
77.4
111.3
110.1
107.9
109.6
108.1+4.4
109.2#1.4
108.4=3.7
halfchectah-rand
2.6
35.9
11.0
17.5
13.1
1.611..1
14.2.10.7
12.2-13.2
hopper-rand
4.1
16.7
5.6
8.5
7.9
7.9
3.74:2.6
3.142.8
0,,1101
walker2d-rand
1.2
4.2
6.9
1.6
5.1
5.4
-0.1+.0.0
0.240.4
1.411.1
locomotion total
472.7
593.3
735.4
698.5
708.9
718.8
606.7
714.6
810.1
antmaze-umaze
66.8
0.0
54.0
73.0
82.6
89.6
87.4+5.0
81443.8
90.4-4.33
antmaze-umaze-divv
56.8
0.0
57.8
47.0
10.2
65.6
55.618.0
49.6.110.4
63.8 16.7
antmaze-med-plaa
0.0
0.0
0.0
0.0
59.0
76.4
22.617..6
55.011..6
76.611.9
antmaze-med-div
0.0
0.0
0.6
0.2
46.6
72.8
19.6455.8
56.61110.3
5.44
antmaze-large-pla
0.0
0.0
16.4
42.0
0.0.10.0
20.8
49.0
antmaze-large
0.0
0.0
0,2
0.0
3,2
46.0
0.0:00.0
25.847.5
50.6=7.2
antmaze total
123.6
0.0
112.6
120.2
218
392.4
185.2
289.2
410.8
runtime
30m
900m
120m
60m
250m
100m
300m
420m
140m
hyperparameter tuning
w/o
w/
w/o
w/o
w/
w/o
w/
w/
w/o
Baselines. We compare SCAS with prior state-of-the-arr offline RL methods as well as the ones
specifically designed for OOD state correction, including BC [48], MOPO [73], OneStep RL [5], CQL [3]], TD3+BC [10], IQL [29], SDC [75] and OSR [22].
Hyperparamter tuning. Offline RL methods are appealing for their ability to generate effective policies without online interaction. Nevertheless, many existing offline RL works involve dataset- specific hyperparameter tuning. The reduction of hyperparameter tuning is crucial for improving practical applicability. In this work, SCAS uses a single ser of hyperparameters for all datasets in D4RL and NeoRL benchmarks to obtain the reported results.
Comparisons with baselines. On D4RL,, comparisons of performance, runtime, and hyperparameter tuning information are shown in Table 1. We refer the reader to Appendix E.8 for learning curve details of SCAS. On the Gym locomotion tasks, SCAS outperforms prior methods on most datasets and achieves the highest total score with a single set of hyperparameters. On the challenging AntMaze tasks, SCAS performs better than IQL and outperforms other methods by very large margin. In NeoRL (Table 4), SCAS performs comparably to MOBILE [59] and outperforms other baselines.
Runtime. We present the runtime of algorithms the bottom of Table 1. SCAS exhibits significantly lower runtime than MOPO, SDC, and OSR and comparable to other model-free baselines.
Generality, SCAS is generic model-basee regularizer that can be easily integrated into existing offline RL. algorithms. The corresponding results and analysis are provided in Appendix E.5.
6.3 Comparisons in Perturbed Environments
In this section, we evaluate the algorithms in a more real-world setting where the agent receives uncertain perturbations during test time. OOD state correction is even more critical in such scenarios since the agent can enter OOD states after perturbation. To simulate this scenario, we add varying steps of Gaussian noise with magnitude of 0.5 to the actions conducted by the policy during test time. Specifically, the policy is trained on standard D4RI. datasets but tested the perturbed environments. We control the strength of perturbations by adjusting the number of perturbation steps.

---

SCAS
SDC
CQL
TD3BC
hopper-modiugg experi-y2
hopper eeplay.v2
120
horper
120
wwl.warddddddoo medium expert-
Normalized Retum
Normaizzed Retum
Normalized Retum
Normolzzed Return
20
30 Pattrrition Steps
10
20
40
20
30 Perturbation Steps
10
20
40
20
30 Perturbation Steps
10
20
40
60
10
20
30
40
Perturbation Steps
Figure 3: Comparisons in the perturbed environments with varying perturbation levels. The pertur- bation steps are the steps of Gaussian noise added to the conducted actions in an episode. SCAS exhibits better robustness against environmental perturbations during the test phase.
0:5
0.25
A-0.5
untmaze-large- parpa-e-rrrg-dieeee---
halfchoctah-mee--xxp.o
antmaze- large dingee-dverse22222
hulfcheetah-mmee -ee---medeeeee exp-22
100
100
100
Normllzze Reum
Normalized Return
50-
50
50
0.25 0.5
0.75
0
0.25 0.5
0.75
0
0.25 0.5 0.75
0.25 0.5 0.75
Gradient Steps (x109)
Gradient Steps (x109)
Gradient Stops (x10%)
Gradient Steps (x109)
(a) Effects of the inverse temperature a.
(b) Effects of the balance coefficient A.
Figure 4: Parameter study on the inverse temperature and the balance coefficient A. (a) An appro- priately large a is crucial for achieving good performance. (b) The proposed SCAS regularization is essential and demonstrates robustness to changes in X.
Figure shows the results of TD3+BC, CQL, SDC, and SCAS on various datasets over five random seeds. We observe that SCAS consistently outperforms previous methods across different perturbation levels and also exhibits less performance degradation against perturbations. Therefore, SCAS enjoys better robustness against perturbations in the complex and unpredictable environments.
6.4 Parameter Study
We examine the effects of the inverse temperature a. the balance coefficient X, and the noise scale a. Due to the space limit, the results for and on additional datasets deferred Appendix E.6. A sensitivity analysis on dynamics model errors is also provided in Appendix E.7.
is the key hyperparameter in SCAS for achieving value-aware OOD state correction. = 0, the effect degenerates to vanilla OOD state correction. Figure 4(a) displays the learning curves of SCAS with different The results show that a large is crucial for achieving good performance (also verified on more tasks Figure 6), clearly demonstrating the effectiveness of our value-aware OOD state correction. However, too large (a 10) induces less satisfying performance, probably due to the increased variance of the learning objective.
Inverse temperature
Balance coefficient X. À in Eq. (20) controls the balance between vanilla policy improvement and SCAS regularization. We vary within the range [0, 1] and present the learning curves of SCAS in Figure 4(b). Notably, SCAS is able to converge to good performance over very wide range of À (also verified on more tasks Figure 7). An interesting finding is that even when from RL improvement (max Q is removed, SCAS still performs well on most tasks. This could be attributed to the fact that value-aware aware OOD state correction implies some sort of improvement in policy by maximizing the values of policy-ine policy-induced induced next states.
and the signal
7 Conclusion and Limitations
In this paper, systematically analyze the OOD state issue in offline RI. and propose SCAS, simple yet effective approach that unifies 0OD state correction and 0OD action suppression. SCAS also achieves value-aware OOD state correction, significantly improving performance over vanilla

---

OOD state correction, Empirical results validate the properties of SCAS, showcasing its superior performance on the offline RL benchmarks and its enhanced robustness perturbed environments.
However, our work also has some limitations. For example, current SCAS primarily focuses on continuous control tasks. In discrete settings, algorithmic components like state perturbation strategy would be different, which would be an interesting direction for future work. Moreover, we anticipate employing more advanced dynamics models, such as ensembles [73] and diffusion models [21]. to further improve the performance of our method.
Acknowledgment
We thank the anonymous reviewers for feedback on an early version of this paper. This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801, National Natural Science Foundation of China under Grant 61827804.
References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22 22-31. PMLR, 2017.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- -based offline reinforcement learning with diversified g-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
[3] Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-dii offline reinforcement learning. In International Conference Learning Representations, 2022. URL https:///p//eereee
net/forummid=Y4coo 123HnQL.
[4] Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson, A survey of meta-reinforremmett learning. arXiv preprint arXiv:2301.0802 2023.
[5] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances Neural Information Processing Systems, 34:4933 4946, 2021.
[6] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint arX1v:2202.02446, 2022.
[7] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414-419, 2022.
[8] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model aggostic meta-learning for fast adap- tation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
[9] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXhv:2004.072199 2020.
[10] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-200 35-114 2021.
[11] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018
[12] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
10

---

[13] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[14] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning Maxent RL without entropy. In The Eleventh International Conference on Learning Representations, 2023. URL https: / / /pwpoeveeeeeeeeeeeeeeee net/ dordmrid--JJLdde3tRRL
[15] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages (682-3691. PMLR, 2021.
[16] Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55(2):895- 943, 2022.
[17] Shangding Gu, Long Yang. Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 2022.
[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[19] Zhang-Wa Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing Systems, 36:4985-5009, 2023.
[20] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[21] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902-9915. PMLR, 2022.
[22] Ke Jiang, Jia- Yu Yao, and Xiaoyang Tan. Recovering from out-of-sample states via inverse dynamics in offline reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[23] Alistair EW Johnson, Tom Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):199, 2016.
[24] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. arXiv preprint arXiv:1903.00377 2019.
[25] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 11111810-21223 2020.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[27] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. 2013.
The International Journal of Robotics Research,
[28] Ilya Kostrikov, Rob Fergus. Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
[29] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https:
//openreview

---

[30] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction, Advances in Neural Information Processing Systems, 32, 2019.
[31] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1 1179-115 1191, 2020.
[32] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[33] Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offiin policy optimization via stationary distribution correction estimation. In International Conference Machine Learning, pages 6120-6130. PMLR, 2021.
[34] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[35] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abheel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments. Advances in neural information processing systems, 30, 2017.
[36] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongging Lu. Mildly conservative q q-learning for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https:// openreview. http://ee/ffooeeeee mwt/fooumm www.ooorummiiiideeeeeee.. pdV
[37] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman, 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3-15, 2017. 2017
[38] Liyun Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. ODICE: Revealing the mystery of distribution correction estimation via orthogonal -randeet update, The Twelfth International Conference on Learning Representations, 2024. URL httLhtt//////eeeee //pp///eee/eeeeeeee/eeeeeee wwt.vviwwwwe/teeeee net/forum? id=L8UNn7LIt4.
[39] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pages 23829-23851. PMLR, 2023.
[40] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
[41] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efffcieet reinforcement learning via model-based offline optimization. In- ternational Conference on Learning Representations, 2021. URL https://openrevier. net/forumiid 1ifforummiddhhhKKqPIAW.
[42] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):599-53, 2015.
[43] Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
[44] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02072 2019.
[45] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint 200.090006.09359, 2020.
[46] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
12

---

[47] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. weeigtt regression: Simple and scalable off-po reinforcement learning. arXiv preprint arXiv:1110.0017,, 2019.
[48] Dean Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, I, 1988.
[49] Rong-Jun Qin, Xingyuan Zhang. Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: near real-world benchmark for offline reinforcement learning. Advances in 2022.
Neural Information Processing Systems,
[50] Yun Qu, Boyuan Wang, Jianzhun Shao, Yuhang Jiang, Chen Chen, Zhenbin Ye, Linc Liu, Junfeng Yang, Lin Lai, Hongyang Qin, et al. Hokoff: Real game dataset from honor of kings and its offline reinforcement learning benchmarks. In Thirty-seventh Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, 2023.
[51] Yun Qu, Boyuan Wang, Yuhang Jiang, Jianzhun Shao, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Choices are more important than efforts: Llm enables efficient multi-agent exploration. arXiv preprint (xXi:22410002511 2024.
[52] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
[53] Julian Schritwwisser, Joannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):004-099 2020.
[54] Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conservative q learning for offline multi-agent reinforcement learning. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https https:: //op::////////vi////// /ett/ forum?id 2tw.:idd2222mmmm.....m.mmm
[55] Jianzhun Shao, Hongchang Zhang, Yun Qu, Chang Liu, Shuncheng He, Yuhang Jiang, and Xiangyang Ji. Complementary attention for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 30776-30793, PMLR, 2023.
[56] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. Pmir, 2014.
[57] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 350(7676):354 2017.
[58) Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[59] Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. inconsistency for model-based offline reinforcement learning. In International Conference on Machine Learning, pages 33177-33199 PMLR, 2023.
[60] Richard Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4): 1991.
[61] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[62] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11). 2008
[63] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-333 2019.
13

---

[64] Cheems Wang, Yigin Lv. Yixiu Mao, Yun Qu, Yi Xu, and Xiangyang Ji. Robust fast adaptation from adversarially explicit task distribution generation. arXiv preprint 2XX1::2077.19523, 2024.
[65] Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference on Machine Learning, pages 23055-23077. PMLR, 2022.
[66] Qi Wang, Yiqin Lv, Zheng Xie, Jincai Huang, et al. A simple yet effective strategy to robustify the meta learning paradigm. Advances in Neural Information Processing Systems, 36, 2024.
[67] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[68] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022. URL https //penrevvei net/forrumiid=KKXX55aa
[69] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11360 2019.
[70] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-6694, 2021.
[71] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023.
URI. htttp::://peereiieeeet net./ wwweet//oormmiito www.rrrmimuuuuroooo YYgosss
[72] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offfine reinforcement learning via conservative smoothing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL /ttttps:/oppnree wwweeeewwwweeeeeeeeeeeeeeeeeeeee
KE.
[73] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 13114112-4442, 2020.
[74] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline mosel-based policy optimization. Advances in neural information processing systems, 34:28954 22967, 2021.
[75] Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xi- angyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9022-9030, 2022.
[76] Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In The Eleventh International Con- ference on Learning Representations, 2023. URL dfDv0WU853R.
net/ wwt./eeent/iiiieee forum?ddd
14

---

A Related Work
Model-free offline RL. In offline RL, extrapolation error and overestimation caused by OOD ac- tions pose significant challenges. Among model-free solutions, value regularization methods penalize the Q-values of OOD actions [31, 2, 28, 36, 3, 72, 40], while policy constraint approaches compel the trained policy to be close to the behavior policy, either explicitly via divergence penalties [69, 30, 10], implicitly by weighted behavior cloning [47, 45, 67, 39], or directly through specific parameterization of the policy [12, 15J. Relatively independently, in-sample learning methods formulate the Bellman target using only the actions in the dataset to avoid OOD actions [5, 29, 76, 71J. Recently, some works aim to learn the optimal policy within the support of the dataset (known as in-support or in in-samlle optimal policy) in a theoretically sound way and are less affected by the average quality of the dataset [39, 40, 68]. However, existing popular offline RL approaches primarily focus on the OOD action issue during training and often neglect the OOD state issue during the test phase.
Model-based offline RL. Model-bases RL methods learn model of the environment and generate synthetic data from that model to optimize the policy [60, 20, 24]. To ensure conservatism in offline RL. Kidambi et al. [25] and Yu al. [73] estimate the uncertainty in the model and apply reward penalties for state-action pairs with high uncertainty. Some model-based approaches also introduce conservatism similarly to model-free ones, employing techniques like value regularization [74] and policy constraint [41]. Recently, Sun et al. [59] conducts uncertainty quantification through the inconsistency of Bellman estimations under the learned dynamics ensemble. However, model- methods often come with high computational burden [20], and their effectiveness relies heavily on the quality of the trained model [43]. In contrast, our algorithm leverages the dynamics model to propagate policy gradients, make one-step predictions, and regularize policy training, leading to significantly improved computational efficiency and relatively high prediction accuracy.
OOD state correction. In offline RL, OOD state correction deserves more attention as the state deviation during the test phase can accumulate over time steps, severely degrading performance [34]. Existing limited solutions aim to train the policy to correct the agent from OOD states to ID states [75, 22]. Specifically, SDC [75] builds dynamics model and a state transition model, and aligns the policy-induces next state distributions at OOD states with the state transition model. On the other hand, OSR [22] utilizes an inverse dynamics model to constrain the policy at OOD states. Compared with prior methods, our proposed SCAS efficiently unifies OOD state correction and OOD action suppression in offline RL and additionally achieves value-aware OOD state correction. The DICE series of works [44, 33, 38] share similar motivations with SCAS to some extent; however, there are significant differences between the two. Firstly, DICE is based linear programming framework of RL, while SCAS is based on a dynamic programming framework. Therefore, the theoretical foundations and learning paradigms of the two are inherently different. Secondly, SCAS only corrects encountered OOD states, whereas DICE algorithms require the policy-induced occupancy distribution to align with the dataset distribution. Therefore, DICE's 's constraints stricter, potentially making it more susceptible to the average quality of the dataset. Lastly, theoretical and empirical evidence indicate that DICE algorithms have a problem of gradient cancellation [38], which imposes certain limitations on their practical effectiveness.
B Proofs
In this section, we present the proofs for the theories in the paper.
B.1 Derivation of the Value-aware State Transition Distribution
We show that Eq. (6) is the optimal solution of the optimization problem Eq. (5):
max a N
E V(s') DKL (16))
(21)
anD
3'~N"' (-1s)
15

---

We can optimize No at each E D separately. Thus we consider the following optimization problem:
max N
E
V(s') (
N
(22)
s.t. Σ Nsssss) = 1, E
This constrained optimization problem is convex, and the Lagrangian is:
V(d) DKL (8(()) + Σ [((()))-1
((N) = a (-1o)
(23)
The KKT condition gives:
ac
aV(s') log (s'(s) log ((s'(s)) + (N
(24)
Solving for Ñ gives the closed form solution N
(s'ls) = exp (aV (s') - 1 + ()'(s)) ~
(25)
By the condition Σ. (8'ls) = 1, we can directly solve the Lagrangian multiplier v and replace exp(v 1) with normalization factor:
(b'18) = 7(x) exp (aV (s')) N(11s), A8 D
(26)
where Z(s) = Eg exp (aV (s')) (s'(s) is the normalization factor.
B.2 Proof of Proposition
Proposition (Proposition in the main paper). Suppose that the environment dynamics is determin- istic, then both R(x)) and (T) achieve their global maximum at the policy x², where
T (a 8) = Z(s) exp (aV (s, B(als)
(27)
The support of is within that of the behavior policy 8:
supp(x" (-pp(="(-1s)) S supp(3(-1s)), vs ~ D
(28)
and π* makes the following equation hold:
N (+1s)) = M(-1s, "" (+|s)), Vs
(29)
Proof. We start with R(x).
argmax R(TT)
(30)
argmax E (r,8")-D
exp (aV (s')) log ((ssss
(31)
(s))
argmax
Z(s) exp (aV (s')) log M(5 ((1s)
(32)
anD s' N(1ss)
argmax
E
[log ((' (-|s))
(33)
anD o' (-1a)
argmin E argmin E. (N*s (((ss)) aND
[log No (s'ls) log M(s ss ((1ss))
(34)
AND (-1s)
(35)
'Here for clarity, we use the notation M with slightly different meanings in different cases: in the stochastic setting, M S x A + A(S); in the deterministic setting, M S x A - S.
16

---

The third equality holds because of the relationship between N and N in Eq. (6):
(s'sss)) ((s) exp (aV (s') ((11s), Vs D
(36)
Therefore, the maximizer of R(T) is equal to the solution of the minimization problem in Eq. (35). Now consider the two distributions N (-Is) and (-]s, a-1ss) in Eq. (35).
1
(5 = Z(s) exp (aV (6')) (s
(37)
1
exp (aV (6)) Σ (1
(38)
((s)
For analytical clarity, we use the notation M with slightly different meanings in different cases: in
the stochastic setting, M : S x - A(S); in the deterministic setting, M : S x A + S. With the deterministic dynamics assumption,
N' (s'ls) = [(s) exp (aV (s') E ((s =s
(39)
Σ 7s) exp (aV (s B()s)( = s')
(40)
Σ 7(8) exp (aV ((sss =]]
(41)
On the other hand,
(165's,7((155 Σ ((d((,
(42)
Σ (s).) 87)
(43)
Now we define n² (a(ss) as
1
क (als)) = 7(s) exp (aV (M(s, a))) B(als)
(44)
We first show that n" is a valid policy, that is, n² is normalized.
1
x" (a|ss)
Z(s) exp (aV (M(s. ((ass)
(45)
exp (aV ())ss,a))))))))
(46)
Σn (aV (s') (('lss)
exp (aV
(47)
Σ. (aV (s')) E. ( exp (aV (M(s
(48)
E. E. exp (aV (5') ((
exp (aV (M a))) B(als) Σ. exp (aV (M ((((s((s)
(49)
Therefore, Σ. T"' (als)
Substitute Eq. (44) into Eq. (41),
N (s'ls)
Σ x (als)I = =s']
(50)
Comparing Eq. (43) with Eq. (50), it holds that (s'1ss)
Vs ~ D. As a result,
E DKT (FN)) " (((sss) -0 BND
(51)
17

---

Considering the non-negativity of KL divergence, the optimization problem in Eq. (35) achieves its global minimum at n²" Therefore, RT))) also achieves its global maximum
Now we consider Pe (T²)
arginxx Ri (T)
(52)
arginxx
E (N
(exp (a (V (8') - V log ((s'ss.
log
(53)
Z(s)
argmaxx
E
exp (aV (s log M (s
(54)
(6,0 (nD
exp (aV (s)) ((s)
Z(s)
arggax E
E
exp (aV (s log M M(s (ss)s
(55)
o' ((()(
exp (aV (s) (())
((s)
argmax E
E
exp (aV (s)) log M(8'ss M(s'(s,
(56)
8 N+1(())))) (-1s)
Z(s)
argmin E
E
exp (aV (s)) (log N (s'ls) log M(s'ss M('s
(57)
SnD s' (1())
((s)
argmin
E
exp (aV (s)) DKL (N (+ss)
(58)
5ND
The fourth equality holds because of the relationship between N and N in Eq. (36).
As shown above, it holds that (s'(s(s) M('sss, x"
x* (+1s)), Vs D. As result,
2(s) DRL (N (-16)))) =
E
(59)
exp (a (s))
Considering 7(x)/ /exp (aV (s)) > O and the non-negativity of KI. divergence, the optimization problem in Eq. (58) achieves its global minimum at x² Therefore, Rí R1(x) also achieves its global maximum at n²
In conclusion, when the environment dynamics is deterministic, both R(x) and (x) achieve their global maximum at the policy n², and n= makes the following equation hold:
(-Is) M(-ss, T" (-1s)), Vs D
(60)
Moreover, because n" (a|s) = .ोσ exp (aV (s.(ss))) Baass), the support of x is included by that of the behavior policy 3:
supp(= (+|s) C supp(3(-1s)) as D
(61)
B.3 Proof of Proposition
Proposition 4 (Proposition 2 in the main paper). When the dynamics is stochastic, the maximizers of both R(T) and Rı (T) are constrained within the support of the behavior policy:
(upp(="(+1s)) S supp(3(-1s)), As ~ DD supp(=i(-ls)) S supp(3(-ls)), D
(62)
(63)
Proof. We start with R(T).
E R(T):
((s) exp (a.V (s')) log M(s ((s'ls,
(64)
(DD
[(s) exp (aV (s')) log Σ Ms (f ]
E DD
(65)
Let denote any valid policy. For Vs E D, define E(s) and n(s) as follows:
E(s) Σ [[B(als) (als)
(66)
n(s) = Σ [[{(a]s) o]
(67)
18

---

For Vs E D, there exists at least one action such that (s, E D. Thus it holds that n(s) > 0, 4/ E D. Then for Vs E D, V'T, define Vin as follows:
Tin (als) { <(als)-
c(s) x())
B(als) > 0, B(a|s) = 0.
(68)
O,
Tin can be seen as projection of onto (3s support. Besides, for Vs D,
c(s) Σ Tin Tin (a|s) = Σ [B(aab) > 0] [T(aa)) + n(s)
(69)
U[B(als) > (as(a|s) c(s)
(70)
Σ L[B(als)> (al(s + Σ [[3(a]s) (al(s)s)
(71)
= Σ {a]s)
(72)
1
(73)
Thus Tm is a valid policy.
Now we compare R(Tm) with R(T). For
Σ M(d(ss - Σ a(s
(74)
() =((lss)
(75)
Σ ((5')s,a)TTn(al - π (als))
(76)
(a]3(a)))>0)
M('(s, a) c(s)
Σ
(77)
n(s)
(a]3(a)))>0)
0
(78)
The second equality holds because, in tabular MDPs, the empirical dynamics model M exactly computes the conditional distribution observed in the dataset. For transitions not contained in the dataset, M = 0 [12]. The final inequality holds because e(s) 2 0.
Therefore,
R(mm)
R(T)
(79)
1
Ea M ( En (((5"),,)TT(ap))
]
E
(s) exp (aV (s') log
(80)
E
(s)) exp (a (s')) log (1)
(81)
(s,s")~
0
(82)
Now suppose æ is not constrained within the support of the behavior policy at some state E D:
(upp(a((-(s1) g supp(3(-1s))). That is, 3ā, such that (ā]])))) > and ((ār(s)) = 0. Thus
holds that e(s1) = E. [[]{a]s)) (ar(als)) O. On the other hand, since 5 D, there exists at least one action a and one state of such that M (sf(s1,a1) a1) > 0. As result,
E D. Thus it holds that ((alss)) > 0 and
Σ a))
Σ ()((ls)
(83)
M(silsi, a) <(s1)
(84)
n(81)
{a]
(85)
19

---

In such case, R(TT)) > R(T.). Therefore, if π is not constrained within the support of the behavior policy at some state $1 D, we can find another policy Tin that is constrained within the support of the behavior policy and achieves higher objective function R R(Tm). Consequently, R((r) must achieve its maximum at support constrained policy n²: supp("" (-Is) C supp(3(-1s)) Vs D.
Now we consider R1 (T).
(T( :
E (
lexp (a (V (s') - V (8)) log (ssss
(86)
E (a,8")~
cxp (V (5') V (s))) log Σ (M
(87)
With the same definition of E(s), n(s) and Tin as in Eq. (66), Eq. (67) and Eq. (68), it also holds that
R1 (Tin) - R
(88)
M (m)
E. M E. M(51s,
)
E
exp (a (V (8') - (s))) log
(89)
E
(exp (a (V (5') - V (s))) log (1)]
(90)
(e
0
(91)
As before, when supposing ग is not constrained within the support of the behavior policy some state E DD, it holds that R2 (mm) > Rı (T). Therefore, R2 (T) must achieve its maximum at support constrained policy कं: supp(zī ( (-[s)) S supp(5(-Is)), Vs D.
In conclusion, when the environment dynamics is stochastic, the maximizers of both R(T)) and R1(T)
constrained within the support of the behavior policy:
supp(T" (-1s)) S supp({(-Is)), (-pp(Ti(-Is)) S supp((((1s)). Vs D
(92)
C Further Discussions
C.1 Rationale for Choosing (x)(aV(s)) as the Empirical Normalizer
Firstly, choosing exp(aV(s)) is intended to obtain something similar to the advantage function.
With this normalizer, the weight of our regularizer is exp(a(V Vs))), which is comparable to the weight (xp(a.(ss, a)) in Advantage Weighted Regression (AWR) [47]. Here, V(s') V(s) represents the relative advantage of the next state s' compared to the current state s. while A(s, a) reflects the relative advantage of taking action a in s compared to following the current policy. Comparison of the objectives of SCAS and AWR:
SCAS:
(s(((((ss)) - V (o (()))
(93)
AWR: exp(aA(s, (8, a)) log T(ass)
(94)
Secondly, as discussed in the paper, introducing any normalizer that depends only on (independent of s') does not affect the development and analysis of our method; it is merely for computational stability. In AWR-based methods, there also exists normalizer Z(x) and they usually disregard [47, 45). The rationale behind this is similar.
C.2 Pessimism and Robustness in SCAS
In a specific sense, SCAS, which unifies OOD state correction and OOD action suppression, also integrates pessimism and state robustness. (1) Regarding pessimism: The OOD action suppression effect of SCAS aligns with the pessimism commonly discussed in offline RL work (being pessimistic
about OOD actions) [31, 70, 30, 3, 54]. Unlike traditional policy constraint methods [69,30 30, 10, 47], our approach does not require the training policy to align with the behavior policy; it only requires the successor states to be within the dataset support, which is more relaxed constraint. (2) Regarding
state robustness: The OOD state correction effect of SCAS is aimed at improving the agent's
20

---

robustness to OOD states during the test phase. Compared with previous works, SCAS unifies OOD state correction and OOD action suppression and additionally achieves value-aware OOD state correction. Some offline RI literature on state robustness differs from our approach; they typically consider noisy observations [72], such as sensor errors. In contrast, SCAS addresses state robustness concerning actual OOD states encountered during test time, rather than noisy observations.
C.3 Regularization Effect at ID States
In SCAS, there regularization on the policy's output actions at ID states. In our regularizer, the perturbed states are sampled from (s, o²), and large portion of will fall near the original ID state s or even be approximately equal to s. Therefore, the policy's s output actions at states are also regularized. For this part of the regularization, its role is equivalent to the ID state regularizer analyzed in Section 4, which has been theoretically shown to have the effect of suppressing OOD actions. Moreover, the experimental results in Section 6 also demonstrate that OOD state correction regularizer addresses the traditional issue with OOD actions.
C.4 Differences between the OOD Action Issue and the OOD State Issue
We further elucidate the differences between the well-known OOD action issue and the OOD state issue we analyzed. Most offline RL works focus on the OOD action issue in the training phase. That is, the trained policy outputs OOD actions to compute the target Q which results in extrapolation error and value divergence during training [12]. In contrast, the OOD state issue we defined and analyzed is in the test phase. That the agent can enter states out of the offline dataset during test, potentially resulting in catastrophic failure.
D Experimental Details
Table 2: Hyperparameters in SCAS.
Hyperparameter
Value
Optimizer
Adam [26]
Critic learning rate
3 x 10
Actor learning rate
x 10 with cosine schedule
Batch size
256
Discount factor
0.99
Gradient Steps
106
Policy training
Target network update rate
0.005
Policy update frequency
2
Number of Critics
4
Inverse temperature
Balance coefficient X
0.25
Noise scale
0.003
Optimizer
Adam
Learning rate
1 x 10-3
Dynamics training
Batch size
256
Gradient Steps
x 105
Actor
(oput-256-256-oupph
Architecture
Critic
input-256-256-
Dynamics
22r-2266255-25-255--2-2 256-255--25-uuuu input-256-256-255
All hyperparameters of SCAS are included in Table 2. Note that we use this same set of hyper- parameters to obtain all the results reported this paper (except for parameter study). Following TD3+BC [10], we normalize the states in all datasets except for antmaze-large. We clip the expo- nentiated weight exp (aVo (s') - V (s)) in Eq. (19) to -o,, 50). Following the suggestions the benchmark [9], subtract from the rewards for the Antmaze datasets.
21

---

Our evaluation criteria follow those used in most previous works, For the Gym locomotion tasks, we average returns over 10 evaluation trajectories and 5 random seeds, while for the Ant Maze tasks, we average over 100 evaluation trajectories and 5 random seeds. The reported results are the normalized scores, which are offered by the D4RL. benchmark [9] to measure how the learned policy compared with random and expert policy:
learned policy return random policy return D4RL score = 100 x expert policy return random policy return
The results of baselines reported in Table I are obtained as follows. We re-run OSR [22] on all datasets using their official codebase" and tune the hyperparameters for each dataset as specified in their paper. We implement SDC [75] and re-run it on all datasets. We use the SDC-related hyperparameters as specified in their paper, and sweep the CQL-related hyperparameters in 1,2,5.10,200 for each dataset. We re-run OneStep RL [5] all datasets using their official codebase" and the default hyperparameters. We implement BC [48] based on the TD3+BC repository" and re-run it on all datasets. The results of other baselines taken from [3] and [68]. The runtime in Table is obtained by running offline RL algorithms on
on a GeForce RTX 3090.
Figures ((a) to I(d) share the same embedding function obtained by rumning t-SNE on the set of all 200,000 samples (50,000 samples each from the dataset, CQL, TD3+BC, and SCAS). This ensures clear visual comparison. Figure (dd contains all the 200,000 samples, which is the union of the points in Figures ((a) to 1(c).
E
Additional Experimental Results
E.1
Additional Value Estimation Results
Under the same setting of Figure 2, we conduct experiments on the additional datasets. The results are shown in Figure 5. We omit the Q values of Off-policy RL, SDC w/o CQL, and OSR w/o CQL higher numbers of optimization steps, because these Q values diverge in the early learning stage, and plotting their Q values later optimization steps would result in excessive range the vertical axis. The additional results also show that only SCAS's OOD state correction term can achieve OOD action suppression and prevent value over-estimation.
meller22
walker2d-medium-oxpetvy
10
SCAS: MC
10
SCAS:
10
Offipolicy Q
10
SDC wio CQL:
10
OSR w/o CQL: Q
10a.
103
10'
103 0
10²
0.25
0.5
0.75
0.25
0.5
0.75
Gradient Steps (x10*)
Gradient Steps (x10%)
Figure 5: Oracle Q-values of SCAS (estimated by MC return) and learned Q-values of SCAS and other algorithms across optimization steps. Here Off-policy RL is SCAS with weight À 0 in Eq. (20). Only SCAS's OOD state correction term can achieve OOD action suppression and prevent value over-estimation (divergence).
E.2 Additional Results on OOD State Correction
To further examine the OOD state correction effects of SCAS, we conduct experiments on a modified D4RL maze2d-open-v0 [9]. It is 2D point robot navigation task in a rectangle map with vertices (0, 0) and (3, 5). The agent needs to reach the goal at (2, 3). We modify the dataset by removing all
"https://gitann httggttub.cco/aackk00/////////// /tttts://github... httppe///ghaab...mmaavataataaaa wwt.///wiidrrraaaaaaaaeeeeeeeere /https://gi wttpttuww..co/tttin///....11
22

---

the transitions containing states in a rectangle with vertices (0, 0) and (1.5, 2.5). During test, we let the initial state be randomly distributed this OOD region. We train algorithms over 10 gradient steps and average returns over 1000 evaluation trajectories.
The results of BC [48], TD3+BC [10], CQL. [31], MOPO [73], IQL [29], and SCAS reported in Table 3. With the OOD state correction signals, SCAS corrects the agent out of the OOD region more quickly and stably, achieving significantly better performance than typical offline RL methods.
Table 3: Comparisons in modified maze2d over five random seeds.
BC
TD3+BC
CQL
MOPO
IQL
SCAS
Steps of OOD 84.7144.7 58.0+33.7
63.8+33.0
50.6::25.4
37.7+6.7
22.8
D4RL score
38.5+2554 63.9439..3 41.2 110.1488.8 335.0+114..9
571.9
E.3
Comparisons on the NeoRL Benchmark
Table 4: Averaged normalized scores the NeoRL benchmark four random seeds.
BC
TD3BC
CQL
EDAC
MOPO
MOBILE
SCAS
Hopper-High
43.1
75.3
76.6
52.5
11.5
87.8
100.5
Hopper-Mets
51.3
70.3
64.5
44.9
1.0
51.1
94.6=19.3
Hopper-Low
15.1
15.8
16.0
18.3
6.2
17.4
19.7=1.2
Walker2d-Hig
72.6
69.6
75.3
75.5
18.0
74.9
74.6=0.7
Walker2d-Med
48.7
58.5
57.3
57.6
39.9
62.2
63.4+1.00
Walker2d-Low
28.5
43.0
44.7
40.2
11.6
37.6
34.4+113
HalgCheetah-High
71.3
75.3
77.4
81.4
65.9
83.0
77.0=
HalfCheetah-Med
49.0
52.3
54.6
54.9
62.3
77.8
53.1
HalfCheetah-Low
29.1
30.0
38.2
31.3
40.1
54.7
31.5
total
408.7
490.1
504.6
456.6
256.5
546.5
548.7
hyperparameter tuning
w/o
w/
w/
w/
w/
w/
w/o
We also evaluate SCAS on the NeoRL benchmark [49]. NeoRL is a benchmark designed to simulate real-world scenarios by collecting datasets using more conservative policy, aligning closely with realistic data collection scenarios. The narrow and limited data makes challenging for offline RL algorithms. The results are shown in Table 4. The results of baselines taken directly from the MOBILE paper [59]. According to Appendix C in [59], these results are obtained by tuning hyperparameters per dataset. For SCAS, use the same fixed of hyperparameters specified in Appendix D. Without additional hyperparameter tuning, SCAS still performs comparably to MOBILE and outperforms other baselines in total scores.
E.4
Comparisons with Additional Baselines
The original SCAS requires only one single hyperparameter configuration in implementations. For fair comparison with DW [19], EDAC [2], RORL [72], SQL [71], and EQL [7]], roughly select À from (0.025, 0.25) for each dataset, referring to this variant as SCAS-ht. The results of SCAS-ht and these methods are reported in Table 5. Among the ensemble-fie methods, SCAS achieves the highest performance in both mujoco locomotion and antmaze domains. Compared with ensemble-based methods, SCAS-ht also performs better on antmaze tasks. DW [19] reweights ID data points by their values for behavior regularization and does not account for OOD states during the test phase. In contrast, our approach considers an OOD state correction scenario, resulting in enhanced robustness during the test phase and better performance.
E.5
Results of Combining SCAS Regularizer into Various Offline RI. Objectives
The SCAS regularizer is compatible with various offline RL objectives. conduct experiments to combine SCAS with CQL [31], IQL [29], and TD3BC [10]. Comparisons between these combined
23

---

Table 5: Comparisons with additional baselines on the D4RL benchmark. Here SCAS-ht means SCAS with slight hyperparameter tuning, selecting À from (0.025, 0.25). The results of SCAS-hi are averaged over 5 random seeds and the others are taken from their papers.
Ensemble-free
Ensemble-based
Dataset
DW+CQL
DW+IQL
SQL
EQL
DQL
SCAS-ht
EDAC
RORL
halfcheetah-med
47.7
48.3
47.2
51.1
58.5+
65.9
66.8
hopper-med
66.1
62.5
75.5
74.6
90.5
102.5- 0.3
101.6
104.8
walker2d-mpg med
82.1
80.8
84.2
83.2
87
90.81:2.6
92.5
102.4
halfcheetah- -eeeheeet---m-rrepppp
45.1
44.6
44.8
44.5
47.8
52.911.44
61.3
61.9
hopper-med-ree
88.6
79.7
99.7
101.3
101.6-1.0
101.0
102.8
walker2d-men -e-e-rr--
75.3
81.2
76.6
95.5
88.1+44.2
87.1
90.4
halfchcctah-med--ne
86.1
93.7
94.0
90.6
96.8
91.7++2.7
106.3
107.8
hopper-medeexp
92.9
81.0
111.8
105.5
111.1
109.7=3.5
110.7
112.7
walker2d-med-eex
109.7
109.7
110.0
110.2
110.1
110.8
114.7
121.2
locomotion total
692.4
664.8
749.5
730.5
791.2
806.6
841.1
870.8
antmaze-umaze
72.7
81.3
92.2
93.2
93.4
90.4+3.6
0.0
96.7
antmaze-umaze-div
34.0
61.0
74.0
65.4
66.2
66.4 14.3
0.0
90.7
antmaze-meddplaa
78.7
80.2
77.5
76.6
83.6:13.1
0.0
76.3
untmaze-med-div
1.3
64.7
79.1
70.0
78.6
84.6:15.0
0.0
69.3
antmaze-large largzzeelrre-plaay
2.0
40.0
53.2
45.6
46.4
59,415500
0.0
16.3
antmaze-large-div
0.0
42.0
52.3
42.5
56.6
56.2:1:544
0.0
41.0
antmaze total
114.0
367.7
431.0
394.2
417.8
440.6
0.0
390.3
Table 6: Comparisons on the D4RL benchmark. Here +SCAS means adding the SCAS regularizer. The results are averaged over random seeds.
CQL +SCAS
TD3BC +SCAS
IQL +SCAS
Dataset
CQL
TD3BC
IQL
SCAS
halfcheetah-med
47.0
46.5
48.3
44.1
47.4
46.8
46.6
hopper-med
53.0
96.1
59.3
66.6
66.2
76.8
102.5
walker2d- med
73.3
84.9
83.7
81.9
78.3
84.0
82.3
halfcheetah halfchectah-mee-repp
45.5
43.6
44.6
40.5
44.2
44.2
44.0
hopper-med-rey
88.7
100.2
60.9
79.4
94.7
102.3
101.6
walker2d-med-rep
81.8
78.6
81.8
76.2
73.8
76.2
78.1
aalfcheetah-med-ee
75.6
92.9
90.7
89.6
86.7
92.7
91.7
hopper-med-exp
105.6
108.2
98.0
108.9
91.5
101.9
109.7
malke22d-eed-ee
107.9
104.3
110.1
106.0
109.6
105,4
108.4
total
678.4
755.5
677.4
693.2
692.4
730.3
764.9
algorithms and the original ones shown in Table 6. We find that applying the SCAS regularizer leads to improved performance for these popular algorithms, which could be attributed to the OOD state correction effects of SCAS. However, also find that these combined methods do not achieve better performance than the original SCAS (comparable on most tasks and worse on some tasks). We hypothesize that this is because SCAS already has the effect of OOD action suppression, and when combined with offline RL objectives that also aim for OOD action suppression, it may become overly conservative. As result, the combined algorithms may perform worse than the original SCAS some sub-optimal datasets.
E.6 Additional Parameter Study Results
In this section, present additional parameter study results conducted on four challenging Antmaze large----re---deeeeeeeree antmaze- antmaze-medium
tasks, including antmaze-large largal-laagg----a--.,,,. antmaaz--mediun-diverr----
and
Inverse Temperature a. The inverse temperature a is the key hyperparameter in SCAS for achieving value-aware OOD state correction. It controls the significance of the values of next states in SCAS's

---

OOD state correction. If a = O, the effect corresponds to vanilla OOD state correction. As a gets larger, SCAS more inclined to correct the agent to the high-value ID states. Thus we can assess the effectiveness of value-aware OOD state correction compared to vanilla OOD state correction by varying a. Here we test SCAS with different and the results are shown in Figure 6. We observe that large a is crucial for achieving good performance on all the antmaze tasks, clearly demonstrating the effectiveness of our value-aware OOD state correction. However, too large a (a 10) induces less satisfying performance, probably due to the increased variance of the learning objective. In general, we find that choosing a 5 leads to the best performance.
0=5
a=1
a=5
=010
unimmize-lange-ply--222
amimuzz-llmge--dveree--- 100 unimuzz-mechuu-plly-
100 mommmuzz-meciimm-iieee
100,
100
Notmalized Retum
50-
50
50
50
0
0.75 Gradient Steps (x10%)
0.25
0.5
0.25 0.5
0.75
0.25 0.5
0.75
0.25
0.5 0.75
Gradient Steps 10%)
Gradient Steps (x10f)
Gradient Steps (x 10%)
Additional results from the parameter study on the inverse temperature a. The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
Figure 6:
Balance Coefficient X. The balance coefficient À controls the balance between vanilla policy improvement and SCAS regularization. we RL algorithm. Here we vary À in {0. 0.25, 0.5, 0.75, 1} and present the corresponding learning curves of SCAS in Figure Notably, SCAS is able to converge to good performance over a very wide range of X. However, if X = O, the vanilla off-policy RL suffers from extrapolation error and overestimation, demonstrating poor performance. We also observe very interesting fact that even when À = 1 and the signal from RL improvement (max Q) is removed, SCAS still performs well on fact that value-aware OOD state correction implies some
0, SCAS degenerates into the vanilla off-policy
most tasks. This could be attributed sort of improvement in policy by maximizing the values of policy-indes induced next states.
==
0.25
A=
0075
1=1
animaze- large- laaganzz-lagge-aayy-2 play-v2
diver--ve 100 animaze-mediumpplaye
100 paliiazz--mdiium-ivveeeees
100
100
Normalized Return
50
50
50
0.75 Gradient Steps (x10
0.25
0
0.75 Gradient Steps (x 10%)
0.25
0.25 0.5
0.75
0.25 0.5 0.75
Gradient Steps (x10f)
Gradient Steps (x10
Additional results from the parameter study the balance coefficient A. The curves averaged over random seeds, with the shaded area representing the standard deviation across seeds.
Figure 7:
Noise Scale a. The noise scale is the standard deviation of the Gaussian noise added to the original states for formulating the SCAS regularizer. Here we test SCAS with different and present the corresponding learning curves in Figure 8. We observe a significant performance drop with too large (a 1) on all the tasks, due to the heavily corrupted learning signal. On the other hand, when performance is also less satisfying. With a 0, SCAS still able to
0 (without noise), prevent the agent at ID states from entering OOD states, maintaining the agent in safe regions, but it cannot correct the agent from OOD states to ID states as reliably as the original SCAS. In general, find that choosing 0.001 0.01 leads to the best performance.
25

---

o= 0
0.0.00
o= 0.01
0=
o=1
100
aniimareelange-paay-22
100
100
antmaze-medium-plyg
100
Nomalized Return
50
50
0
0
0.25
0.5
0.75
0.25 0.5
0.75
0.25 0.5
0.75
0.25 0.5
0.75
Grudient Steps (x 109)
Gradient Steps (x10%)
Gradient Steps (x10f)
Gradient Steps (x10%)
Figure 8: Additional results from the parameter study on the noise scale Ø. The curves are averaged over random seeds, with the shaded area representing the standard deviation across seeds.
wulker2d-medium-22
wallerr22-mmcinnu--opoo-v-w
0.6
llzed Rabtum
0.6
100
Narmallne londllizee Ketum
Dynanies
Dynomies
50
0.2
0.2
0.0
0.0
10a
102
10
10a
10
10
102
10*
10"
10
Trained stops dynamics Mw
Trained steps of dynamics Ma
hoppor-mediun--2
hopper-mediume expert-v2
Nomollzz Retorn
100
Nomollae Remnnn
Dynamics
Dynamics
0.2
50
0.2
Lock
Return
Loss
Roturn
0.0
0.0
10
10a
10a
103
10a
10a
10a
102
10a
10a
10a
Trained steps of dynamics M
Trained steps of dynamics M
Figure 9: Performance of SCAS under different dynamics model checkpoints, which are obtained at different steps in the dynamics model training process. The figure plots the training loss of the dynamics model Ma and the corresponding normalized return of SCAS over 5 random seeds.
E.7 Sensitivity Analysis on Dynamics Model Errors
To empirically investigate SCAS under different dynamics model errors, SCAS using different checkpoints of the trained dynamics model, which are obtained at different steps in the dynamics model training process. The model error is controlled by the number of trained steps. The results shown in Figure 9. The figure plots the training loss of the dynamics model Ma, and the corresponding normalized return of SCAS over random seeds. We observe that the performance of SCAS increases with the number of trained steps of the dynamics model (i.e. the accuracy of the model) and stabilizes high level.
E.8 Learning Curves of SCAS
Learning curves on Gym locomotion tasks and Antmaze tasks are presented in Figure 10 and Figure 11 respectively. The curves averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
Broader Impact
Offline RL holds promise for facilitating practical RL applicationss domains like robotics, healthcare, and education, where data collection is often costly or risky. However, it is important recognize its potential negative societal impacts. One concern is that biases offline data may transfer to the learned policy. In addition, offline RL may affect employment by automating tasks traditionally
26

---

performed by humans, like factory automation or autonomous driving, Addressing these challenges will contribute the responsible development and deployment offline RL algorithms.
From an academic standpoint, this research systematically analyze the OOD state issue in offline RL and propose SCAS, simple yet effective approach that unifies OOD state correction and OOD action suppression. This work potentially offers researchers a new perspective on analyzing the OOD state issue and enhancing lest-time robustness in offline RL. Besides, SCAS also holds the promise be extended to safe RL [1, 17, 13J, meta RL [8, 65, 66, 64, 4J, and multi-agent RL [35, 52, 55, 51, 16].
100 balfcherrll-mmccummmeo ondiaoo---iddumm---e---
100
halfcheetth meduheeeelb.dddum..ele.... 100
50
halfcheetal: halfchetaal-cndom.v
cindom-222
Fpiiode Rehmm
Upioods Betum
Epiood Return
Ligitode
0.00
1.00
0.00
0.75
1.00
8100
0.50
0.75
1,00
8.30
0:25
0.50
0.75
1000
Stepe (x10f)
Steps (x10%)
Steps
(x10f)
Steps
(x10f)
hogper modimmdddoooe cannd-eepeeee--2 oxpent-22
hoppor-medium-- v2
bagger-mandm--22
100
100
lipisods Retum
Tipheds Rotum
50
Eptoode
50
25
8.00
8:00
8000
8.00
0.50
0.75
Grddies Steps (x10f)
Gindient Steps (x10%)
Gradicnt Stepe (x11ff)
Grndien Steps (x110f)
wwlllordd-medium--e exportvv222
2alke:24-mddian--ee
wwlleer2d-medium.vz
walkor26-masom--22
100
Epicode Retum
Hipisode Retum
Eptoode Retum
Lipisode
50
25
0.00
0.25
0.30
0.75
1.90
0.00
0.25
0.30
0.75
1.00
0.00
0.25
0.50
0.73
1.00
0.30
0.25
0.50
0.75
1.00
Gradient Stops (x10f)
Grddient Steps (x10%)
Gradient (x10f)
Gradiem
(x10f)
Figure 10: Learning curves of SCAS on Gym locomotion tasks. The curves averaged over random seeds, with the shaded area representing the standard deviation across seeds.
27

---

umizzer22
diverse-v2
medium play-r2
Episode Relum
Exiinde Retum
Epicnde Rettin
50
50
0.00
1.00
0.0
1.00
800
1.00
Gradiant Stops (x110f)
Gradicnt Skeps (x10*)
Gradicat Steps (x10%)
100
100
antrazz-large-play.e
100
Episode Retums
Eqpisdde Retumm
Episode Rerum
50c
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.30
0.75
1.0
0.00
0.25
0.50 0.75 Steps (x10%)
1,00
Steps (x100)
Steps (x<10*)
Figure 11: Learning curves of SCAS on AntMaze tasks. The curves are averaged over random seeds, with the shaded area representing the standard deviation across seeds.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
Please refer to Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification: Please refer to the code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt S ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Appendix D.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results in the paper accompanied by standard deviations across multiple seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
31

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neee //tpp/////////p////eeeeeeeeessse cc cc/public/Bthiicciuiiee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is well documented and anonymized.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "BQJs8kUdzO", "reviewer": "Reviewer_j7HR", "summary": "The paper proposes a regularization term in offline RL, which can simultaneously address OOD state correction and OOD action suppression without pretraining a state transition model $N(s'|s)$. It shows good performance in D4RL benchmarks.", "questions": "1. For Definition 1, if $d^\\pi_{M_\\tau}$ is a continuous probability density function of $s$, and if $d_D(s)$ only has support on $s$ which appears in the dataset, does it mean that any state which doesn't show up in the dataset is OOD? If that's the case, the probability of an OOD state is 1 since the number of ID states must be finite in the dataset?\n\n2. Instead of pretraining $N(s'|s)$, the paper chooses to add Gaussian noise to $s$, with the noise $\\sigma=0.003$. Is this hyperparameter important? If $\\sigma$ increases, can we have a more robust algorithm since it makes more OOD states into consideration?\n\n3. The method uses a deterministic policy. What if we choose a Gaussian policy?\n\n4. In Figure 2, why do Off-policy, SDC, and OSR only have one dot? Shouldn't it also be a line?\n\n5. How are the results in Table 1 recorded? Final round results, last 5 epoch average, or another method?\n\n6. Where do the baseline method results come from, their original paper? I ask because I found SDC doesn't have results for antmaze tasks. Did the authors implement it by themselves?\n\n7. The authors use antmaze-v2 for their method evaluation, while some baseline methods, for example, IQL and OSR, use antmaze-v0. In my experience, the v2 dataset can always provide higher rewards than v0. Can the authors explain the mismatch? Or can the authors show results on v0 as well?", "limitations": "I believe the theory of the paper is solid, while the experimental performance is my concern. I am happy to increase my score if the antmaze version mismatch issue is resolved.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I found it an interesting topic to consider the OOD state issue. Most existing offline RL methods focus on addressing the OOD action issue, while this paper highlights that OOD states may also cause training collapse. Another good property of the method is that it only needs one single hyperparameter configuration to achieve good empirical performance. The theory is also solid.", "weaknesses": "The method utilizes a deterministic policy, which is often regarded as lacking expressiveness. Thus, the performance is not as good as diffusion-based policy methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: About antmaze version.**\n\nSpecial thanks for your careful review. Actually, all the antmaze results presented in the paper are obtained from the **antmaze-v2** datasets. We noted that some baseline methods report results from antmaze-v0 (or not include antmaze experiments) in their original papers. For the results reported in Table 1 of the paper, we reran OSR, SDC, OneStep RL, and BC on antmaze-v2 datasets and took other baselines' antmaze-v2 results from the SPOT paper[2].\n\nTo answer your question more comprehensively, we have also run SCAS on the antmaze-v0 datasets. Since IQL has demonstrated superior performance compared to other baselines on the v2 datasets, we present a comparison of SCAS and IQL on antmaze-v0 in the following table. The results show that SCAS also outperforms IQL on antmaze-v0 tasks.\n\nTable: Comparison of SCAS and IQL on antmaze-v0 over 5 random seeds.\n\n| Dataset | IQL | SCAS |\n| --- | --- | --- |\n| antmaze-umaze-v0 | 87.5 | **90.4$\\pm$3.6** |\n| antmaze-umaze-div-v0 | 62.2 | **66.4$\\pm$14.3** |\n| antmaze-med-play-v0 | 71.2 | **76.4$\\pm$4.0** |\n| antmaze-med-div-v0 | 70.0 | **76.0$\\pm$3.2** |\n| antmaze-large-play-v0 | 39.6 | **45.6$\\pm$4.8** |\n| antmaze-large-div-v0 | **47.5** | **47.2$\\pm$7.2** |\n| antmaze-v0 total | 378.0 | **402.0** |\n\n**Q2: The method utilizes a deterministic policy, which is often regarded as lacking expressiveness. Thus, the performance is not as good as diffusion-based policy methods.**\n\nThanks for the meaningful comments. We address the concerns from three aspects.\n\n(1) About performance. One important reason why SCAS's performance might appear less impressive compared to diffusion-based policy methods is that SCAS uses only a single hyperparameter configuration. With slight parameter tuning, SCAS's performance can also be significantly improved. We roughly select $\\lambda$ from {0.025, 0.25} for each dataset, referring to this variant as SCAS-ht. Comparisons of SCAS-ht with Diffusion QL [1] and additional recent algorithms are presented in **Table 1** of the PDF (attached to the global response). Among the ensemble-free methods, SCAS-ht achieves the highest performance in both mujoco locomotion and antmaze domains. Compared with ensemble-based methods, SCAS-ht also performs better on antmaze tasks.\n\n(2) About deterministic policy. Diffusion-based policies are highly expressive, allowing them to better represent multi-modal distributions such as behavior policies. However, some work also hypothesizes that deterministic policy may be more suitable than stochastic policy for representing the optimal policy in offline RL [2]. To answer this question in the context of SCAS, we have also included experiments with Gaussian policies. The results and analysis are detailed in our response to Q5. In addition, compared with diffusion-based policies, the deterministic one is computationally efficient as no sampling procedure about latent variables is required. This makes SCAS more suitable for time-sensitive decision-making scenarios.\n\n(3) About compatibility. The SCAS framework does not conflict with diffusion-based policies, and exploring this integration would be an interesting direction for future work.\n\n**Q3: For Definition 1, if $d^\\pi_{M_\\mathcal T}$ is a continuous probability density function of $s$, and if $d_\\mathcal{D}(s)$ only has support on $s$ which appears in the dataset, does it mean that any state which doesn't show up in the dataset is OOD?**\n\nIn this paper, we refer to in-dataset states as ID and states outside the dataset as OOD for convenience. According to this definition of OOD states, the probability of an OOD state in a continuous space is indeed 1. For a more general mathematical definition, $d_\\mathcal{D}(s)$ in Definition 1 should be replaced with $d^\\beta_\\mathcal{M}$, where $\\beta$ and $\\mathcal{M}$ are the behavior policy and environment used to collect the dataset. Under this definition of OOD states, the probability of an OOD state in a continuous space is no longer 1. Essentially, $d_\\mathcal{D}(s)$ is the empirical distribution obtained by sampling from $d^\\beta_\\mathcal{M}$, and the development and analyses of the method in this paper remain unaffected.\n\n**Q4: Is this hyperparameter ($\\sigma$) important? If $\\sigma$ increases, can we have a more robust algorithm since it makes more OOD states into consideration?**\n\nThanks for this question. The ablation results on $\\sigma$ are presented in Fig. 8 in the Appendix. The results show that despite considering more OOD states, too large $\\sigma$ leads to a significant performance drop, due to the heavily corrupted learning signal. On the other hand, when $\\sigma=0$ (without noise), the performance is also less satisfying. With $\\sigma=0$, SCAS can still suppress OOD actions, but it cannot correct the agent from OOD states to ID states as reliably as the original SCAS. In general, we find that choosing $\\sigma$ within the range {0.001, 0.01} leads to the best performance.\n\n**Q5: The method uses a deterministic policy. What if we choose a Gaussian policy?**\n\nThanks for the question. We implement a version of SCAS with Gaussian policy and report its results in **Table 2** of the PDF. Overall, the performance of SCAS-Gaussian is comparable to the original SCAS on most tasks but is slightly worse on some tasks. We hypothesize that there may be two reasons for this: (1) Stochastic policy optimizes a lower bound of Eq. (10) while deterministic policy ensures the equality case. (2) Deterministic policy may, empirically, be more suitable than Gaussian policy in offline RL [2].\n\n**Due to the page limit, please refer to the next block. Thanks!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHPuSD9R50", "reviewer": "Reviewer_VgaK", "summary": "This paper focuses on OOD state issue, an important but overlooked issue in offline RL. This paper proposes aligning the OOD state towards In-Distribution (ID) states with high value, named as value-aware OOD state correction. Additionally, the paper discovers that the overestimation of OOD actions can also be mitigated during the implementation of OOD state correction. The experimental results demonstrate that the proposed algorithm is superior in performance and robustness.", "questions": "1. Figure 2 suffers from an excessive range on the vertical axis, causing the two curves of SCAS to almost overlap. A suggestion would be to take the log of the Q values on the vertical axis; this should make the two curves more distinguishable.", "limitations": "Please see weakness.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an intuitive solution to the OOD state iusse, while also mitigating the overestimation of OOD actions. \n\n2. The algorithm is highly robust, requiring only a single set of hyperparameters across different environments. This robustness is particularly important in the context of offline reinforcement learning, as online interaction for tuning parameters is costly or even dangerous.", "weaknesses": "1. The paper proposes that during the process of OOD state correction, the overestimation of OOD actions will also be addressed. I have some doubts about this point. The constraint $R_2$ seems to only regularize the policy to output actions at OOD states that result in the next state being within the offline dataset. However, there is no regularization on the policy's output actions at ID states. Therefore, I am confused about how the paper solves the traditional issue with OOD actions. If I have misunderstood anywhere, please point it out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1:  There is no regularization on the policy's output actions at ID states. How the paper solves the traditional issue with OOD actions?**\n\nWe apologize for the confusion. Actually, in our method, there is regularization on the policy's output actions at ID states. In our regularizer $R_2$, the perturbed states $\\hat{s}$ are sampled from $\\mathcal{N}(s,\\sigma^2)$, and a large portion of $\\hat{s}$ will fall near the original ID state $s$ or even be approximately equal to $s$. Therefore, the policy's output actions at ID states are also regularized in $R_2$. For this part of the regularization, its role is equivalent to the ID state regularizer analyzed in Section 4, which has been theoretically shown to have the effect of suppressing OOD actions. Moreover, the experimental results in Section 5 also demonstrate that our OOD state correction regularizer addresses the traditional issue with OOD actions.\n\n**Q2: Figure 2 suffers from an excessive range on the vertical axis. A suggestion would be to take the log of the Q values on the vertical axis.**\n\nThanks for pointing that out and for the advice. Actually, we already use a log scale for the vertical axis in Figure 2. To further address this excessive range issue, we have replotted this figure in **Figure 2** of the PDF (attached to the global response), which adds more data points to the early learning stage and uses a smaller vertical axis range.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "POoB0I7nNZ", "reviewer": "Reviewer_VyLC", "summary": "This paper presents SCAS (OOD State Correction and Action Supression), a model-based regularization approach that effectively addresses the challenges of out-of-distribution (OOD) states and actions in offline reinforcement learning (RL) algorithms.\nThe method unfolds in two main stages: (1) training a transition model, and (2) training a policy that incorporates a model-based regularizer. \nThis regularizer is designed to steer the policy towards high-value in-distribution (ID) successor states and thereby away from OOD successor states.\nWhile the primary focus of SCAS is to resolve OOD state issues, this paper shows that the regularization strategy not only mitigates visiting OOD states but, as a byproduct, also suppresses OOD actions in the training of the value function.\nEmpirical results demonstrate that the proposed method performs comparably on standard benchmarks, showing its effectiveness in refining offline RL algorithms.", "questions": "Q1. How sensitive is the performance to the accuracy of the model?\n\nQ2. Given that a main contribution of this work is a model-based regularizer, it appears that SCAS could be compatible with various offline RL methods. Is it feasible to apply the regularizer across different offline RL objectives, such as CQL, IQL, and others?\n\nQ3. In this study, the trained model is exclusively utilized as a regularizer during policy training. Could this approach be extended to also incorporate the model during testing time?", "limitations": "The authors included their limitations in Section 7 and limitations are addressed appropriately.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**1. Theoretical connection between OOD state correction and OOD action suppression**\n\nA notable strength of this paper is the establishment of a theoretical link between OOD state correction and OOD action suppression. The authors articulate how the mechanism designed to correct OOD states implicitly suppresses OOD actions, by showing an optimal policy of the regularized objective will produce actions inside of the support of behavior policy for every dataset state. This implicit suppression also explains successful empirical results on offline RL benchmarks.\n\n**2. Robustness to Hyperparameter Selection**\n\nAs detailed in Section 6.2 and illustrated in Figure 4, the proposed approach demonstrates considerable robustness to variations in hyperparameter settings. \nThis attribute is particularly valuable in practical applications of offline RL where optimal hyperparameter settings can be elusive or computationally expensive to determine.\n\n**3. Evaluation on standard benchmarks and implementation code is attached for reproduction.**\n\nThe paper's evaluation methodology is another major strength. The authors have rigorously tested their approach on standard benchmarks, D4RL and NeoRL, providing a comprehensive assessment of its performance relative to existing methods. Moreover, the inclusion of implementation code enhances the paper’s contribution by facilitating reproducibility and further experimentation.", "weaknesses": "**1. Unclear Motivation for Value-Aware OOD State Correction**\n\nThis paper proposes shifting the OOD state distribution not to a standard ID state distribution, but to a high-value ID state distribution instead. This choice raises questions about the specific objectives of value-aware state correction. Is it strategically targeting high-value data points, or is it intended to mitigate the effects of distribution shifts? It seems that the manuscript primarily focuses on the former, in that case, it would benefit from a comparison with existing works like DW[1], which also focus on high-value data points for behavior regularization. If the goal is indeed to mitigate the effects of distribution shifts, then it is crucial for the manuscript to better articulate and emphasize how value-aware OOD state correction contributes to the robustness against OOD state visitation. Moreover, a clearer explanation of how this approach aligns with offline RL principles like pessimism and robustness to OOD states would greatly strengthen the manuscript's validity and impact.\n\n**2. Lack of Comparisons with Comparable Offline RL Approaches**\n\nWhile the paper tackles a fundamental problem in offline RL, the evaluations presented are somewhat constrained by a lack of comparison with key recent advancements in the field. Notable works such as DW[1], EDAC[2], RORL[3], and SQL/EQL[4], which explore similar offline RL settings, are not considered in the main experiments. This gap is particularly significant as DW[1] employs a value-aware strategy that closely mirrors the approach of this paper, focusing on constraining policies to high-value data points. To convincingly justify the need for correcting OOD states—or for employing a value-aware correction approach—a comparative analysis with these influential studies is essential. This would not only place the findings in a broader context but also potentially highlight the contributions and advantages of the proposed method.\n\n[1] Hong et al., “Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets.”, NeurIPS 2023.\n\n[2] An et al., \"Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble.\", NeurIPS 2021.\n\n[3] Yang et al., “RORL: Robust Offline Reinforcement Learning via Conservative Smoothing.”, NeurIPS 2022.\n\n[4] Xu et al., “Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization.”, ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments. We have conducted extensive experiments to address your questions and concerns.\n\n**Q1: Unclear Motivation for Value-Aware OOD State Correction.**\n\n> Is it strategically targeting high-value data points, or is it intended to mitigate the effects of distribution shifts?\n\nIn this work, (i) the mitigation of state distribution shift, (ii) the mitigation of action distribution shift, and (iii) the tendency towards high-value data points are seamlessly integrated into a single framework (regularizer), whereas previous OOD state correction works only consider (i).\n\nValue-aware OOD state correction is a concept introduced in the manuscript. Since previous work has already considered (i), its motivation is to additionally achieve (iii). We demonstrate that within the SCAS framework, value-aware OOD state correction can be easily implemented, and it also implicitly achieves (ii). It is worth noting that SCAS with $\\alpha=0$ (vanilla OOD state correction in SCAS's framework, which differs from previous works) still has the effects of (i) and (ii). Compared to it, the motivation behind value-aware OOD state correction is primarily to achieve a tendency towards high-value data points, rather than to further improve the mitigation of distribution shifts. We do not claim that SCAS with $\\alpha>0$ is more effective in mitigating distribution shifts compared to SCAS with $\\alpha=0$.\n\nIn the following, we detail why SCAS not only mitigates state and action distribution shifts but also targets high-value data points.\n\nMitigating distribution shifts: From Eq. (5) and Eq. (6), $N^*(\\cdot|s)$ lies within the support of the dataset. Therefore, when aligning the dynamics induced by $\\pi$ on the perturbed state $\\hat{s}$ with $N^*(\\cdot|s)$ in Eq. (8), the result is that $\\pi$-induced $s'$ also lies within the dataset support. On the other hand, theory (Propositions 1 and 2) and experiments (Section 6) indicate that, in ID states, the actions outputted by $\\pi$ also lie in the dataset support. Therefore, the SCAS framework naturally mitigates both state and action distribution shifts, which is a key feature of our design.\n\nTargeting high-value data points: From Eq. (5) and Eq. (6), $N^*(s)$ is skewed towards high-value ID states. As a result, $\\pi$-induced $s'$ is not only corrected back to the dataset support but also guided towards high-value states. Therefore, SCAS is also strategically targeting high-value ID states.\n\n> it would benefit from a comparison with existing works like DW[1], which also focus on high-value data points for behavior regularization.\n\nThanks for the kind suggestion. We've taken your advice to include DW [1] in comparison. Results are presented in **Table 1** of the PDF (attached to the global response). DW reweights ID data points by their values for behavior regularization and does not account for OOD states during the test phase. In contrast, our approach considers an OOD state correction scenario, resulting in enhanced robustness during the test phase and better performance in both MuJoCo locomotion and AntMaze domains. We will discuss DW and include the comparison results in the latter revision.\n\n> Moreover, a clearer explanation of how this approach aligns with offline RL principles like pessimism and robustness to OOD states would greatly strengthen the manuscript's validity and impact.\n\nThanks for the suggestion. In a specific sense, SCAS, which unifies OOD state correction and OOD action suppression, also integrates pessimism and state robustness. (1) Regarding pessimism: The OOD action suppression effect of SCAS aligns with the pessimism commonly discussed in offline RL work (being pessimistic about OOD actions). Unlike traditional policy constraint methods, our approach does not require the training policy to align with the behavior policy; it only requires the successor states to be within the dataset support, which is a more relaxed constraint. (2) Regarding state robustness: The OOD state correction effect of SCAS is aimed at improving the agent's robustness to OOD states during the test phase. Compared with previous works, SCAS unifies OOD state correction and OOD action suppression and additionally achieves value-aware OOD state correction. Some offline RL literature on state robustness differs from our approach; they typically consider noisy observations [2], such as sensor errors. In contrast, SCAS addresses state robustness concerning actual OOD states encountered during test time, rather than noisy observations.\n\n**Q2: Lack of Comparisons with Comparable Offline RL Approaches.**\n\nThanks for the suggestion. Note that the original SCAS requires only one single hyperparameter configuration in implementations. For a fair comparison with DW[1]/EDAC[3]/RORL[2]/SQL[4]/EQL[4], we roughly select $\\lambda$ from {0.025, 0.25} for each dataset, referring to this variant as SCAS-ht. The results of SCAS-ht and these methods are reported in **Table 1** of the PDF. Among the ensemble-free methods, SCAS-ht achieves the highest performance in both mujoco locomotion and antmaze domains. Compared with ensemble-based methods, SCAS-ht also performs better on antmaze tasks. We will include the comparison results in the latter revision.\n\n**Due to the page limit, please refer to the next block. Thanks!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "caQI2CJwuA", "reviewer": "Reviewer_fu1b", "summary": "The issue of OOD actions is a well-known concern in offline RL research. The issue of OOD states, however, is relatively unexplored. In this paper, the authors shed light on the significance of OOD state correction and propose a SCAS algorithm to guide the agent back into high-value ID states when encountering an OOD state. To achieve this goal, SCAS first defines a conditional state transition probability $N^*(s'\\mid s)$ skewed towards high-valued states and regularizes the policy by aligning the dynamics induced by the policy on a perturbed state $\\hat{s}\\sim \\mathcal{N}(s, \\sigma^2I)$ with the value aware state transition probability $N^*(\\cdot \\mid s)$. Also, theoretical analysis shows that SCAS implicitly guarantees the correction of OOD actions, thereby removing the need for additional regularizers.", "questions": "Please refer to the **Weaknesses** section. As a side note, have you tried using the Q and V functions learned by IQL (or any other value-based offline RL algorithm) instead of learning them together with the policy? From Figure 9, the training process seems a bit unstable, so using a stationary training objective might help.", "limitations": "The authors have adequately addressed their work's limitations and potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The authors highlighted the significance of OOD state correction in offline RL, a problem other researchers often neglect. Also, multiple experiments were conducted to show the competence of SCAS and its robustness towards the choice of hyperparameters.", "weaknesses": "1. The authors did not cite works from the DICE literature(e.g., Nachum et al., 2019; Lee et al., 2021; Mao et al., 2024). Although they do not explicitly correct OOD states, I believe they are deeply related to SCAS because they aim to align the policy's state distribution $d^\\pi$ with the dataset's state distribution, thus preventing the policy from visiting OOD states.\n\n2. The choice of $\\exp(\\alpha V(s))$ as the empirical normalizer seems extremely arbitrary. Providing some rationale behind this choice is recommended.\n\n3. All of the experimental analysis was performed on MuJoCo environments, which have deterministic transition dynamics. Since SCAS utilizes a deterministic environmental model, it may not perform well in environments with stochastic transition dynamics.\n\n4. NeoRL is a relatively uncommon benchmark compared to D4RL. A brief explanation of NeoRL would be helpful for the readers.\n\n5. Some experimental details are missing from the paper.\n\n    * Do Figures 1a to 1d share the same embedding function? How was this embedding function obtained? Was it by running t-SNE on the set of 200,000 samples(50,000 samples each from the dataset, CQL, TD3+BC, and SCAS)?\n    * The points in Figure 1d seem to be the union of the points in Figures 1a, 1b, and 1c. Am I correct?\n    * Why does Figure 2 omit Q values of off-policy RL, SDC, and OSR for higher numbers of optimization steps?\n    * Section 6.3 states that varying steps of Gaussian noise were added to the actions during test time. Figure 3 indicates that the authors added noise up to 40 steps. However, trajectories are usually longer than 40 steps. How were those 40 steps selected? Did the authors add noise to the first 40 interactions with the environment? \n\n6. Minor comment: The meaning of the term *$\\mathcal{D}(s)$ weights* on Line 162 is unclear.\n\n\n### References\n\nLee, Jongmin, et al. \"Optidice: Offline policy optimization via stationary distribution correction estimation.\" International Conference on Machine Learning. PMLR, 2021.\n\nMao, Liyuan, et al. \"Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradient update.\" arXiv preprint arXiv:2402.00348 (2024).\n\nNachum, Ofir, et al. \"Algaedice: Policy gradient from arbitrary experience.\" arXiv preprint arXiv:1912.02074 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: The authors did not cite works from the DICE literature.**\n\nThanks for the suggestion. We will cite and add discussions on DICE literature as follows. \n\nAlthough the DICE series of works [1,2,3] share similar motivations with SCAS, there are significant differences between the two. Firstly, DICE is based on a linear programming framework of RL, while SCAS is based on a dynamic programming framework. Therefore, the theoretical foundations and learning paradigms of the two are inherently different. Secondly, SCAS only corrects encountered OOD states, whereas DICE algorithms require the policy-induced occupancy distribution $d^\\pi$ to align with the dataset distribution $d^\\mathcal D$. Therefore, DICE's constraints are stricter, potentially making it more susceptible to the average quality of the dataset. Lastly, theoretical and empirical evidence indicate that DICE algorithms have a problem of gradient cancellation [1], which imposes certain limitations on their practical effectiveness.\n\n**Q2: The choice of $\\exp(\\alpha V(s))$ as the empirical normalizer seems extremely arbitrary. Providing some rationale behind this choice is recommended.**\n\nThanks for the suggestion. Firstly, choosing $\\exp(\\alpha V(s))$ is intended to obtain something similar to the advantage function. With this normalizer, the weight of our regularizer is $\\exp(\\alpha (V(s') - V(s)))$, which is comparable to the weight $\\exp(\\alpha A(s,a))$ in Advantage Weighted Regression (AWR) [4]. Here, $V(s') - V(s)$ represents the relative advantage of the next state $s'$ compared to the current state $s$, while $A(s,a)$ reflects the relative advantage of taking action $a$ in $s$ compared to following the current policy. Comparison of the objectives of SCAS and AWR:\n\nSCAS: $\\exp(\\alpha (V(s’)-V(s))) \\log(M(s’|\\hat s,\\pi(\\hat s)))$;\n\nAWR: $\\exp(\\alpha A(s,a)) \\log\\pi(a|s)$.\n\nSecondly, as discussed in the paper (lines 159-166), introducing any normalizer that depends only on $s$ (independent of $s'$) does not affect the development and analysis of our method; it is merely for computational stability. In AWR-based methods, there also exists a normalizer $Z(s)$ and they usually disregard it. The rationale behind this is similar.\n\n**Q3: Since SCAS utilizes a deterministic environmental model, it may not perform well in environments with stochastic transition dynamics.**\n\nThanks for the comment. Although the implementation of SCAS uses a deterministic dynamics model, the SCAS framework is compatible with stochastic dynamics models. The reasons to employ deterministic models are (1) our considered environments retain in MuJoCo environments, which are mostly deterministic in state transitions; and (2) deterministic models are easy to train and implement in practice. We also admit that deterministic dynamics modeling might encounter more errors when the transition dynamics are multimodal. As mentioned in the limitation section, we expect to find a better dynamics model to further enhance our method in such scenarios.\n\n**Q4: A brief explanation of NeoRL would be helpful for the readers.**\n\nThanks for the kind suggestion. We will include the introduction as follows.\n\nNeoRL is a benchmark designed to simulate real-world scenarios by collecting datasets using a more conservative policy, aligning closely with realistic data collection scenarios. The narrow and limited data makes it challenging for offline RL algorithms.\n\n**Q5: Some experimental details are missing from the paper.**\n\nThanks for the comments and we apologize for any confusion resulting from the missing details. \n\n> Do Figures 1a to 1d share the same embedding function? How was this embedding function obtained? Was it by running t-SNE on the set of 200,000 samples(50,000 samples each from the dataset, CQL, TD3+BC, and SCAS)?\n\nYes. Figures 1a to 1d share the same embedding function obtained by running t-SNE on the set of all 200,000 samples (50,000 samples each from the dataset, CQL, TD3+BC, and SCAS). This ensures a clear visual comparison.\n\n> The points in Figure 1d seem to be the union of the points in Figures 1a, 1b, and 1c.\n\nYes. Figure 1d contains all the 200,000 samples, which is the union of the points in Figures 1a, 1b, and 1c.\n\n> Why does Figure 2 omit Q values of off-policy RL, SDC, and OSR for higher numbers of optimization steps?\n\nSorry for the confusion. Because the Q values of Off-policy RL, SDC w/o CQL, and OSR w/o CQL diverge in the early learning stage, plotting their Q values at later optimization steps would result in an excessive range on the vertical axis. For clearer presentation, we have replotted this figure in **Figure 2** of the PDF (attached to the global response), which adds more data points to the early learning stage.\n\n> How were those 40 steps selected? Did the authors add noise to the first 40 interactions with the environment?\n\nWe choose a maximum of 40 steps for perturbations because we empirically find that, under the noise magnitude of 0.5, adding more than 50 steps of noise often causes the agent to directly enter a terminal state during the perturbation, making it difficult to assess the agent's ability to correct itself from OOD states after the perturbation. Given that the episode length of our tasks is 1000 steps, we start applying perturbations at step 500 during test time. This choice is because, at the beginning, when the agent is still in its initial state and not yet in motion, applying perturbations is essentially like changing the initial state, which is not very meaningful. In contrast, applying perturbations at step 500 helps in evaluating the agent's robustness to perturbations while it is in motion.\n\n We will add these explanations to the latter revision.\n\n**Due to the page limit, please refer to the next block. Thanks!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **Global Response**

We thank all the reviewers for taking the time to read our manuscript carefully and for providing constructive and insightful feedback. We are encouraged by the positive comments of the reviewers, such as:

- Important, interesting, and overlooked research topic (Reviewers VgaK/j7HR/fu1b);
- Robust method requiring only one single hyperparameter configuration (Reviewers VgaK/j7HR/VyLC/fu1b), valuable in offline RL applications (Reviewers VgaK/VyLC);
- Solid theory / establishment of a theoretical link between OOD state correction and OOD action suppression (Reviewers j7HR/VyLC);
- Rigorous evaluation methodology, comprehensive assessment, and good empirical performance (Reviewers VyLC/fu1b/j7HR).

Meanwhile, we have been working hard to address the reviewers' concerns and questions and have provided detailed responses to the individual reviews below. We have also attached a **PDF** to this response containing the additional experiment results. Summary of the PDF:

- Performance of SCAS with slight hyperparameter tuning and comparison with additional baselines in Table 1.
- Experimental results of SCAS+CQL, SCAS+IQL, SCAS+TD3BC, and SCAS with Gaussian policy in Table 2.
- Experimental results of SCAS under different dynamics model errors in Figure 1.
- A replotted version of Figure 2 in the paper in Figure 2.

We hope our response could address the reviewers' concerns. We would be more than happy to resolve any remaining questions in the time we have and are looking forward to engaging in a discussion.

Meta-review: All reviewers, including myself as the Area Chair, commend the originality of this submission, particularly its focus on out-of-distribution (OOD) states rather than the more commonly explored OOD actions. The work is grounded in solid theoretical foundations and presents a novel method that is both simple and elegant. While the empirical results only represent an incremental advance, the submission has significant potential to pave the way for future impactful research in the field of Offline Reinforcement Learning. Therefore, I recommend its acceptance.
Decision: Accept (poster)