Title: Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
Venue: NeurIPS 2024
Accepted: yes
Keywords: offline reinforcement learning
Abstract: In offline reinforcement learning (RL), addressing the out-of-distribution (OOD) action issue has been a focus, but we argue that there exists an OOD state issue that also impairs performance yet has been underexplored. Such an issue describes the scenario when the agent encounters states out of the offline dataset during the test phase, leading to uncontrolled behavior and performance degradation. To this end, we propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL. Technically, SCAS achieves value-aware OOD state correction, capable of correcting the agent from OOD states to high-value in-distribution states. Theoretical and empirical results show that SCAS also exhibits the effect of suppressing OOD actions. On standard offline RL benchmarks, SCAS achieves excellent performance without additional hyperparameter tuning. Moreover, benefiting from its OOD state correction feature, SCAS demonstrates enhanced robustness against environmental perturbations.

Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression

Abstract

In offline reinforcement learning (RL), addressing the out-of-distribution (OOD) action issue has been a focus, but we argue that there exists an OOD state issue that also impairs performance yet has been underexplored. Such an issue describes the scenario when the agent encounters states out of the offline dataset during the test phase, leading to uncontrolled behavior and performance degradation. To this end, we propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL. Technically, SCAS achieves value-aware OOD state correction, capable of correcting the agent from OOD states to high-value in-distribution states. Theoretical and empirical results show that SCAS also exhibits the effect of suppressing OOD actions. On standard offline RL benchmarks, SCAS achieves excellent performance without additional hyperparameter tuning. Moreover, benefiting from its OOD state correction feature, SCAS demonstrates enhanced robustness against environmental perturbations.

1 Introduction

Deep reinforcement learning (RL) shows promise in solving sequential decision-making problems, gaining increasing interest for real-world applications [42, 57, 63, 53, 7]. However, deploying RL algorithms in extensive scenarios poses persistent challenges, such as risk-sensitive exploration [13] and time-consuming episode collection [27]. Recent advances view offline RL as a hopeful solution to these challenges [34]. Offline RL aims to learn a policy from a fixed dataset without further interactions [32]. It can tap into existing large-scale datasets for safe and efficient learning [23, 37, 50].

In offline RL research, a well-known concern is the out-of-distribution (OOD) action issue: the evaluation of OOD actions causes extrapolation error [12], which can be exacerbated by bootstrapping and result in severe value overestimation [34]. To address this issue, a large body of work has emerged to directly or indirectly suppress OOD actions during training, employing various techniques such as policy constraint [12, 30, 10], value penalization [31, 2, 6], and in-sample learning [29, 14, 71].

Distinguished from most previous works, this paper argues that, apart from the OOD action issue, there exists an OOD state issue that also impairs performance yet has received limited attention in the field. Such an issue refers to the agent encountering states out of the offline dataset during the policy deployment phase (i.e., test phase). The occurrence of OOD states can be attributed to OOD actions, stochastic environments, and real-world perturbations. Since typical offline RL algorithms do not involve policy training in OOD states, the agent tends to behave in an uncontrolled manner once entering OOD states in the test phase. This can further exacerbate the state deviation from the offline dataset and lead to severe degradation in performance [34, 75].

In mitigating this OOD state issue, existing limited work attempts to train the policy to correct the agent from OOD states to in-distribution (ID) states [75, 22]. Technically, Zhang et al. [75] construct a dynamics model and a state transition model and align them to guide the agent to ID regions, while

(a) CQL

(b) TD3BC

(c) SCAS (Ours)

(d) Optimal Value

StateDistribution

StateDistribution

StateDistribution

OptimalValueofEachState

value

CQL

TD3BC

SCAS

Dataset

Dataset

Dataset

800

600

400

200

0

200

Jiang et al. [22] resort to an inverse dynamics model for policy constraint. However, they deal with the OODstate and OOD action issues separately, requiring extra OOD action suppression components and complex distribution modeling, which sacrifices computational efficiency and algorithmic simplicity. Moreover, correcting the agent to all ID states impartially could be problematic, especially when the dataset contains substantial suboptimal states. As a result, the performance of prior methods also leaves considerable room for improvement.

In this paper, we aim to address these two fundamental OOD issues simultaneously by proposing a simple yet effective approach for offline RL. We term our method SCAS due to its integration of OOD S tate C orrection and OOD A ction S uppression. We start with solving an analytical form of a value-aware state transition distribution, which is within the dataset support but skewed toward highvalue states. Then, we align it with the dynamics induced by the trained policy on perturbed states via KL divergence. This operation intends to correct the agent from OOD states to high-value ID states, a concept we refer to as value-aware OODstate correction. Through some derivations, it also eliminates the necessity of training a multi-modal state transition model. Furthermore, we show theoretically and empirically that, while designed for OOD state correction, SCAS regularization also exhibits the effect of OOD action suppression. We evaluate SCAS on the offline RL benchmarks including D4RL [9] and NeoRL [49]. SCAS achieves excellent performance with consistent hyperparameters without additional tuning. Moreover, benefiting from its OOD state correction ability, SCAS demonstrates improved robustness against environmental perturbations.

To summarize, the main contributions of this work are:

We systematically analyze the underexplored OOD state issue in offline RL and propose a simple yet effective approach SCAS unifying OOD state correction and OOD action suppression .

Our approach achieves value-aware OOD state correction, which circumvents modeling complex distributions and significantly improves performance over vanilla OOD state correction methods.

Empirically 1 , our approach demonstrates superior performance on standard offline RL benchmarks and enhanced robustness in perturbed environments without additional hyperparameter tuning .

2 Preliminaries

In reinforcement learning, we generally characterize the environment as a Markov Decision Process (MDP) M = ( S , A , P, R, γ, d 0 ) , with state space S , action space A , transition dynamics P : S × A → ∆( S ) , reward function R : S × A → R , discount factor γ ∈ [0 , 1) , and initial state distribution d 0 [61]. The agent interacts with the environment and seeks a policy π : S → ∆( A ) to maximize the expected discounted return η ( π ) :

For any policy π , we define the value function as V π ( s ) = E π [ ∑ ∞ t =0 γ t R ( s t , a t ) | s 0 = s ] and the state-action value function ( Q -value function) as Q π ( s, a ) = E π [ ∑ ∞ t =0 γ t R ( s t , a t ) | s 0 = s, a 0 = a ] .

Offline RL. In offline RL, the agent can only access a static dataset D = { ( s i t , a i t , s i t +1 , r i t ) } . We denote the empirical behavior policy of D by β ( a | s ) and the empirical dynamics model by M ( s ′ | s, a ) , both of which depict the conditional distributions observed in the dataset [12]. Typical actor-critic algorithms [56, 18] evaluate policy π by minimizing Bellman loss:

where π ϕ and Q θ are the parameterized policy and Q function, and Q θ ′ is a target network whose parameters are updated via Polyak averaging [42].

Simultaneously, policy improvement in policy iteration is achieved via maximizing the Q-value:

OOD action issue. In offline RL, OOD actions refer to actions outside the support of the behavior policy β ( ·| s ) at a specific state s ∈ D . Since the Q-values of OOD actions can be poorly estimated and the policy improvement is towards maximizing the estimated Q θ , the resulting policy tends to prioritize the OOD actions with overestimated values, leading to poor performance [12].

3 OODState Correction

The following focuses on the OOD state issue and OOD state correction in offline RL. In Section 3.1, we systematically analyze the OOD state issue, introduce the concept of OOD state correction, and point out limitations of prior methods. Then we present the proposed approach SCAS in Section 3.2.

3.1 OODState Issue in Offline RL

In offline RL, OOD states refer to states not in the offline dataset. The OOD state issue (Definition 1) pertains to scenarios where the agent enters OOD states during the test phase, potentially resulting in catastrophic failure [34]. However, such a topic is rarely investigated in the literature, and existing studies lack deep insights. We mathematically formulate the OOD state issue as follows.

Definition 1 (OOD state issue) . There exists s ∈ S , such that d π M T ( s ) > 0 and d D ( s ) = 0 , where M T is the MDP of the test environment, π is any learned policy, d π M T is the state probability density induced by π in M T , and d D is the state probability density in the offline dataset.

Origins and consequence of OOD states. During the test phase, the OOD states occur primarily in three scenarios: (i) OOD actions: the learned policy, not perfectly constrained within the support of the behavior policy, executes unreliable OOD actions, leading to OOD states. (ii) Stochastic environment: the initial state of the actual environment may fall outside the offline dataset. In addition, stochastic dynamics can also lead to states outside the dataset, even when taking ID actions in ID states. (iii) Perturbations: commonly seen in real-world robot applications, some unexpected perturbations can propel the agent into OOD states (e.g., wind, human interference).

During offline training, the typical Bellman updates involve only ID states, and the policies in OOD states are not trained. As a result, when encountering OOD states in the test phase, the agent would exhibit uncontrolled behavior, and the state deviation from the offline dataset can be further exacerbated over time steps, severely degrading performance [34].

OODstate correction. To mitigate this OOD state issue, an intuitive solution is to train a policy capable of correcting the agent from OOD states to ID states, a concept known as OOD state correction [75]. Specifically, during offline training, we can perturb the original state s in the dataset into ˆ s to generate substantial OOD states. Then consider the scenario where the agent starts from ˆ s , follows the trained policy π , and transitions to the next state ˆ s ′ . To reduce state deviation, ˆ s ′ is expected to be close to the offline dataset. Thus we can align the distribution of ˆ s ′ with an ID state distribution to regularize the policy and achieve OOD state correction.

Continuing the above train of thought, SDC [75] generates the ID state distribution by feeding the original state s into a trained state transition model N ( s ′ | s ) of the dataset. This model characterizes

the conditional state transition distribution in the dataset and is implemented by a conditional variational auto-encoder (CVAE) [58]. After pretraining a dynamics model M ( s ′ | s, a ) and the state transition model N ( s ′ | s ) , SDC introduces the following policy regularizer for OOD state correction:

where ˆ s is a Gaussian noise perturbed version of the original state s , σ is the standard deviation of the Gaussian, M ( ·| ˆ s, π ( ·| ˆ s )) is shorthand for E ˆ a ∼ π ( ·| ˆ s ) M ( ·| ˆ s, ˆ a ) , and MMD is the maximum mean discrepancy measure. More recently, OSR [22] directly aligns the trained policy distribution at the perturbed state with a CVAE inverse dynamics model to constrain the policy in OOD states.

Limitations. However, the regularizers of prior methods are only designed to deal with this OOD state issue. To mitigate OOD actions, they require an additional conservative Q learning (CQL) term [31] in value estimation to penalize Q-values of OOD actions. In addition, the state transition distribution and the inverse dynamics distribution are multi-modal in many scenarios [43]. The necessity of extra OOD action suppression components and complex distribution modeling compromises their computational efficiency and algorithmic simplicity. Moreover, correcting the agent to all ID states impartially could be problematic, particularly when the offline dataset contains a large portion of suboptimal states. In such cases, vanilla OOD state correction can lead to suboptimal behaviors. Consequently, there is also significant potential for improvement in the performance of prior methods.

For a more comprehensive discussion of related work, please refer to Appendix A.

3.2 Value-aware OOD State Correction

The objective of this work is to formulate a simple yet effective policy regularizer for offline RL that unifies OOD state correction and OOD action suppression. Moreover, we aim to achieve value-aware OOD state correction, involving the correction of the agent from OOD states to high-value ID states.

Value-aware state transition. For the ID state distribution to which the agent is corrected, we expect a value-aware state transition distribution N ∗ ( ·| s ) that lies within the support of the dataset state transition distribution N ( ·| s ) but is skewed toward high-value states s ′ . To ensure stability and, more importantly, to enable our subsequently designed algorithm to circumvent modeling complex distributions, we seek a soft optimal version of it. To this end, we consider the following problem 2 :

where α is a hyperparameter to balance the two terms.

The optimization problem above has a closed-form solution:

where Z ( s ) = ∑ s ′ exp( αV ( s ′ )) N ( s ′ | s ) is a normalization factor. It can be seen from Eq. (6) that supp( N ∗ ( ·| s )) ⊆ supp( N ( ·| s )) . Note that α is a key hyperparameter that controls the significance of the values of next states in SCAS's OOD state correction. As α increases, N ∗ ( ·| s ) becomes more skewed toward the optimal s ′ in the support of N ( ·| s ) .

OODstate correction. In order to produce substantial OOD states, we perturb each state s ∈ D with Gaussian noise N (0 , σ 2 ) , resulting in perturbed state ˆ s . It is worth noting that the dataset used for RL training remains unchanged. We perturb the states solely to formulate the regularizer.

We anticipate the following value-aware OOD state correction scenario, where the agent starts from OOD state ˆ s , follows the trained policy π , and transitions to the high-value ID state s ′ in the distribution of N ∗ ( ·| s ) . To this end, we train the policy π to align the dynamics induced by π on the perturbed state ˆ s with the value-aware state transition distribution at the original state s via KL divergence. That is, we regularize π by minimizing:

By substituting the analytical solution of N ∗ from Eq. (6) into the KL divergence, we have

Note that N is the state transition distribution in the dataset, and s ∼ D , s ′ ∼ N ( ·| s ) is equivalent to ( s, s ′ ) ∼ D . Thus minimizing Eq. (7) is equivalent to maximizing following regularizer:

As a result, R ( π ) effectively eliminates the need for a pre-trained multi-modal state transition model ( N or N ∗ ) and enables direct sampling from the dataset for optimization.

However, the normalization factor Z ( s ) in R ( π ) can be challenging to compute. We note that the regularizer R ( π ) is derived from the minimization of the KL divergence in Eq. (7). Since we aim to minimize this KL at every state s in D and Z ( s ) only affects the relative weights at different s , it matters less to precisely restore the correct state weights in D by computing Z ( s ) , which is empirically hard to estimate and may bring more instability. Thus, we replace Z ( s ) in R ( π ) with an empirical normalizer exp( αV ( s )) for computational stability:

We provide further rationale behind this choice of the empirical normalizer in Appendix C.1.

Tractable optimization. Now we shift focus to the optimization of R 1 ( π ) . The expectation with respect to π can be moved outside the logarithm by Jensen's inequality:

where the equality holds when π is deterministic. In general, it is convenient to maximize the lower bound in Eq. (10) using the reparameterization trick. However, to ensure the equality case in Eq. (10), we opt to train a deterministic policy π . In this case, we can directly maximize R 1 ( π ) by computing the gradient of π using automatic differentiation [46].

In contrast to model-based RL methods that typically use the learned dynamics model to roll out multi-step trajectories for policy training [20, 73], our algorithm utilizes the dynamics model to propagate the gradient of policy and regularize policy training, resulting in significantly enhanced computational efficiency. Moreover, the nature of one-step dynamics prediction in our method is advantageous for maintaining relatively high prediction accuracy.

4 Analysis of OOD Action Suppression

This section focuses on the OOD action issue and shows that the proposed regularizer also exhibits the effect of OOD action suppression . In other words, it can also prevent the policy from taking OOD actions, thereby simultaneously addressing the fundamental OOD action issue in offline RL. In offline RL, OOD actions are exclusively defined on ID states. This is because actor-critic training is limited to ID states, and any actions on OOD states would not affect training and cause the OOD action issue mentioned in Section 2. Consequently, for the analysis of OOD actions, it is essential to consider ID states. We define ¯ R, ¯ R 1 as the ID state version of R,R 1 , where ˆ s = s . ¯ R and ¯ R 1 can be regarded as special cases of R and R 1 , when ˆ s sampled from N ( s, σ 2 ) is equal to s :

The proposed regularizer functions as follows: when the agent encounters OOD states, it drives the agent to choose actions leading to ID states, as discussed in Section 3.2. When the agent is in ID states, the ID state part of it comes into play. In the following, we show that it helps circumvent taking OOD actions by analyzing the maximizer of ¯ R, ¯ R 1 in tabular MDPs.

Proposition 1. Suppose that the environment dynamics is deterministic, then both ¯ R ( π ) and ¯ R 1 ( π ) achieve their global maximum at the policy π ∗ , where 3

The support of π ∗ is within that of the behavior policy β :

and π ∗ makes the following equation hold:

Under the deterministic dynamics condition, Proposition 1 shows that π ∗ is constrained within the support of the behavior policy. Thus, our regularizer helps to keep the policy from taking OOD actions. Moreover, π ∗ is able to exactly align M ( ·| s, π ∗ ( ·| s )) with N ∗ ( ·| s ) , indicating the guidance of the agent to the high-value ID state distributions.

Furthermore, we show in Proposition 2 that even under stochastic dynamics, the optimization of ¯ R and ¯ R 1 still yields policies constrained within the support of β . Hence, SCAS also exhibits the effect of OOD action suppression in this more general scenario.

Proposition 2. When the dynamics is stochastic, the maximizers of both ¯ R ( π ) and ¯ R 1 ( π ) are constrained within the support of the behavior policy:

5 Implementation Details

SCAS is easy to implement and we design the practical algorithm to be as simple as possible, retaining algorithmic simplicity and improving computational efficiency.

Dynamics model. We employ a deterministic dynamics model M ω . The loss for training the model is

Policy improvement. With a deterministic model, we replace the log-likelihood in R 1 ( π ) with mean squared error. It is a common approach in RL algorithms to convert a maximum likelihood estimation problem into a re-

Algorithm 1 SCAS

1: Initialize dynamics model M ω , policy network π ϕ , Q -network Q θ , and target Q -network Q θ ′

2: // Dynamics Model Training

3: for each gradient step do

4: Update ω by minimizing L M ( ω ) in Eq. (18)

5: end for

6: // Policy Training

7: for each gradient step do

8: Update θ by minimizing L Q ( θ ) in Eq. (2)

9: Update ϕ by maximizing J π ( ϕ ) in Eq. (20)

10: Update target network: θ ′ ← (1 -τ ) θ ′ + τθ 11: end for

gression problem when dealing with Gaussians with fixed variance [10]. As discussed in Section 3.2, we also adopt a deterministic policy model π ϕ . Thus, we have the following policy regularizer:

where V θ ( s ) = Q θ ( s, ¯ π ϕ ( s )) and ¯ π ϕ means π ϕ with detached gradients. Using deterministic policy also simplifies the training process without learning a V -function. Combining R 2 ( π ϕ ) with the standard policy improvement objective, we update the policy by maximizing:

where λ is a hyperparameter to balance the two terms. Additionally, following TD3+BC [10], we also normalize Q θ in the first term in each mini-batch to maintain a balanced scale across tasks.

Overall algorithm. Putting everything together, we present our final algorithm in Algorithm 1.

6 Experiments

In this section, we conduct several experiments to examine the performance and properties of SCAS. Please refer to Appendices D and E for experimental details and additional results.

6.1 Empirical Evidence of OOD State Correction and OOD Action Suppression

OODstate correction. To examine the OOD state correction ability, we compare the state distributions generated by the learned policies of different algorithms with the state distribution of the offline dataset. In detail, we first train SCAS, CQL [31], and TD3+BC [10], and then collect 50,000 samples by running the trained policies separately. We also sample 50,000 states randomly from the offline dataset for comparison. Figures 1(a) to 1(c) plot the state distributions in halfcheetah-mediumexpert [9] with t-SNE [62], and Figure 1(d) visualizes the optimal value of each state. We access these values from the learned value function obtained by running TD3 [11] online to convergence.

In Figures 1(a) and 1(b), we observe that the policies learned by CQL and TD3+BC tend to produce OOD states. As depicted in Figure 1(d), these OOD states have extremely low values, so entering them can be detrimental to performance. In contrast, the state distribution induced by SCAS is almost entirely within the support of the offline distribution, demonstrating the OOD state correction ability of SCAS. Moreover, we also note that in the low-value area of the offline state distribution (the grey circle in Figure 1(d)), SCAS exhibits a very low state density, which could be attributed to SCAS's value-aware OOD state correction. We refer the reader to Appendix E.2 for additional experiments validating the OOD state correction effects.

OODaction suppression. We empirically evaluate the OOD action suppression effects through the lens of value estimates. We compare SCAS with three baselines: (1) ordinary offpolicy RL which is SCAS with λ = 0 (all other implementations are the same); (2) SDC [75] without additional CQL [31] term to suppress OOD actions; (3) OSR [22] without additional CQL term. We conduct experiments on D4RL datasets [9]. Since value over-estimation (divergence) is the main consequence and evidence of OOD actions [12], we plot the learned Q-values of SCAS and the baselines in Figure 2.

0

0.1

0.2

0.3

0.4

0.5

Gradient Steps

(×10 6 )

10

3

10

4

hopper-medium-v2

SCAS: MC return

SCAS: Q

Off-policy RL: Q

SDC w/o CQL: Q

OSR w/o CQL: Q

0

0.1

0.2

0.3

0.4

0.5

Gradient Steps

(×10 6 )

10

3

10

4

10

5

hopper-medium-expert-v2

We also include the oracle Q-values of SCAS by rollouting the trained policy for 1 , 000 episodes and evaluating the Monte-Carlo return. Additional results are provided in Appendix E.1.

The results show that the learned Q-values of ordinary off-policy RL, SDC without CQL, and OSR without CQL diverge at early learning stages, suggesting that the algorithms suffer from severe OOD actions. By contrast, the learned Q-values of SCAS stay close to the oracle Q-values. This indicates that SCAS regularization alone is able to suppress OOD actions.

6.2 Comparisons on Offline RL Benchmarks

Tasks. We evaluate SCAS on D4RL [9] and NeoRL [49] benchmarks. In D4RL, we conduct experiments on Gym locomotion tasks and much more challenging AntMaze tasks. Due to the space limit, the results on NeoRL are deferred to Table 4 in Appendix E.3.

Baselines. We compare SCAS with prior state-of-the-art offline RL methods as well as the ones specifically designed for OOD state correction, including BC [48], MOPO [73], OneStep RL [5], CQL [31], TD3+BC [10], IQL [29], SDC [75] and OSR [22].

Hyperparamter tuning. Offline RL methods are appealing for their ability to generate effective policies without online interaction. Nevertheless, many existing offline RL works involve datasetspecific hyperparameter tuning. The reduction of hyperparameter tuning is crucial for improving practical applicability. In this work, SCAS uses a single set of hyperparameters for all datasets in D4RL and NeoRL benchmarks to obtain the reported results.

Comparisons with baselines. On D4RL, comparisons of performance, runtime, and hyperparameter tuning information are shown in Table 1. We refer the reader to Appendix E.8 for learning curve details of SCAS. On the Gym locomotion tasks, SCAS outperforms prior methods on most datasets and achieves the highest total score with a single set of hyperparameters. On the challenging AntMaze tasks, SCAS performs better than IQL and outperforms other methods by a very large margin. In NeoRL (Table 4), SCAS performs comparably to MOBILE [59] and outperforms other baselines.

Runtime. Wepresent the runtime of algorithms at the bottom of Table 1. SCAS exhibits significantly lower runtime than MOPO, SDC, and OSR and is comparable to other model-free baselines.

Generality. SCAS is a generic model-based regularizer that can be easily integrated into existing offline RL algorithms. The corresponding results and analysis are provided in Appendix E.5.

6.3 Comparisons in Perturbed Environments

In this section, we evaluate the algorithms in a more real-world setting where the agent receives uncertain perturbations during test time. OOD state correction is even more critical in such scenarios since the agent can enter OOD states after perturbation. To simulate this scenario, we add varying steps of Gaussian noise with a magnitude of 0 . 5 to the actions conducted by the policy during test time. Specifically, the policy is trained on standard D4RL datasets but is tested in the perturbed environments. We control the strength of perturbations by adjusting the number of perturbation steps.

0

10

20

30

40

Perturbation Steps

20

70

120

Normalized Return

hopper-medium-expert-v2

0

10

20

30

40

Perturbation Steps

20

70

120

Normalized Return

hopper-medium-replay-v2

0

10

20

30

40

Perturbation Steps

20

70

120

Normalized Return

hopper-medium-v2

0

10

20

30

40

Perturbation Steps

60

90

120

Normalized Return

walker2d-medium-expert-v2

SCAS

SDC

CQL

TD3BC

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

Normalized Return

antmaze-large-diverse-v2

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

halfcheetah-med-exp-v2

=0

=0.5

=1

=5

=10

(a) Effects of the inverse temperature

α

.

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

Normalized Return

antmaze-large-diverse-v2

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

halfcheetah-med-exp-v2

=0

=0.25

=0.5

=0.75

=1

(b) Effects of the balance coefficient

λ

.

Figure 3 shows the results of TD3+BC, CQL, SDC, and SCAS on various datasets over five random seeds. We observe that SCAS consistently outperforms previous methods across different perturbation levels and also exhibits less performance degradation against perturbations. Therefore, SCAS enjoys better robustness against perturbations in the complex and unpredictable environments.

6.4 Parameter Study

We examine the effects of the inverse temperature α , the balance coefficient λ , and the noise scale σ . Due to the space limit, the results for σ and on additional datasets are deferred to Appendix E.6. A sensitivity analysis on dynamics model errors is also provided in Appendix E.7.

Inverse temperature α . α is the key hyperparameter in SCAS for achieving value-aware OOD state correction. If α = 0 , the effect degenerates to vanilla OOD state correction. Figure 4(a) displays the learning curves of SCAS with different α . The results show that a large α is crucial for achieving good performance (also verified on more tasks in Figure 6), clearly demonstrating the effectiveness of our value-aware OOD state correction. However, too large α ( α = 10 ) induces less satisfying performance, probably due to the increased variance of the learning objective.

Balance coefficient λ . λ in Eq. (20) controls the balance between vanilla policy improvement and SCAS regularization. We vary λ within the range [0 , 1] and present the learning curves of SCAS in Figure 4(b). Notably, SCAS is able to converge to good performance over a very wide range of λ (also verified on more tasks in Figure 7). An interesting finding is that even when λ = 1 and the signal from RL improvement (max Q) is removed, SCAS still performs well on most tasks. This could be attributed to the fact that value-aware OOD state correction implies some sort of improvement in policy by maximizing the values of policy-induced next states.

7 Conclusion and Limitations

In this paper, we systematically analyze the OOD state issue in offline RL and propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression . SCAS also achieves value-aware OOD state correction, significantly improving performance over vanilla

OOD state correction. Empirical results validate the properties of SCAS, showcasing its superior performance on the offline RL benchmarks and its enhanced robustness in perturbed environments.

However, our work also has some limitations. For example, current SCAS primarily focuses on continuous control tasks. In discrete settings, algorithmic components like state perturbation strategy would be different, which would be an interesting direction for future work. Moreover, we anticipate employing more advanced dynamics models, such as ensembles [73] and diffusion models [21], to further improve the performance of our method.

Acknowledgment

We thank the anonymous reviewers for feedback on an early version of this paper. This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801, National Natural Science Foundation of China under Grant 61827804.

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436-7447, 2021.

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=Y4cs1Z3HnqL .

Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028 , 2023.

David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in Neural Information Processing Systems , 34:4933-4946, 2021.

Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint arXiv:2202.02446 , 2022.

Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897): 414-419, 2022.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=SJ0Lde3tRL .

Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning , pages 3682-3691. PMLR, 2021.

Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review , 55(2):895-943, 2022.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Zhang-Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing Systems , 36:4985-5009, 2023.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems , 32, 2019.

Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , pages 9902-9915. PMLR, 2022.

Ke Jiang, Jia-Yu Yao, and Xiaoyang Tan. Recovering from out-of-sample states via inverse dynamics in offline reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Modelbased reinforcement learning for atari. arXiv preprint arXiv:1903.00374 , 2019.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems , 33:21810-21823, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , pages 5774-5783. PMLR, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=68n2s9ZJWF8 .

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing offpolicy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33: 1179-1191, 2020.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning , pages 45-73. Springer, 2012.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR, 2021.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems , 30, 2017.

Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https:// openreview.net/forum?id=VYYf6S67pQc .

Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research , 36(1):3-15, 2017.

Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. ODICE: Revealing the mystery of distribution correction estimation via orthogonal-gradient update. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=L8UNn7Llt4 .

Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning , pages 23829-23851. PMLR, 2023.

Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations , 2021. URL https://openreview. net/forum?id=3hGNqpI4WS .

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning , 16(1):1-118, 2023.

Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 , 2019.

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems , 1, 1988.

Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:24753-24765, 2022.

Yun Qu, Boyuan Wang, Jianzhun Shao, Yuhang Jiang, Chen Chen, Zhenbin Ye, Linc Liu, Junfeng Yang, Lin Lai, Hongyang Qin, et al. Hokoff: Real game dataset from honor of kings and its offline reinforcement learning benchmarks. In Thirty-seventh Conference on Neural Information Processing Systems Track on Datasets and Benchmarks , 2023.

Yun Qu, Boyuan Wang, Yuhang Jiang, Jianzhun Shao, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Choices are more important than efforts: Llm enables efficient multi-agent exploration. arXiv preprint arXiv:2410.02511 , 2024.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research , 21(178):1-51, 2020.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conservative q learning for offline multi-agent reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/ forum?id=62zmO4mv8X .

Jianzhun Shao, Hongchang Zhang, Yun Qu, Chang Liu, Shuncheng He, Yuhang Jiang, and Xiangyang Ji. Complementary attention for multi-agent reinforcement learning. In International Conference on Machine Learning , pages 30776-30793. PMLR, 2023.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning , pages 387-395. Pmlr, 2014.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems , 28, 2015.

Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning. In International Conference on Machine Learning , pages 33177-33194. PMLR, 2023.

Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin , 2(4):160-163, 1991.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Cheems Wang, Yiqin Lv, Yixiu Mao, Yun Qu, Yi Xu, and Xiangyang Ji. Robust fast adaptation from adversarially explicit task distribution generation. arXiv preprint arXiv:2407.19523 , 2024.

Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference on Machine Learning , pages 23055-23077. PMLR, 2022.

Qi Wang, Yiqin Lv, Zheng Xie, Jincai Huang, et al. A simple yet effective strategy to robustify the meta learning paradigm. Advances in Neural Information Processing Systems , 36, 2024.

Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems , 33:7768-7778, 2020.

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=KCXQ5HoM-fy .

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellmanconsistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=ueYYgo2pSSU .

Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline reinforcement learning via conservative smoothing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=_QzJJGH_KE .

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems , 34:28954-28967, 2021.

Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xiangyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9022-9030, 2022.

Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= dfDv0WU853R .

A Related Work

Model-free offline RL. In offline RL, extrapolation error and overestimation caused by OOD actions pose significant challenges. Among model-free solutions, value regularization methods penalize the Q -values of OOD actions [31, 2, 28, 36, 3, 72, 40], while policy constraint approaches compel the trained policy to be close to the behavior policy, either explicitly via divergence penalties [69, 30, 10], implicitly by weighted behavior cloning [47, 45, 67, 39], or directly through specific parameterization of the policy [12, 15]. Relatively independently, in-sample learning methods formulate the Bellman target using only the actions in the dataset to avoid OOD actions [5, 29, 76, 71]. Recently, some works aim to learn the optimal policy within the support of the dataset (known as in-support or in-sample optimal policy) in a theoretically sound way and are less affected by the average quality of the dataset [39, 40, 68]. However, existing popular offline RL approaches primarily focus on the OOD action issue during training and often neglect the OOD state issue during the test phase.

Model-based offline RL. Model-based RL methods learn a model of the environment and generate synthetic data from that model to optimize the policy [60, 20, 24]. To ensure conservatism in offline RL, Kidambi et al. [25] and Yu et al. [73] estimate the uncertainty in the model and apply reward penalties for state-action pairs with high uncertainty. Some model-based approaches also introduce conservatism similarly to model-free ones, employing techniques like value regularization [74] and policy constraint [41]. Recently, Sun et al. [59] conducts uncertainty quantification through the inconsistency of Bellman estimations under the learned dynamics ensemble. However, model-based methods often come with a high computational burden [20], and their effectiveness relies heavily on the quality of the trained model [43]. In contrast, our algorithm leverages the dynamics model to propagate policy gradients, make one-step predictions, and regularize policy training, leading to significantly improved computational efficiency and relatively high prediction accuracy.

OODstate correction. In offline RL, OOD state correction deserves more attention as the state deviation during the test phase can accumulate over time steps, severely degrading performance [34]. Existing limited solutions aim to train the policy to correct the agent from OOD states to ID states [75, 22]. Specifically, SDC [75] builds a dynamics model and a state transition model, and aligns the policy-induced next state distributions at OOD states with the state transition model. On the other hand, OSR [22] utilizes an inverse dynamics model to constrain the policy at OOD states. Compared with prior methods, our proposed SCAS efficiently unifies OOD state correction and OOD action suppression in offline RL and additionally achieves value-aware OOD state correction. The DICE series of works [44, 33, 38] share similar motivations with SCAS to some extent; however, there are significant differences between the two. Firstly, DICE is based on a linear programming framework of RL, while SCAS is based on a dynamic programming framework. Therefore, the theoretical foundations and learning paradigms of the two are inherently different. Secondly, SCAS only corrects encountered OOD states, whereas DICE algorithms require the policy-induced occupancy distribution to align with the dataset distribution. Therefore, DICE's constraints are stricter, potentially making it more susceptible to the average quality of the dataset. Lastly, theoretical and empirical evidence indicate that DICE algorithms have a problem of gradient cancellation [38], which imposes certain limitations on their practical effectiveness.

B Proofs

In this section, we present the proofs for the theories in the paper.

B.1 Derivation of the Value-aware State Transition Distribution

We show that Eq. (6) is the optimal solution of the optimization problem Eq. (5):

We can optimize N ∗ at each s ∈ D separately. Thus we consider the following optimization problem:

This constrained optimization problem is convex, and the Lagrangian is:

The KKT condition gives:

Solving for ˜ N gives the closed form solution N ∗ :

By the condition ∑ s ′ N ∗ ( s ′ | s ) = 1 , we can directly solve the Lagrangian multiplier ν and replace exp( ν -1) with a normalization factor:

where Z ( s ) = ∑ s ′ exp( αV ( s ′ )) N ( s ′ | s ) is the normalization factor.

B.2 Proof of Proposition 1

Proposition 3 (Proposition 1 in the main paper) . Suppose that the environment dynamics is deterministic, then both ¯ R ( π ) and ¯ R 1 ( π ) achieve their global maximum at the policy π ∗ , where 4

The support of π ∗ is within that of the behavior policy β :

and π ∗ makes the following equation hold:

Proof. We start with ¯ R ( π ) .

The third equality holds because of the relationship between N ∗ and N in Eq. (6):

Therefore, the maximizer of ¯ R ( π ) is equal to the solution of the minimization problem in Eq. (35). Now consider the two distributions N ∗ ( ·| s ) and M ( ·| s, π ( ·| s )) in Eq. (35).

For analytical clarity, we use the notation M with slightly different meanings in different cases: in the stochastic setting, M : S × A → ∆( S ) ; in the deterministic setting, M : S × A → S . With the deterministic dynamics assumption,

We first show that π ∗ is a valid policy, that is, π ∗ is normalized.

Therefore, ∑ a π ∗ ( a | s ) = 1 .

Substitute Eq. (44) into Eq. (41),

Comparing Eq. (43) with Eq. (50), it holds that N ∗ ( s ′ | s ) = M ( s ′ | s, π ∗ ( ·| s )) , ∀ s ∼ D . As a result,

On the other hand,

Now we define π ∗ ( a | s ) as

Considering the non-negativity of KL divergence, the optimization problem in Eq. (35) achieves its global minimum at π ∗ . Therefore, ¯ R ( π ) also achieves its global maximum at π ∗ .

Now we consider ¯ R 1 ( π ) .

The fourth equality holds because of the relationship between N ∗ and N in Eq. (36).

As shown above, it holds that N ∗ ( s ′ | s ) = M ( s ′ | s, π ∗ ( ·| s )) , ∀ s ∼ D . As a result,

Considering Z ( s ) / exp( αV ( s )) > 0 and the non-negativity of KL divergence, the optimization problem in Eq. (58) achieves its global minimum at π ∗ . Therefore, ¯ R 1 ( π ) also achieves its global maximum at π ∗ .

In conclusion, when the environment dynamics is deterministic, both ¯ R ( π ) and ¯ R 1 ( π ) achieve their global maximum at the policy π ∗ , and π ∗ makes the following equation hold:

Moreover, because π ∗ ( a | s ) = 1 Z ( s ) exp( αV ( M ( s, a ))) β ( a | s ) , the support of π ∗ is included by that of the behavior policy β :

B.3 Proof of Proposition 2

Proposition 4 (Proposition 2 in the main paper) . When the dynamics is stochastic, the maximizers of both ¯ R ( π ) and ¯ R 1 ( π ) are constrained within the support of the behavior policy:

Proof. We start with ¯ R ( π ) .

Let π denote any valid policy. For ∀ s ∈ D , define ϵ ( s ) and n ( s ) as follows:

For ∀ s ∈ D , there exists at least one action a such that ( s, a ) ∈ D . Thus it holds that n ( s ) > 0 , ∀ s ∈ D . Then for ∀ s ∈ D , ∀ π , define π in as follows:

π in can be seen as a projection of π onto β 's support. Besides, for ∀ s ∈ D ,

Thus π in is a valid policy.

Now we compare ¯ R ( π in ) with ¯ R ( π ) . For ∀ ( s, s ′ ) ∈ D ,

=

a

|

β

0

(77)

(78)

The second equality holds because, in tabular MDPs, the empirical dynamics model M exactly computes the conditional distribution observed in the dataset. For transitions not contained in the dataset, M = 0 [12]. The final inequality holds because ϵ ( s ) ≥ 0 .

Therefore,

∑

(

|

a

s

)

M

(

s

′

|

s, a

)

ϵ

(

n

s

(

)

s

)

{

>

}

0

Now suppose π is not constrained within the support of the behavior policy at some state s 1 ∈ D : supp( π ( ·| s 1 )) ⊈ supp( β ( ·| s 1 )) . That is, ∃ ˜ a 1 such that π (˜ a 1 | s 1 ) > 0 and β (˜ a 1 | s 1 ) = 0 . Thus it holds that ϵ ( s 1 ) = ∑ a I [ β ( a | s 1 ) = 0] π ( a | s 1 ) > 0 . On the other hand, since s 1 ∈ D , there exists at least one action a 1 and one state s ′ 1 such that ( s 1 , a 1 , s ′ 1 ) ∈ D . Thus it holds that β ( a 1 | s 1 ) > 0 and M ( s ′ 1 | s 1 , a 1 ) > 0 . As a result,

In such case, ¯ R ( π in ) > ¯ R ( π ) . Therefore, if π is not constrained within the support of the behavior policy at some state s 1 ∈ D , we can find another policy π in that is constrained within the support of the behavior policy and achieves higher objective function ¯ R ( π in ) . Consequently, ¯ R ( π ) must achieve its maximum at support constrained policy π ∗ : supp( π ∗ ( ·| s )) ⊆ supp( β ( ·| s )) , ∀ s ∼ D .

Now we consider ¯ R 1 ( π ) .

With the same definition of ϵ ( s ) , n ( s ) and π in as in Eq. (66), Eq. (67) and Eq. (68), it also holds that

As before, when supposing π is not constrained within the support of the behavior policy at some state s 1 ∈ D , it holds that ¯ R 1 ( π in ) > ¯ R 1 ( π ) . Therefore, ¯ R 1 ( π ) must achieve its maximum at support constrained policy π ∗ 1 : supp( π ∗ 1 ( ·| s )) ⊆ supp( β ( ·| s )) , ∀ s ∼ D .

In conclusion, when the environment dynamics is stochastic, the maximizers of both ¯ R ( π ) and ¯ R 1 ( π ) are constrained within the support of the behavior policy:

C Further Discussions

C.1 Rationale for Choosing exp( αV ( s )) as the Empirical Normalizer

Firstly, choosing exp( αV ( s )) is intended to obtain something similar to the advantage function. With this normalizer, the weight of our regularizer is exp( α ( V ( s ′ ) -V ( s ))) , which is comparable to the weight exp( αA ( s, a )) in Advantage Weighted Regression (AWR) [47]. Here, V ( s ′ ) -V ( s ) represents the relative advantage of the next state s ′ compared to the current state s , while A ( s, a ) reflects the relative advantage of taking action a in s compared to following the current policy. Comparison of the objectives of SCAS and AWR:

Secondly, as discussed in the paper, introducing any normalizer that depends only on s (independent of s ′ ) does not affect the development and analysis of our method; it is merely for computational stability. In AWR-based methods, there also exists a normalizer Z ( s ) and they usually disregard it [47, 45]. The rationale behind this is similar.

C.2 Pessimism and Robustness in SCAS

In a specific sense, SCAS, which unifies OOD state correction and OOD action suppression, also integrates pessimism and state robustness. (1) Regarding pessimism: The OOD action suppression effect of SCAS aligns with the pessimism commonly discussed in offline RL work (being pessimistic about OOD actions) [31, 70, 30, 3, 54]. Unlike traditional policy constraint methods [69, 30, 10, 47], our approach does not require the training policy to align with the behavior policy; it only requires the successor states to be within the dataset support, which is a more relaxed constraint. (2) Regarding state robustness: The OOD state correction effect of SCAS is aimed at improving the agent's

robustness to OOD states during the test phase. Compared with previous works, SCAS unifies OOD state correction and OOD action suppression and additionally achieves value-aware OOD state correction. Some offline RL literature on state robustness differs from our approach; they typically consider noisy observations [72], such as sensor errors. In contrast, SCAS addresses state robustness concerning actual OOD states encountered during test time, rather than noisy observations.

C.3 Regularization Effect at ID States

In SCAS, there is regularization on the policy's output actions at ID states. In our regularizer, the perturbed states ˆ s are sampled from N ( s, σ 2 ) , and a large portion of ˆ s will fall near the original ID state s or even be approximately equal to s . Therefore, the policy's output actions at ID states are also regularized. For this part of the regularization, its role is equivalent to the ID state regularizer analyzed in Section 4, which has been theoretically shown to have the effect of suppressing OOD actions. Moreover, the experimental results in Section 6 also demonstrate that our OOD state correction regularizer addresses the traditional issue with OOD actions.

C.4 Differences between the OOD Action Issue and the OOD State Issue

We further elucidate the differences between the well-known OOD action issue and the OOD state issue we analyzed. Most offline RL works focus on the OOD action issue in the training phase. That is, the trained policy outputs OOD actions to compute the target Q, which results in extrapolation error and value divergence during training [12]. In contrast, the OOD state issue we defined and analyzed is in the test phase. That is, the agent can enter states out of the offline dataset during test, potentially resulting in catastrophic failure.

D Experimental Details

All hyperparameters of SCAS are included in Table 2. Note that we use this same set of hyperparameters to obtain all the results reported in this paper (except for parameter study). Following TD3+BC [10], we normalize the states in all datasets except for antmaze-large. We clip the exponentiated weight exp( αV θ ( s ′ ) -αV θ ( s )) in Eq. (19) to ( -∞ , 50] . Following the suggestions in the benchmark [9], we subtract 1 from the rewards for the Antmaze datasets.

Our evaluation criteria follow those used in most previous works. For the Gym locomotion tasks, we average returns over 10 evaluation trajectories and 5 random seeds, while for the Ant Maze tasks, we average over 100 evaluation trajectories and 5 random seeds. The reported results are the normalized scores, which are offered by the D4RL benchmark [9] to measure how the learned policy compared with random and expert policy:

The results of baselines reported in Table 1 are obtained as follows. We re-run OSR [22] on all datasets using their official codebase 5 and tune the hyperparameters for each dataset as specified in their paper. We implement SDC [75] and re-run it on all datasets. We use the SDC-related hyperparameters as specified in their paper, and sweep the CQL-related hyperparameters in {1,2,5,10,20} for each dataset. We re-run OneStep RL [5] on all datasets using their official codebase 6 and the default hyperparameters. We implement BC [48] based on the TD3+BC repository 7 and re-run it on all datasets. The results of other baselines are taken from [3] and [68]. The runtime in Table 1 is obtained by running offline RL algorithms on halfcheetah-medium-replay-v2 on a GeForce RTX 3090.

Figures 1(a) to 1(d) share the same embedding function obtained by running t-SNE on the set of all 200,000 samples (50,000 samples each from the dataset, CQL, TD3+BC, and SCAS). This ensures a clear visual comparison. Figure 1(d) contains all the 200,000 samples, which is the union of the points in Figures 1(a) to 1(c).

E Additional Experimental Results

E.1 Additional Value Estimation Results

Under the same setting of Figure 2, we conduct experiments on the additional datasets. The results are shown in Figure 5. We omit the Q values of Off-policy RL, SDC w/o CQL, and OSR w/o CQL at higher numbers of optimization steps, because these Q values diverge in the early learning stage, and plotting their Q values at later optimization steps would result in an excessive range on the vertical axis. The additional results also show that only SCAS's OOD state correction term can achieve OOD action suppression and prevent value over-estimation.

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

10

2

10

3

10

4

10

5

10

6

walker2d-medium-v2

SCAS: MC return

SCAS: Q

Off-policy RL: Q

SDC w/o CQL: Q

OSR w/o CQL: Q

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

10

2

10

3

10

4

10

5

10

6

walker2d-medium-expert-v2

E.2 Additional Results on OOD State Correction

To further examine the OOD state correction effects of SCAS, we conduct experiments on a modified D4RL maze2d-open-v0 [9]. It is a 2D point robot navigation task in a rectangle map with vertices (0 , 0) and (3 , 5) . The agent needs to reach the goal at (2 , 3) . We modify the dataset by removing all

the transitions containing states in a rectangle with vertices (0 , 0) and (1 . 5 , 2 . 5) . During test, we let the initial state be randomly distributed in this OOD region. We train algorithms over 10 6 gradient steps and average returns over 1000 evaluation trajectories.

The results of BC [48], TD3+BC [10], CQL [31], MOPO [73], IQL [29], and SCAS are reported in Table 3. With the OOD state correction signals, SCAS corrects the agent out of the OOD region more quickly and stably, achieving significantly better performance than typical offline RL methods.

E.3 Comparisons on the NeoRL Benchmark

We also evaluate SCAS on the NeoRL benchmark [49]. NeoRL is a benchmark designed to simulate real-world scenarios by collecting datasets using a more conservative policy, aligning closely with realistic data collection scenarios. The narrow and limited data makes it challenging for offline RL algorithms. The results are shown in Table 4. The results of baselines are taken directly from the MOBILE paper [59]. According to Appendix C in [59], these results are obtained by tuning hyperparameters per dataset. For SCAS, we use the same fixed set of hyperparameters as specified in Appendix D. Without additional hyperparameter tuning, SCAS still performs comparably to MOBILE and outperforms other baselines in total scores.

E.4 Comparisons with Additional Baselines

The original SCAS requires only one single hyperparameter configuration in implementations. For a fair comparison with DW [19], EDAC [2], RORL [72], SQL [71], and EQL [71], we roughly select λ from {0.025, 0.25} for each dataset, referring to this variant as SCAS-ht. The results of SCAS-ht and these methods are reported in Table 5. Among the ensemble-free methods, SCAS-ht achieves the highest performance in both mujoco locomotion and antmaze domains. Compared with ensemble-based methods, SCAS-ht also performs better on antmaze tasks. DW [19] reweights ID data points by their values for behavior regularization and does not account for OOD states during the test phase. In contrast, our approach considers an OOD state correction scenario, resulting in enhanced robustness during the test phase and better performance.

E.5 Results of Combining SCAS Regularizer into Various Offline RL Objectives

The SCAS regularizer is compatible with various offline RL objectives. We conduct experiments to combine SCAS with CQL [31], IQL [29], and TD3BC [10]. Comparisons between these combined

algorithms and the original ones are shown in Table 6. We find that applying the SCAS regularizer leads to improved performance for these popular algorithms, which could be attributed to the OOD state correction effects of SCAS. However, we also find that these combined methods do not achieve better performance than the original SCAS (comparable on most tasks and worse on some tasks). We hypothesize that this is because SCAS already has the effect of OOD action suppression, and when combined with offline RL objectives that also aim for OOD action suppression, it may become overly conservative. As a result, the combined algorithms may perform worse than the original SCAS on some sub-optimal datasets.

E.6 Additional Parameter Study Results

In this section, we present additional parameter study results conducted on four challenging Antmaze tasks, including antmaze-large-play-v2, antmaze-large-diverse-v2, antmaze-medium-play-v2, and antmaze-medium-diverse-v2.

Inverse Temperature α . The inverse temperature α is the key hyperparameter in SCAS for achieving value-aware OOD state correction. It controls the significance of the values of next states in SCAS's

OOD state correction. If α = 0 , the effect corresponds to vanilla OOD state correction. As α gets larger, SCAS is more inclined to correct the agent to the high-value ID states. Thus we can assess the effectiveness of value-aware OOD state correction compared to vanilla OOD state correction by varying α . Here we test SCAS with different α and the results are shown in Figure 6. We observe that a large α is crucial for achieving good performance on all the antmaze tasks, clearly demonstrating the effectiveness of our value-aware OOD state correction. However, too large α ( α = 10 ) induces less satisfying performance, probably due to the increased variance of the learning objective. In general, we find that choosing α = 5 leads to the best performance.

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

Normalized Return

antmaze-large-play-v2

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

antmaze-large-diverse-v2

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

=0

=0.5

=1

=5

=10

Balance Coefficient λ . The balance coefficient λ controls the balance between vanilla policy improvement and SCAS regularization. If we set λ = 0 , SCAS degenerates into the vanilla off-policy RL algorithm. Here we vary λ in { 0 , 0 . 25 , 0 . 5 , 0 . 75 , 1 } and present the corresponding learning curves of SCAS in Figure 7. Notably, SCAS is able to converge to good performance over a very wide range of λ . However, if λ = 0 , the vanilla off-policy RL suffers from extrapolation error and overestimation, demonstrating poor performance. We also observe a very interesting fact that even when λ = 1 and the signal from RL improvement (max Q) is removed, SCAS still performs well on most tasks. This could be attributed to the fact that value-aware OOD state correction implies some sort of improvement in policy by maximizing the values of policy-induced next states.

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

Normalized Return

antmaze-large-play-v2

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

antmaze-large-diverse-v2

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

=0

=0.25

=0.5

=0.75

=1

Noise Scale σ . The noise scale σ is the standard deviation of the Gaussian noise added to the original states for formulating the SCAS regularizer. Here we test SCAS with different σ and present the corresponding learning curves in Figure 8. We observe a significant performance drop with too large σ ( σ = 1 ) on all the tasks, due to the heavily corrupted learning signal. On the other hand, when σ = 0 (without noise), the performance is also less satisfying. With σ = 0 , SCAS is still able to prevent the agent at ID states from entering OOD states, maintaining the agent in safe regions, but it cannot correct the agent from OOD states to ID states as reliably as the original SCAS. In general, we find that choosing σ = 0 . 001 or 0 . 01 leads to the best performance.

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

Normalized Return

antmaze-large-play-v2

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

antmaze-large-diverse-v2

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

0

0.25

0.5

0.75

1

Gradient Steps

(×10 6 )

0

50

100

=0

=0.001

=0.01

=0.1

=1

10 1

10 2

10 3

10 4

10 5

Trained steps of dynamics

M

0.0

0.2

0.4

0.6

Dynamics Loss

M

Loss

0

25

50

75

Normalized Return

walker2d-medium-v2

Return

10 1

10 2

10 3

10 4

10 5

Trained steps of dynamics

M

0.0

0.2

0.4

0.6

Dynamics Loss

M

Loss

0

50

100

Normalized Return

walker2d-medium-expert-v2

Return

10 1

10 2

10 3

10 4

10 5

Trained steps of dynamics

M

0.0

0.2

0.4

Dynamics Loss

M

Loss

0

50

100

Normalized Return

hopper-medium-v2

Return

10 1

10 2

10 3

10 4

10 5

Trained steps of dynamics

M

0.0

0.2

0.4

Dynamics Loss

M

Loss

0

50

100

Normalized Return

hopper-medium-expert-v2

Return

E.7 Sensitivity Analysis on Dynamics Model Errors

To empirically investigate SCAS under different dynamics model errors, we run SCAS using different checkpoints of the trained dynamics model, which are obtained at different steps in the dynamics model training process. The model error is controlled by the number of trained steps. The results are shown in Figure 9. The figure plots the training loss of the dynamics model M ω and the corresponding normalized return of SCAS over 5 random seeds. We observe that the performance of SCAS increases with the number of trained steps of the dynamics model (i.e. the accuracy of the model) and stabilizes at a high level.

E.8 Learning Curves of SCAS

Learning curves on Gym locomotion tasks and Antmaze tasks are presented in Figure 10 and Figure 11 respectively. The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.

F Broader Impact

Offline RL holds promise for facilitating practical RL applications in domains like robotics, healthcare, and education, where data collection is often costly or risky. However, it is important to recognize its potential negative societal impacts. One concern is that biases in offline data may transfer to the learned policy. In addition, offline RL may affect employment by automating tasks traditionally

performed by humans, like factory automation or autonomous driving. Addressing these challenges will contribute to the responsible development and deployment of offline RL algorithms.

From an academic standpoint, this research systematically analyze the OOD state issue in offline RL and propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression. This work potentially offers researchers a new perspective on analyzing the OOD state issue and enhancing test-time robustness in offline RL. Besides, SCAS also holds the promise to be extended to safe RL [1, 17, 13], meta RL [8, 65, 66, 64, 4], and multi-agent RL [35, 52, 55, 51, 16].

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

25

50

halfcheetah-random-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

hopper-medium-expert-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

hopper-medium-replay-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

hopper-medium-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

25

50

hopper-random-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

walker2d-medium-expert-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

walker2d-medium-replay-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

walker2d-medium-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

25

50

walker2d-random-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

antmaze-umaze-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

antmaze-umaze-diverse-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

antmaze-large-play-v2

0.00

0.25

0.50

0.75

1.00

Gradient Steps

(×10 6 )

0

50

100

antmaze-large-diverse-v2

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to the code in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results in the paper are accompanied by standard deviations across multiple seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to Appendix F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code is well documented and anonymized.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe issue of OOD actions is a well-known concern in offline RL research. The issue of OOD states, however, is relatively unexplored. In this paper, the authors shed light on the significance of OOD state correction and propose a SCAS algorithm to guide the agent back into high-value ID states when encountering an OOD state. To achieve this goal, SCAS first defines a conditional state transition probability $N^*(s'\\mid s)$ skewed towards high-valued states and regularizes the policy by aligning the dynamics induced by the policy on a perturbed state $\\hat{s}\\sim \\mathcal{N}(s, \\sigma^2I)$ with the value aware state transition probability $N^*(\\cdot \\mid s)$. Also, theoretical analysis shows that SCAS implicitly guarantees the correction of OOD actions, thereby removing the need for additional regularizers.\n\nSTRENGTHS:\nThe authors highlighted the significance of OOD state correction in offline RL, a problem other researchers often neglect. Also, multiple experiments were conducted to show the competence of SCAS and its robustness towards the choice of hyperparameters.\n\nWEAKNESSES:\n1. The authors did not cite works from the DICE literature(e.g., Nachum et al., 2019; Lee et al., 2021; Mao et al., 2024). Although they do not explicitly correct OOD states, I believe they are deeply related to SCAS because they aim to align the policy's state distribution $d^\\pi$ with the dataset's state distribution, thus preventing the policy from visiting OOD states.\n\n2. The choice of $\\exp(\\alpha V(s))$ as the empirical normalizer seems extremely arbitrary. Providing some rationale behind this choice is recommended.\n\n3. All of the experimental analysis was performed on MuJoCo environments, which have deterministic transition dynamics. Since SCAS utilizes a deterministic environmental model, it may not perform well in environments with stochastic transition dynamics.\n\n4. NeoRL is a relatively uncommon benchmark compared to D4RL. A brief explanation of NeoRL would be helpful for the readers.\n\n5. Some experimental details are missing from the paper.\n\n    * Do Figures 1a to 1d share the same embedding function? How was this embedding function obtained? Was it by running t-SNE on the set of 200,000 samples(50,000 samples each from the dataset, CQL, TD3+BC, and SCAS)?\n    * The points in Figure 1d seem to be the union of the points in Figures 1a, 1b, and 1c. Am I correct?\n    * Why does Figure 2 omit Q values of off-policy RL, SDC, and OSR for higher numbers of optimization steps?\n    * Section 6.3 states that varying steps of Gaussian noise were added to the actions during test time. Figure 3 indicates that the authors added noise up to 40 steps. However, trajectories are usually longer than 40 steps. How were those 40 steps selected? Did the authors add noise to the first 40 interactions with the environment? \n\n6. Minor comment: The meaning of the term *$\\mathcal{D}(s)$ weights* on Line 162 is unclear.\n\n\n### References\n\nLee, Jongmin, et al. \"Optidice: Offline policy optimization via stationary distribution correction estimation.\" International Conference on Machine Learning. PMLR, 2021.\n\nMao, Liyuan, et al. \"Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradient update.\" arXiv preprint arXiv:2402.00348 (2024).\n\nNachum, Ofir, et al. \"Algaedice: Policy gradient from arbitrary experience.\" arXiv preprint arXiv:1912.02074 (2019).\n\nQUESTIONS:\nPlease refer to the **Weaknesses** section. As a side note, have you tried using the Q and V functions learned by IQL (or any other value-based offline RL algorithm) instead of learning them together with the policy? From Figure 9, the training process seems a bit unstable, so using a stationary training objective might help.\n\nLIMITATIONS:\nThe authors have adequately addressed their work's limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SCAS (OOD State Correction and Action Supression), a model-based regularization approach that effectively addresses the challenges of out-of-distribution (OOD) states and actions in offline reinforcement learning (RL) algorithms.\nThe method unfolds in two main stages: (1) training a transition model, and (2) training a policy that incorporates a model-based regularizer. \nThis regularizer is designed to steer the policy towards high-value in-distribution (ID) successor states and thereby away from OOD successor states.\nWhile the primary focus of SCAS is to resolve OOD state issues, this paper shows that the regularization strategy not only mitigates visiting OOD states but, as a byproduct, also suppresses OOD actions in the training of the value function.\nEmpirical results demonstrate that the proposed method performs comparably on standard benchmarks, showing its effectiveness in refining offline RL algorithms.\n\nSTRENGTHS:\n**1. Theoretical connection between OOD state correction and OOD action suppression**\n\nA notable strength of this paper is the establishment of a theoretical link between OOD state correction and OOD action suppression. The authors articulate how the mechanism designed to correct OOD states implicitly suppresses OOD actions, by showing an optimal policy of the regularized objective will produce actions inside of the support of behavior policy for every dataset state. This implicit suppression also explains successful empirical results on offline RL benchmarks.\n\n**2. Robustness to Hyperparameter Selection**\n\nAs detailed in Section 6.2 and illustrated in Figure 4, the proposed approach demonstrates considerable robustness to variations in hyperparameter settings. \nThis attribute is particularly valuable in practical applications of offline RL where optimal hyperparameter settings can be elusive or computationally expensive to determine.\n\n**3. Evaluation on standard benchmarks and implementation code is attached for reproduction.**\n\nThe paper's evaluation methodology is another major strength. The authors have rigorously tested their approach on standard benchmarks, D4RL and NeoRL, providing a comprehensive assessment of its performance relative to existing methods. Moreover, the inclusion of implementation code enhances the paper’s contribution by facilitating reproducibility and further experimentation.\n\nWEAKNESSES:\n**1. Unclear Motivation for Value-Aware OOD State Correction**\n\nThis paper proposes shifting the OOD state distribution not to a standard ID state distribution, but to a high-value ID state distribution instead. This choice raises questions about the specific objectives of value-aware state correction. Is it strategically targeting high-value data points, or is it intended to mitigate the effects of distribution shifts? It seems that the manuscript primarily focuses on the former, in that case, it would benefit from a comparison with existing works like DW[1], which also focus on high-value data points for behavior regularization. If the goal is indeed to mitigate the effects of distribution shifts, then it is crucial for the manuscript to better articulate and emphasize how value-aware OOD state correction contributes to the robustness against OOD state visitation. Moreover, a clearer explanation of how this approach aligns with offline RL principles like pessimism and robustness to OOD states would greatly strengthen the manuscript's validity and impact.\n\n**2. Lack of Comparisons with Comparable Offline RL Approaches**\n\nWhile the paper tackles a fundamental problem in offline RL, the evaluations presented are somewhat constrained by a lack of comparison with key recent advancements in the field. Notable works such as DW[1], EDAC[2], RORL[3], and SQL/EQL[4], which explore similar offline RL settings, are not considered in the main experiments. This gap is particularly significant as DW[1] employs a value-aware strategy that closely mirrors the approach of this paper, focusing on constraining policies to high-value data points. To convincingly justify the need for correcting OOD states—or for employing a value-aware correction approach—a comparative analysis with these influential studies is essential. This would not only place the findings in a broader context but also potentially highlight the contributions and advantages of the proposed method.\n\n[1] Hong et al., “Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets.”, NeurIPS 2023.\n\n[2] An et al., \"Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble.\", NeurIPS 2021.\n\n[3] Yang et al., “RORL: Robust Offline Reinforcement Learning via Conservative Smoothing.”, NeurIPS 2022.\n\n[4] Xu et al., “Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization.”, ICLR 2023.\n\nQUESTIONS:\nQ1. How sensitive is the performance to the accuracy of the model?\n\nQ2. Given that a main contribution of this work is a model-based regularizer, it appears that SCAS could be compatible with various offline RL methods. Is it feasible to apply the regularizer across different offline RL objectives, such as CQL, IQL, and others?\n\nQ3. In this study, the trained model is exclusively utilized as a regularizer during policy training. Could this approach be extended to also incorporate the model during testing time?\n\nLIMITATIONS:\nThe authors included their limitations in Section 7 and limitations are addressed appropriately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper focuses on OOD state issue, an important but overlooked issue in offline RL. This paper proposes aligning the OOD state towards In-Distribution (ID) states with high value, named as value-aware OOD state correction. Additionally, the paper discovers that the overestimation of OOD actions can also be mitigated during the implementation of OOD state correction. The experimental results demonstrate that the proposed algorithm is superior in performance and robustness.

STRENGTHS:
1. The paper presents an intuitive solution to the OOD state iusse, while also mitigating the overestimation of OOD actions. 

2. The algorithm is highly robust, requiring only a single set of hyperparameters across different environments. This robustness is particularly important in the context of offline reinforcement learning, as online interaction for tuning parameters is costly or even dangerous.

WEAKNESSES:
1. The paper proposes that during the process of OOD state correction, the overestimation of OOD actions will also be addressed. I have some doubts about this point. The constraint $R_2$ seems to only regularize the policy to output actions at OOD states that result in the next state being within the offline dataset. However, there is no regularization on the policy's output actions at ID states. Therefore, I am confused about how the paper solves the traditional issue with OOD actions. If I have misunderstood anywhere, please point it out.

QUESTIONS:
1. Figure 2 suffers from an excessive range on the vertical axis, causing the two curves of SCAS to almost overlap. A suggestion would be to take the log of the Q values on the vertical axis; this should make the two curves more distinguishable.

LIMITATIONS:
Please see weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a regularization term in offline RL, which can simultaneously address OOD state correction and OOD action suppression without pretraining a state transition model $N(s'|s)$. It shows good performance in D4RL benchmarks.\n\nSTRENGTHS:\nI found it an interesting topic to consider the OOD state issue. Most existing offline RL methods focus on addressing the OOD action issue, while this paper highlights that OOD states may also cause training collapse. Another good property of the method is that it only needs one single hyperparameter configuration to achieve good empirical performance. The theory is also solid.\n\nWEAKNESSES:\nThe method utilizes a deterministic policy, which is often regarded as lacking expressiveness. Thus, the performance is not as good as diffusion-based policy methods.\n\nQUESTIONS:\n1. For Definition 1, if $d^\\pi_{M_\\tau}$ is a continuous probability density function of $s$, and if $d_D(s)$ only has support on $s$ which appears in the dataset, does it mean that any state which doesn't show up in the dataset is OOD? If that's the case, the probability of an OOD state is 1 since the number of ID states must be finite in the dataset?\n\n2. Instead of pretraining $N(s'|s)$, the paper chooses to add Gaussian noise to $s$, with the noise $\\sigma=0.003$. Is this hyperparameter important? If $\\sigma$ increases, can we have a more robust algorithm since it makes more OOD states into consideration?\n\n3. The method uses a deterministic policy. What if we choose a Gaussian policy?\n\n4. In Figure 2, why do Off-policy, SDC, and OSR only have one dot? Shouldn't it also be a line?\n\n5. How are the results in Table 1 recorded? Final round results, last 5 epoch average, or another method?\n\n6. Where do the baseline method results come from, their original paper? I ask because I found SDC doesn't have results for antmaze tasks. Did the authors implement it by themselves?\n\n7. The authors use antmaze-v2 for their method evaluation, while some baseline methods, for example, IQL and OSR, use antmaze-v0. In my experience, the v2 dataset can always provide higher rewards than v0. Can the authors explain the mismatch? Or can the authors show results on v0 as well?\n\nLIMITATIONS:\nI believe the theory of the paper is solid, while the experimental performance is my concern. I am happy to increase my score if the antmaze version mismatch issue is resolved.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}