Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
Authors: Qin-Wen Luo
Ming-Kun Xie
Ye-Wen Wang
Sheng-Jun Huang
Venue: ICML 2025 poster (2025)
Keywords: offline reinforcement learning
selective state-adaptive regularization
offline-to-online reinforcement learning
Primary area: reinforcement_learning->batchoffline

Abstract: Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset.
To alleviate extrapolation errors, existing studies often uniformly regularize the value function or policy updates across all states.
However, due to substantial variations in data quality, the fixed regularization strength often leads to a dilemma: 
Weak regularization strength fails to address extrapolation errors and value overestimation, while strong regularization strength shifts policy learning toward behavior cloning, impeding potential performance enabled by Bellman updates.
To address this issue, we propose the selective state-adaptive regularization method for offline RL. Specifically, we introduce state-adaptive regularization coefficients to trust state-level Bellman-driven results, while selectively applying regularization on high-quality actions, aiming to avoid performance degradation caused by tight constraints on low-quality actions.
By establishing a connection between the representative value regularization method, CQL, and explicit policy constraint methods, we effectively extend selective state-adaptive regularization to these two mainstream offline RL approaches.
Extensive experiments demonstrate that the proposed method significantly outperforms the state-of-the-art approaches in both offline and offline-to-online settings on the D4RL benchmark. The implementation is available at https://github.com/QinwenLuo/SSAR.

Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
Qin-Wen Luo" 1
Ming-Kun Xie
Ye-Wen Wang' Sheng-Jun Huang'
Abstract
et al., 2023; Haarnoja et al., 2024). However, RL typically improves performance through trial and error, which lim- its its practical application to critical scenarios, such as healthcare decision-making (Tang & Wiens, 2021; Fatemi et al., 2022) and autonomous driving (Fang et al., 2022; Diehl et al., 2023). These tasks often have such strict re- quirements regarding potential risks and time costs that the interaction with the environment is limited or inaccessible.
Offline reinforcement learning (RL) aims to learn effective policy from a static dataset. To alle- viate extrapolation errors, existing studies often uniformly regularize the value function policy updates across all states. However, due to substan- tial variations in data quality, the fixed regulariza- tion strength often leads to a dilemma: Weak reg- ularization strength fails to address extrapolation errors and value overestimation, while strong reg- ularization strength shifts policy learning toward behavior cloning, impeding potential performance enabled by Bellman updates. To address this issue, we propose the selective state-adaptive regulariza- tion method for offline RL. Specifically, we intro- duce state-adaptive regularization coefficients to trust state-level Bellman-drin -riven results, while se- lectively applying regularization high-quality actions, aiming to avoid performance degrada- tion caused by tight constraints on low-quality actions. By establishing connection between the representative value regularization method, CQL,, and explicit policy constraint methods, we effectively extend selective state-adaptive regular- ization to these two mainstream offline RL ap- proaches. Extensive experiments demonstrate that the proposed method significantly outper- forms the state-of-the-art approaches in both of- fline and offline-to-onlin settings on the D4RL benchmark. The implementation available at
Offline RL has mitigated this problem by deriving policies from static datasets without actual environment interactions. Direct application of the off-policy algorithms could result in value overestimation and extrapolation errors (Fujimoto et al., 2019; Kumar et al., 2019). To address these chal- lenges, based on how they resolved around the policy it- eration process, existing RL-basedd offline methods can be roughly divided into two groups, value regularization (Ku- mar et al., 2020; Cheng et al., 2022; Nakamoto et al., 2024) and policy constraint (Fujimoto & Gu, 2021; Kostrikov et al., 2021b; Nair et al., 2020). The former suppresses the Q-values of -idtiiibbttiino (OOD) actions in the policy evaluation function alleviate value overestimation; while the latter restricts updates near the dataset in the policy im- provement function avoid OOD actions. Both approaches incorporate regularization terms into their online counter- parts, shifting the update from an optimistic to a pessimistic manner. Despite their demonstrated advances, existing meth- ods still face the critical challenge of identifying optimal regularization strengths that balance trust in the outcomes derived from Bellman updates and the imitation of dataset actions to fully unlock the potential of RL-based learning.
Given that in model-free RL methods, performance gains beyond the behavior policy are fundamentally enabled by Bellman updates, is crucial to balance confidence in these updates and conservative regularization. Since the balance is typically achieved by the fixed global coefficient in the reg- ularization term, identifying its optimal value is paramount for effective performance. This presents challenges from three key aspects. Firstly, from a lesk- perspective, the optimal global coefficient can vary significantly across dif- ferent tasks. Certain tasks may benefit from a larger coeffi- cient to enforce stricter regularization and avoid overestima- tion, while others may require a lower coefficient to enable more flexible policy updates, resulting in better performance
1. Introduction
Reinforcement Learning (RL) has made remarkable strides across diverse domains (Degrave et al., 2022; Kaufmann
'Equal contribution 'Najjing University of Aeronautics and Astronautics, Nanjing, China. Correspondence to: Sheng-Jun Huang <huangsj@ nuaa.cdu.en.>
Proceedings of the 4 International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offliee RL
(Nakamoto et al., 2024; Lyu al., 2022). Secondly, from the perspective of training dynamics, the optimal coeffi- cient should evolve throughout the training process. When the policy is less reliable and poorly constrained by the dataset at the early stage, a larger coefficient is essential to restrict policy updates. As the training advances and the distribution of the learned policy becomes sufficiently close to the dataset, a lower coefficient may be more ben- eficial for states where the policy actions are close to the dataset, thanks to the reliable and reasonable generalization enabled by Bellman updates (Beeson & Montana, 2022). Thirdly, considering the variability in data densities, it is intuitive to trust the Q-values learned by Bellman updates for states with high data density, while conversely tighten- ing the constraints for states with low data density. Despite some nuances, existing methods predominantly determine the global coefficient through extensive experiments, which struggles to address these challenges effectively.
ficient 020 RL through simple linear coefficient annealing.
2. Preliminaries
Offline RL The environment in RL is typically modeled as a Markov Decision Process (MDP), which is defined by the tuple 7. (Sutton & Barto, 2018), where S is the state space, A is the action space. P A + A(S) is the transition function, R S A + R is the re- ward function, 1l is the initial state distribution and y is discount factor. The goal of RL is to find policy S + A(A) that maximizes the expected discounted return: E Boo-m...-<m((a)).. ( ()) 12 R at)] For any policy n, we define the state value function as (*)))= = E EE 7 R(s,, |s) 1o = s] and the state-action value function as (s)
((((s))a))ss) a;)]s0 = 8, 0,00 r(-]su)]. The agent interacts with the environment by ob- serving states, taking actions and receiving rewards, and improves the policy through the interactive data.
Ex [2 R'
Another critical issue with the fixed global regularization lies in its impact on the efficiency of transitioning from offline-to-online (020) RI. The uniform regularization strength presents a dilemma: low regularization coefficients may amplify the risks of extrapolation errors through in- sufficient behavioral constraints, yielding poorly initialized policies. Conversely, overly high regularization coefficients induce significant discrepancies between the offline and on- line Q-values in value regularization methods, as well as misalignment between the learned policy and the policy directly induced by the critic in policy constraint methods. To enhance the efficiency of O20 RI., researchers have plored various approaches, including gradually relaxing con- straint terms (Beeson& & Montana, 2022: Zhao et al., 2022) and adopting alternative action sampling strategies (Zhang et al., 2023; Uchendu et al., 2023). However, these meth- ods make modifications based on global coefficients, which prevents further improvement in fine-tuning efficiency.
In offline RL, the agent has no access to the environment but only to a fixed dataset D, which is typically assumed to be collected by behavior policy xg. Existing works follow the key insight of maintaining pessimism in policy learning, which constrains the learned policy close to the dataset. For the value function and the policy, which are the two crucial components of model-free RL, value regularization and policy constraint are commonly applied in pursuit of this goal.
Value regularization In the policy iteration, OOD actions are inevitably taken to compute the Bellman targets for the Q function update. This induces significant extrapolation error and overestimation of OOD actions (Kumar al., 2019; Fujimoto et al., 2019), ultimately resulting in poor or even negative performance improvement. Some work combats this problem by applying value regularization to the Q function update to suppress the overestimation of OOD actions (Cheng et al., 2022; Kumar et al., 2020; Nakamoto et al., 2024). A representative algorithm is conservative Q-learning (CQL) (Kumar et al., 2020), which learns conservative Q function such that the expected values of a policy under this Q function approximate the lower bound of its true values. The crucial modification is the value regularization applied to the Q function update
In this work, we propose state-adaptive regularization that dynamically quantifies the reliability of Bellman updates, guiding the policy to trust optimistic outcomes at the state level. Instead of a fixed coefficient, our method employs a neural network as a state- dete-deeendent coefficient generator and adaptively updates it to adjust regularization strengths based on the discrepancy between the dataset and the dis- tribution of the learned policy. To enhance its universality, establish connection between CQL, a representative value regularization method, and explicit policy constraint methods, extending it to both popular offline approaches. Furthermore, we propose a selective regularization method by selecting a subset of data with high rewards. This allows the policy update to focus on high-quality actions and avoid excessive constraints on low-quality parts. With the integra- tion of selective state-adaptive regularization, we effectively bridge the gap between offline and online RL, achieving ef-
min B EEND log Σ exp (Q(s[((sss - EL-D [Q(s,))] aa)
Q
1 + 1 1 Readdw 2
((((=0))==*(00000))) ())))())))) 27
(1)
The second term is the empirical Bellman error objective used to update the Q function in RI., where the empiri-
2

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offline RL
cal Bellman operator is computed as BTx = r(s, a) (-1e)) (s', ,.)]. The first term is a value regular- izer, aiming to bound the Q values to avoid overestimation of OOD actions, and 3 is tradeoff factor controlling the intensity of pessimism. Given that the regularization only affects the update of the Q function, the policy update fol- lows the form of online RL. Akin to the implementation of SAC (Haarnoja et al., 2018), the policy of CQL is modeled as Gaussian distribution but updates by approximating the Boltzmann distribution of Q values with the following loss:
3.1. State-Adaptive Coefficients
To address the challenges outlined in Section 1, we pro- pose to use state-adaptive coefficients instead of a fixed global coefficient, which allows for adaptively controlling the strength of regularization at the state level. This mod- ification exhibits two advantages. On the one hand, the adaptive update manner performs automatic adjustments to accommodate differences between tasks and dynamics across training stages; on the other hand, the state- adae-ie nature enables to determine the proper coefficients by leveraging Bellman-drive results, which are closely related to data density information.
min BawDDann (-s)) la log ग (als) - Q(s,a))
(2)
Explicit policy constraint Some other works focus on directly constraining the policy update without the change of Q function update. TD3+BC (Fujimoto & Gu, 2021) is a minimalist but efficient approach by only adding a regularization term of the MSE loss between the actions output by the deterministic policy and the actions in the dataset. With the same Q update function as TD3 (Fujimoto et al., 2018), the update loss functions are defined as
Considering that different offline methods have different constraint objectives, it is challenging to directly apply state- adaptive coefficients to these methods. Recalling that the core principle behind these algorithms is to learn a desir- able policy within reliable region near the dataset, we provide a unified framework to integrate different constraint objectives.
We begin with a classic offline RL algorithm CQL (Kumar et al., 2020), which is also a representative of value regular- ization methods, and bridge its relationship with the explicit policy constraint methods.
min E. Ea - lIQ (r(s, + 2 (5', Tx (s)))) Q
(3)
max Es (AQ (AQ(ss (s)) - (s)) - =
(4)
Proposition 3.1. With the policy ा modeled as a Boltz- mann distribution, e.g. T(ls) x exp (Q(s, a)), the reg- ularization term of Eq. (1) is equivalent to the negative bog-likelihood term about x at the dataset actions, that is
where x = / E Eis,a,) ((s;;::) is an adaptive scalar that controls the intensity of the regularizer and normalizes the loss term about Q values.
min / E log Σ exp (Q(s, - EAND [Q(s,a) a)] Q
3. Selective State-Adaptive Regularization
In this section, to adaptively adjust the regularization strength across different states and maximize the potential benefits of Bellman updates, we propose the Selective State- Adaptive Regularization method. This approach com- prises three key components: In Section 3.1, we establish the equivalence between Conservative Q-Learning (CQL) and explicit policy constraint methods. Building upon this relationship, we introduce learnable state- adaptive regu- larization mechanism that dynamically adjusts hate-wise regularization coefficients based on the divergence between the log-likelihood of dataset actions under the learned pol- icy and a given threshold. In Section 3.2, we introduce a distribution- strategy based on the learned policy to automatically determine the appropriate threshold for this mechanism. To further ensure the validity of the constraints and fully exploit the advantages of the RL paradigm, we propose selective regularization strategy in Section 3.3.
$
min 3 B [ea)D log (alss)
Proposition 3.1 shows that the equivalent regularization term can be considered as an explicit policy constraint, similar to SAC+ML used in (Yu & Zhang, 2023). This observation tells us that the coefficients in both approaches directly af- feet the probabilities of dataset actions in the learned policy, This motivates us to use a unified framework to adjust the coefficients for these two types of methods.
It becomes intuitive to relax the constraints when the policy actions are sufficiently close to the dataset, while enhanc- ing them in regions of higher uncertainty. Building on this intuition, given the formulations of policy updates in Eq. (4) and Proposition 1. we propose a simple yet effective method to derive state-adaptive coefficients by constrain- ing action probabilities to exceed the state-level thresholds determined by the policy distribution. Specifically, for stochastic policy, we define the objective as
Additionally, in Section 3.4, we extend our method to deter- ministic policy algorithms that lack explicit policy distribu- tions. Finally, Section 3.5 presents our online fine-tuning method, highlighting its simplicity and minimal reliance on the offline data.
(+) = Be [log (als) - Ca(s)) (o(s) (5)
3

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offliee RL
where
hopper-medium-exper.
2.0
(n(s) min ( nols), log (u nols)}
(6)
Uniform Regularization
1.5-
Selective Regularization
where I represents the mean of the learned policy, and o denotes its standard deviation. (s) state-adaptive coef- ficients modeled by neural network, and is parameter controlling the width of the trust region. Considering that in some implementations of CQL, the policy is modeled as squashed Gaussian distribution, we use the min operator to capture the appropriate thresholds.
Values
1.0
0.5
0.0
With the proposed method, the state-adaptive coefficients will decrease when the probabilities of the dataset actions exceed the thresholds, and conversely, increase when they fall below them. This ensures that the learned policy not only restricts the updates within a reliable region but also obtains the potential performance improvements enabled by the generalization capabilities of Bellman updates.
0.5
Average Log
Averago Log
Normalized Score
Likelihood on D Likelihood on D-
Metries
Figure I. Comparison of Uniform vs. Selective Regularization. The values are evaluated by the policies trained by with diffcrent regularization on all data in the datasct.
By incorporating state-adaptive coefficients, we define the regularization term in CQL
between the policy distribution and the dataset can reflect the dynamic of the training progress, we propose to per- form adjustment of in distribution-awar manner to dynamically expand the trust region.
min (s) (log Σ exp(Q(s, - (1Q(s, Q
+ E. Be. 8 {
We start by setting the initial value of n to a low value Nstarts ensuring sufficient constraint, and gradually increasing n un- til it reaches nend- Throughout the training process, a simple linear schedule for is applied to relax the constraint:
(7)
which allows for flexible trade-off between optimism and pessimism at the state level.
n1 = n + An
(8)
3.2. Distribution-Aware Thresholds
where An = (Nend Netart) Tine/TT /T, Time represents the update interval and represents the total steps.
An important problem is how large the reliable region should be to impose an effective regularization. It is noteworthy that the size of reliable region controlled by typically reflects the confidence of the learned policy based on Bell- man updates. Due to the variability in task complexity. it is inherently challenging to determine a proper n. across different tasks. For instance, in the dataset with low data coverage, larger value might be necessary to account for the increased uncertainty, whereas in well-sampled tasks, lower one avoid unnecessary constraints. Using task- agnostic approach for determining is not good choice and often yields unfavorable performance. This motivates the need for an adaptive mechanism that performs dynami- cal adjustments based on the characteristics of the task and the underlying policy distribution.
By using this approach, the threshold n is initially set to small value, causing the loss in Eq. (5) to be negative. As result, the coefficients of most states will increase during the early stages to enforce the policy distribution towards the dataset. As training progresses, the learned policy gradually approximates the dataset actions across most states. With the expansion of the trust region, the constraints on certain states are gradually relaxed, allowing for broader exploration. To reach balance in the strength of the constraint, we terminate the update once the condi- tion ((s.a)=D [log (a|s) C, (s)] > 0 is satisfied. This adaptive scheme ensures that the trust region evolves in alignment with the distribution of the policy.
Intuitively, the parameter n should be gradually increased throughout the training process. In the early stage, the policy distribution deviates significantly from the dataset, making it suffer from extrapolation errors. Once the policy is well- learned and sufficiently bounded around the dataset, looser constraints can allow the learning process to potentially benefit from Bellman updates. Given that the divergence
3.3. Selective Regularization
Although we can adaptively control the strength of regular- ization with state-adaptive coefficients, it has still proven to be difficult to obtain favorable performance on -ualliyyyy datasets. As shown in Figure I, this is because the proposed coefficients updating mechanism encourages the policy to

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offliee RL
Sub-Dataset
Offline Dataset
walker2d-medium--2
walker22-mediun--eelayyy2
walker22--ediium-rplaa
(a) Data with high returns in the dataset (h) Data with high returns in the dataset (c) Data with positive advantages in the with high quality variance
with a low quality variance
dataset with high quality variance
Figure 2. t-sne visualization of different sub-dataset selection methods
output high probabilities for the actions within the dataset, including those that lead to suboptimal or even poor per- formance. To address this problem, we propose a selective regularization strategy to impose the constraint on a sub- set of high-quality actions, which prevents the policy from being misguided by low-quality actions. To achieve this, prioritize trajectories with high returns, emphasizing the regularization on valuable actions, while overlooking other constraints with minimal returns.
For the datasets composed of heterogeneous data with a wide-ranging distribution, such as walker2ddmmeiiu--rrpple v2, which records all samples the replay buffer during training, the sub-dataset can only cover a portion of the entire distribution, as shown in Figure 2(b). In such situ- ation, solely adjusting constraints the states within the sub-dataset may cause catastrophic overestimation for the remaining states.
To address this issue, we utilize the approaches from IQL Kostrikov et al. (2021b) to pre-train a state-action value net- work Q and state value network V to capture the relative value of the data. As a representative offline RL algorithm, IQL has demonstrated the superiority of advantage- -eeighee behavior cloning, where Q-values and V-values are learned through expectile regression in a non-iterative manner with in-sample learning
we construct a sub-dataset denoted by ò by selecting tra- jectories with returns greater than a selective threshold Gr- To mitigate the negative impact of low- low-quality actions, we update the state-adaptive coefficients only within Î as
(Ø( = E ((s,a)> 7 (als) C.. (*)
where C, (8) is still computed as Eq. (6).
When the sub-dataset ò can cover the most region of the whole dataset D, we perform regularization selectively on the actions with the sub-dataset. For CQL, the objective can be reformulated as
Ly ((s.e) (3,,),) V(s))]
(11)
LQ B(sa,,))-l(rrsss.0 + 7V'ss) Q(s,a)))
where Lī(x) = [[x < 0)]2² and I[-) is the indicator
function.
min (x)) Q
& flgg Σ exp(Q(s, a)) - En-D|Q(s, a)]]
With the pre-trained Q and V, we can filter the valuable actions on the whole dataset by the condition Q(s, a) V(s) > O. Analogous to Eq. (9) and Eq. (10). we can construct the sub-dataset Á by the data that satisfies the metric and favor these selected actions. Given that this approach guarantees that at least one action is constrained for all states, the sub-dataset can capture the approximate distribution of the offline dataset, as shown in Figure 2(c).
a
+ E. Eea. $=*
(10)
Generally, there exist two situations for different datasets. For datasets with low quality variances, the sub-dataset con-
sisting of high-return trajectories can effectively capture the distribution of the offline dataset, as shown in Figure 2(a). This motivates to use selective regularization for increas- ing the Q-values of valuable actions in the dataset while allowing the Q-values of low-quality actions to update natu- rally through the Bellman backup. This mitigates the over- estimation of Q values by leveraging the generalization of similar actions or states to constrain the policy.
3.4. Extend to Explicit Policy Constraint Methods
Based on the relationship between the value regularization and the explicit policy constraint methods as shown in Propo- sition 3.1, the idea of the selective regularization method can also be applied to explicit policy constraint methods, In this
5

---

Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offliee RL
section, we provide an extension of the proposed method to the representative algorithm TD3+BC. However, it is infea- sible to directly update the state-adaptive coefficients using Eq. (9). as TD3+BC models the policy as deterministic form. Recalling that the interaction process in TD3, we find that the exploration noise behaves similarly to the standard deviation, allowing us to treat the actions as samples from a Gaussian distribution with the mean of the policy actions and fixed standard deviation. This modification enables to rewrite Eq. (9) as
directly applied during the online fine-tuning stage. Two factors ensure stable performance improvements: i) there are far fewer out-of-distribution (OOD) actions during the interaction phase, and ii) the coefficient network can gener- alize its adaptability to new states.
Thanks to the advantages of state-adaptive coefficients, we can efficiently implement online fine-tuning by fixing the parameters of the coefficient network, with linear annealing applied to the outputs as
N Newd
(Ø) = E
- (a (a- (s))(s))))88(())
(12)
Bon (8) = min
0} (())
(15)
where d is the exploration noise in TD3, which is usually set as 0.1. Note that Eq. (9) and Eq. (12) are grounded in the same underlying mechanism, with the primary difference lying in the specific policy formulations to which they are applied. See Appendix C for the derivation.
where N is the number of online interaction steps and Nend is the given number of decay steps.
During online fine-tuning, the update functions of the actor and the critic hold on and (sn(s) is used to replace the regularization coefficients.
By incorporating the state-adaptive coefficients, we define the objective of the policy improvement in TBC+BC as
Although the form of method is similar unified ap- proaches across 020 phases, such as IQL, our method offers advantages due to the well-trained coefficient network and its generalization. As a result, we can reserve only a subset of trajectories for initializing the online replay buffer, or even discard the offline dataset entirely and use only the online data to update the policy. This is infeasible for the unified methods, as they require the offline dataset to con- strain the policy update, thereby hindering the utilization efficiency of online data and resulting privacy breaches in some scenarios. On the other hand, with low coefficients in some states, the Q-values are not severely underestimated, which is helpful for avoiding drastic jumps and the con- sequent performance degradation for value regularization methods. For explicit policy constraint methods, low co- efficients enable the actor update in the direction of the maximal Q-values, which prevents potentially inaccurate evaluations.
max (Q (s, T())) - (=())) a) (13)
Similarly, for the datasets with low variance in quality, the policy can be only constrained near the sub-dataset /
max a DI [Qnom (s, (s))
(14)
- II -I((s.a) E D) (T()) a)²1
where ((s ()) E b) denotes an indicator function that specifies whether the data point a) belongs to the sub- dataset 1) Quorm (s, π (s)) follows the normalization trick of TD3+BC (Fujimoto & Gu, 2021) and can be computed as ((s,T(s))) N E(s.a)) (Qss,
3.5. Efficient Offline-to-Oninee RL
Since the offline policy is well-trained to collect online data with higher quality, we apply the constraint term for all online data to achieve stable performance improvement.
Generally, the state-adaptive coefficients can be viewed as the confidence of the policy in given state. When the coef- ficient is low, the policy update closely follows the results of bellman updates. Similar to the online approaches, this up- date has the optimistic property that encourages the policy to explore higher-value regions. When the coefficient is high, it serves to constrain the policy near the dataset for safety, as the critic updated by the Bellman backup may guide the policy toward unreliable regions. These state-adaptive co- efficients allow the policy to determine whether trust the critic at the state level, enabling a flexible trade-off between optimism and pessimism, which efficiently minimizes the discrepancies between the offline and online update mecha- nisms.
The complete training procedure is outlined in the pseudo- code provided in Algorithm 1.
4. Experiments
In this section, we incorporated the proposed method with CQL (Kumar al., 2020) and TD3+BC (Fujimoto & Gu, 2021) and conducted extensive experiments the effectiveness D4RL (Fu et al., 2020) MuJoCo and AntMaze tasks, including HalfCheetah, Hopper, Walker2d and AntMaze environments. In Section 4.1, we first demon- strate significant performance improvements compared to the backbone methods, CQL and TD3+BC. Then, in Sec- tion 4.2, we highlight the effectiveness of our approach in
validate
Additionally, since the policy is trained on an offline dataset that spans wide range of states, the coefficients can be
6

---

Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
Table Offline performance comparison with backbone algorithms TD3+BC (Fujimoto Gu, 2021) and CQL. (Kumar 2020) on the D4RL benchmark. We evaluate the D4RL normalized scores of standard base algorithms (denoted as "Base") against those augmented with selective state-adaphe adaptive regularization (referred to as "Ours"). The symbol represents the standard deviation across the seeds. Superior scores are highlighted in bold.
TD3+BC
CQL
Avg.
Dataset
Base
Ours
Base
Ours
Base
Ours
halfcheetah-m-v2
0..
56.5:13.7
47.1=0.2
63.9:11.2
47.7
60.0
hopper-m-v2
58.7+3.9
101.6++0.4
65.6=3.55
89.1+:9.7
62.1
95.4
walker2d-m-v2
82.3+2.2
87.91:2.4
81.6=1.2
84.9111.7
81.9
86.4
halfcheetah--mr-22
44.4=0.6
49.6:10.3
45.7=004
53.8:10.4
45.0
51.7
hopper-mr-v2
66.44-27.1
101.6+00.7
92.3+99.3
101.4+22.1
79.3
101.5
walker2d-mr-v2
81.6:17.11
93.5+:2.0
19.2
94.7.13.3
80.4
94.1
halfcheetab-me-v2
92.9+2.0
94.9+1122
93.0=4.2
102.1+1.2
93.0
98.5
hopper-me-v2
101.448.2
103.8+6.7
97.8=8.6
109.6+3.2
99.6
106.7
walker2d-me-v2
110.3
112.5+1.4
109.240.2
112.2:10.9
109.8
112.4
halfcheetah-e-v2
95.911111
95.5
97.0=0.5
105.9-00.9
96.5
100.7
hopper-e-v2
108.4+3.6
10..8+4.3
108.7+2.8
112.4+00.
108.6
110.6
walker2d-e-v2
1.0.1+0.55
109.6+00.3
110.1+0.2
110.2+0..2
110.1
110.0
locomotion total
1000.8
1116.7
1030.4
1139.1
1015.6
1128.0
95% Cls
917.9~1083.7
1196.2~1137.33
990.4~1070.1
1111~1167.3
937.5~1078.6
1093.2~1162.8
umaze-v2
88.6
93.4
11.
96.0.t2.3
90.7
94.7
umaze-diverse-v2
18.2
50.0
27.8
80.2.1..9
35.5
65.1
medium-play-v2
0.0+0..0
49.4+13.4
67.0=4..2
70.2:16.7
33.5
59.8
medium-diverse-v2
0.0+0.00
47.6+11.11
60.5+99.2
71.6:19..
30.3
59.6
large-play-v2
0.010.0
18.0:44.6
24.8
53.0-14.1
12.4
35.5
large-diverse-22
0.0+0.0
17.6:19.8
21.2
35.8
10.6
26.7
antmaze total
131.8
276.0
294.1
406.8
213.0
341.4
95% Cls
18.2~1855.5
246.5~-305.5
230.9~357.3
334.9~478.7
130.1~~955.8
273.7~<40.1
the offline-to-online setting by applying simple annealing scheme to the state-adaptive coefficients. Lastly, to validate the contributions of individual components, we perform ab- lation studies to evaluate the impact of the state-adaptive coefficients and the selection methods for the sub-dataset D. All methods are run with four random seeds, and the averaged results reported.
attribute this improvement primarily to the unique ability of our method to selectively identify and focus on valuable sub-datasets, combined with the enhanced generalization enabled by bellman updates with state-adaptive coefficients.
By leveraging the sub-dataset D, the policy selectively constrained focus high-value actions, resulting in servative updates but with high returns. Additionally, the state-adaptive coefficients allow Bellman updates to general- ize more effectively, achieving superior performance within a reliable region near the sub-dataset.
4.1. Performance in Offline Setting
We first demonstrate a significant performance improve- ment over the backbone methods, CQL and TD3+BC, shown in Table 1. Our method consistently outperforms both CQL and TD3+BC across a wide range of datasets, with particularly notable improvements in datasets contain- ing lower-quality data, such as hopper-medium--v. We
In a manner similar to TD3+BC, we incorporate a behavior cloning term (.s..)) (a' - a)² /2 into the policy update for CQL the Antmaze tasks ensure stable perfor- mance. We also compare method with several advanced offline methods in Appendix B.1, further highlighting the

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offline RL
Table 2. Comparison of online fine-tuning (250k steps) performance with the baseline algorithms D4RL. benchmark.
Dataset
IQL
SPOT
FamCQL
CQL
TD3+BC
TD3+BC(SA)
CQL(SA)
halfcheetah-mediim--2
49.7
58.6
65.3
48.0
52.5
82.9+22.5
95.3++1.5
hopper-medium--2
75.2
99.9
101.0
63.8
63.7
103.5:10.
99.313.8
walker2dlmecduum-22
80.8
82.5
93.3
82.8
86.6
101.617.4
105.9=3.7
halicheccal-mmciiu---eelyyyy
45.2
57.6
73.1
49.4
49.3
73.1=3.0
2..4
hopper-meddium--replaay22
91.1
97.3
102.8
101.3
97.0
102.940.9
103.1:0.2
walkeezi-mmeddum-rrppaa--2
89.2
86.4
103.6
87.9
89.9
100.9+5.4
116.3#2.1
halfchettah-meddum-eepperrv
92.4
91.9
95.7
95.7
93.2
98.5#4.1
115.4=1.5
hopper-mneciuum-exxprttt2222
109.6
106.5
104.4
110.8
99.8
111.24:299
109.5
walker22dmmcciium-xxxett--2
115.0
110.6
110.4
109.8
115,8
115.745.5
117.5
halfcheerah-expeerr-22
96.4
94.1
106.5
97.3
95.8
102.5+0.9
113.3+00.
hopper-expert-v2
100.3
111.8
109.6
111.9
109.5
112.0+-2.4
110.8+1.4 1.6
walker2d-experrtv2
112.5
109.9
112.6
109.7
111.4
113.8
112.6
locomotion total 95% Cls min
1057.4
1107.1
1178.3
1068.4
1064.5
1218.6
1278.4
981.5
1093.1
1165.3
1058.9
1039.8
1165.4
1254.9
95% Cls max
1133.2
1121.4
1191.5
1080.1
1089.2
1248.9
1303.5
antmaze-umaze-v2
83.0
98.8
95.2
72.8
96.513.2
99.0110.6
antmaze-umarre-divree--2
38.2
56.8
59.2
39.8
87.2+5.0
95.0
antmarze-medium-play-222
78.8
92.5
77.0
0.0
16.3
88.0++2.4
attmaze-mediu--divvrs---
80,2
87.0
84.0
0.2
63.0136.1
89.0
antmazzelarge-playy-22
42.8
60.0
51.8
0.0
35.5+
66.5 + 13.4
antmaze-large-diverse-z
40.2
63.0
38.2
0.0
30.5+:5.0
56.8 1
antmaze total
363.2
458.1
405.5
112.8
389.2
494.3
95% Cls min
302.8
384.0
327.9
79.5
316.1
415.6
95% Cls max
423.7
534.0
483.1
146.1
462.4
573.4
superior performance of approach.
method in 020 tasks, even with simple linear anneal- ing of the coefficients, where (SA) indicates the use of our approach. We attribute this to the fact that adaptive con- straints in the offline process can reduce the difference be- tween offline and online updates, thereby ensuring effective exploration online fine-tuning.
4.2. Performance in Oflline-to-Online Setting
We conducted experiments the 020 setting with 250,000 online steps and compared the results with several unified 020 methods, including CQL, TD3+BC, and advanced ap- proaches such as IQL (Kostrikov et al., 2021b), SPOT (Wu et al., 2022), and FamO20 (Wang et al., 2024). FamO20 employs hierarchical models to determine state-adaptive meroovement-constraiin balances. For our experiments, we used an implementation of FamO20 integrated with CQL, referred FamCQL. However, since FamCQL. did provide its hyperparameters for the AntMaze tasks, tested the default settings but observed poor performance. As result, its scores are not included in the comparisons.
Another advantage of our approach is that can eliminate the need for access the offline dataset during online fine- tuning, as discussed in Section 4.2. For better performance improvements, in our experiments, we initialized the on- line replay buffer with a subset of high-return trajectories during online fine-tuning for Mujoco tasks, but with the entire offline dataset for Antmaze tasks. We also conducted experiments with other initialization strategies and listed the results Appendix B.2. The results demonstrate that our method achieves competitive fine-tuning performance without access to the offline dataset or when utilizing only
Table 2 illustrates the competitive performance of our
8

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offline RL
subset of it.
5. Conclusions
To unlock the potential performance improvements enabled by Bellman updates and address the challenges posed by cross-task, cross tross-training stage, and varying data densities in uniform regularization, we propose stateaddaptive regular- ization coefficients on selective states. We replace the global coefficient with state-adaptive coefficients and adaptively ad- just them based on the policy distribution by the relationship between CQL and explicit policy methods. Additionally, we pre-select sub-datasets containing high-quality actions to reasonably relax the constraints. Empirically, extensive experiments show that our approach offers a significant im- provement over various baseline methods, achieving state- of-the-an performance on the D4RL benchmark.
4.3. Ablation Study
In this subsection, we conduct ablation studies on the effec- tiveness of the state-adaptive coefficients and the selective regularization, particularly the IQL-style selection method for datasets with high quality variance.
hugper-medium repla---22
hopper- meweeea-meiimmmcoom expert-v22
Normalized Retuun
100-
Normalized Return
100
80-
40-
adeptive coclficionts
adaptive conTiicionts
20
lied coefficient
20
Fixed cncfficien
%
200
400
600
800 1000
200
4000
600 800 1000
Training Sleps(>1c5)
Trining Steps(x1e3)
Acknowledgements
Figure 3. Offfine performance comparisons of different types of the coefficient used for the regularization.
This work was supported by the NSFC (U2441285, 62222605) and the Natural Science Foundation of Jiangsu Province of China (BK20222012).
Figure 3 shows the necessity of the state-adaptive coeffi- cients, where the fixed coefficient is applied globally. As discussed in Section 1, the global coefficient constrains the policy update to a narrow region of the sub-dataset, limiting the ability to achieve performance beyond it. In contrast, the state-adaptive coefficients allow policy updates within reliable region induced by bellman updates, enabling per- formance that exceeds the dataset.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty- based offline reinforeement learning with diversified q- ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
hooper--edium- veplay.v2
wullerr24medim-eeeeett2
Normalized Retuin
100-
Normalized Return
100-
8D-
80)
60-
IQL syle
60-
TQL atyle
60-
high returns
60-
highh returnss
Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P., and Wang, Z. Pessimistic bootstrapping for uncertainty- driven offline reinforcement learning. arXiv preprint arXiv:2202.1 / 1566, 2022.
20
20
data
%
200
400
600
300
1600
200
400
600 800 1000
Training S1eps(-1c3)
Trinnng Steps(+1e3)
Figure 4. Offline performance comparisons of different selection methods for the sub-dataset.
Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948, 2023.
From Figure 4, for datasets with a wide-ranging distribution, the sub-dataset composed solely of high-return trajectories can lead to significant performance degradation and volatil- ity. This happens because the sub-dataset represents only a portion of the overall distribution and may be overly nar- row, while the entire dataset is used for updating Q-values. As result, the policy update can drift toward unreliable regions in out-of- affects updates for other states. The sub-dataset extracted using the IQL-style approach can mitigate this issue, as data is selected at the state level rather than the trajectory level, resulting in broader yet equally valuable distribution.
Beeson, A. and Montana, G. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning, arXiv preprint arXiv:2211.11800 2022.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 14:45084-5 2021.
states, which subsequently
Cheng, C.-A,, Xie, T., Jiang, N., and Agarwal, A. Adversar- ially trained actor critic for offline reinforcement learning.

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offline RL
In International Conference on Machine Learning, pp. 3852-3878. PMLR, 2022.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off- maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):41444199 2022.
Haarnoja, T., Moran, B., Lever, G. Huang, S. H., Tirumala, D., Humplik, J., Wulfmeier, M., Tunyasuvunakool, S., Siegel, N. Y. Hafner, R., et al. Learning agile soccer skills for a bipedal robot with deep reinforcement learn- ing. Science Robotics, 989))eadi8022, 2024.
Dichl, C., Sievernich, T. S., Krüger. M., Hoffmann, F., and Bertram, T. Uncertainty-awart model-based offline rein- forcement learning for automated driving. IEEE Robotics and Automation Letters, 8(2): 2023.
Hansen-Estrun P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning actor- critic method with diffusion policies. arXiv preprint arXiv:2304. Stti::334..0577.. 2023.
Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rys: What essential for offline rl via supervised learn- ing? arXiv preprint arXiv:2112.1075! 2021.
Janner, M., Du Y., Tenenbaum, J. B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.099: 229509991, 2022.
Eysenbach, B., Geist, M., Levine, S., and Salakhutdinov, R. connection between one-step and critic regularization in reinforcement learning. In International Conference on Machine Learning, pp. 9485-9507. PMLR, 2023.
Kang. B., Ma, X., Du, C., Pang, T., and Yan, S. Efficient diffusion policies for offline reinforcement learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Fang. X., Zhang, Q.. Gao, Y., and Zhao, D. Offline re- inforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Con- ference on Intelligent Transportation Systems (ITSC), pp. 3417-3422. IEEE, 2022.
Kaufmann, E., Bauersfeld, L., Loquercio, A., Müller, M.,
Koltun, V., and Scaramuzza, D. Champion-level drone racing using deep reinforcement learning. Nature, 620 (7976):982-887, 2023.
Fatemi, M., Wu, M., Petch, J., Nelson, W., Connolly, S. J., Benz, A., Carnicelli, A., and Ghassemi, M. Semi-markov offline reinforcement learning for healthcare. Confer- ence on Health, Inference, and Learning, pp. 119-137. PMLR, 2022.
Kostrikov, I,, Fergus, R., Tompson, J., and Nachum, O. Of- fline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783, PMLR, 2021a.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:21 10.06169, 2021b.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint MxXiv:2004007219 2020.
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-2011 201452, 2021.
Kumar, A., Zhou, A., Tucker, G., and Levine, Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Interna- lional conference on machine learning, pp. 1587-1596. PMLR, 2018
1 1179-1191. 2020.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learn- ing, pp. 1702-1712. PMLR, 2022.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Interna- tional conference on machine learning, pp. 2052-2062. PMLR. 2019.
Luo, F.M.,, Xu, T., Cao, X., and Yu. Y. Reward-consisee dynamics models are strongly generalizable for offline reinforcement learning, arXiv preprint arXiv:2310.0542 2023.
Garg. D., Hejna, J., Geist, M., and Ermon, S. Extreme q-learning: Maxent without entropy. arXiv preprint arXiv:2301.02.228 2023.
10

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offliee RL
Lyu, J., Ma, X., Li, X., and Lu, Z. Mildly conservative q- learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
Tang, S. and Wiens, J. Model selection for offline reinforce- ment learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Confer- ence, pp. 2-35. PMLR, 2021.
Ma. Y. Jayaraman, D., and Bastani, O. Conservative offline distributional reinforcement learning. Advances in neural information processing systems, 34: 34:19235-19247, 2021.
Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, V., and Kolesnikov, S. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RI. Work- shop: Offline RL as a "Laumchpad", 2022. URL https: / /poenreveeww net forum? Sd- 9bBcv.
Mao, Y., Zhang, H., Chen, C., Xu, Y., and Ji, X. Supported value regularization for offline reinforeement learning. Advances in Neural Information Processing Systems, 36, 2024.
Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the minimalist approach offline rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mark, M. S., Ghadirzadeh, A., Chen, X., and Finn, C. Fine- tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Uchendu, I,, Xiao, T., Lu, Y., Zhu, B., Yan, M., Simon, J., Bennice, M., Fu, C., Ma, C., Jiao, J., et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pp. 34556-34583. PMLR, 2023.
Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accel- erating online reinforcement learning with offline datasets. arXiv preprint ar http::20000093359 2020.
Wang, S., Yang, Q., Gao, J., Lin, M., Chen, H., Wu, L.,, Jia, N., Song, S., and Huang, G. Train once, get a family: State-adaptive balances for offline-to-online reinforee- ment learning. Advances in Neural Information Process- ing Systems, 36, 2024.
Nakamoto, M., Zhai, S., Singh, A., Sobol Mark, M., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208. 0610061199. 2022a.
Nikulin, A., Kurenkov, V., Tarasov, D., Akimov, D., and Kolesnikov, S. Q-ensemble for offline rl: Don't scale the ensemble, scale the batch size. arXiv preprint arXiv:2211. 10092 2022.
Wang, Z., Hunt, J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208. /93, 2022b.
Wu, J., Wu, H., Qiu, Z., Wang. J., and Long, M. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 3 1127--3121,, 2022.
Paine, T. L., Paduraru, C., Michi, A., Gulcehre, C., Zolna, K., Novikov, A., Wang, Z., and de Freitas, N. Hyper- parameter selection for offline reinforcement learning. arXiv preprint -XXx:2007.00055. 2020.
Xu, H., Jiang, L., Li, Yang. Z., Wang, Z., Chan, V. W. K., and Zhan, X. Offline with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303. 15810, 2023.
Park. S., Ghosh, D., Eysenbach, B., and Levine, S. Higl: Offline conlicondi rl with latent states actions. Advances in Neural Information Processing Systems, 36, 2024.
Yu, T., Thomas, G., Yu, L.,, Ermon, S., Zou, Y., Levine, S., Finn, C., and Ma. T. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129- 14142, 2020.
Peng. X. B., Kumar, A., Zhang, G. and Levine, S. ddvaatage-weiihher regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 2019.
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
Sun, Y., Zhang, J., Jia, C., Lin, H., Ye, J., and Yo, Y. Model- bellman inconsistency for model-based offline reinforce- ment learning. In International Conference on Machine Learning, pp.
PMLR, 2023.
Yu, Z. and Zhang, X. Actor-critic aligmment for offline-to- online reinforcement learning. In International Confer- ence on Machine Learning, pp. 10452-40477 PMLR, 2023.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.

---

Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00095, 2023.
Zhao, K., Ma, Y., Liu, J., Zheng, Y., and Meng, Z. Ensemble- based offline-to-online reinforcement learning: From pes- simistic learning optimistic exploration. arXiv preprint arXiv:2306.066 06871, 2023.
Zhao, Y., Boney, R., Ilin, A., Kannala, J., and Pajarinen, Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning. arXiv preprint arXiv:2210. 124.132200. 2022.
Zhu, Z., Tian, H., Chen, X., Zhang, K., and Yu, Offline model-based reinforcement learning with causal struc- tured world models. Frontiers of Computer Science, 19 (4):194347, 2025.
12

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offline RL
A. Related Work
Offiine RL In offline RL, inaccessible interactions with the environment pose challenges such as extrapolation error and overestimation due to out-of out-of-distribution (OOD) actions (Fujimoto et al., 2019; Kumar et al., 2019). Current model-free offline RL approaches can be categorized into two main groups based on their constraint forms. Value regularization methods learn conservative value function mitigate overestimation (Cheng et al., 2022; Kumar al., 2020; Bai et al., 2022; Ma et al., 2021; Kostrikov et al., 2021a), while policy constraint methods explicitly or implicitly constrain policy updates close to the behavior policy (Fujimoto & Gu, 2021; Wu et al., 2022; Wang et al., 2022b; Kostrikov et al., 2021b; Nair et al., 2020). Recent methods have added regularization terms to the policy objective, deriving the regularization term for both the updates of policy and Q-values (Garg et al., 2023; Xu et al., 2023; Tarasov et al., 2024). Some model-based offline RI. methods achieve exceptional performance ( (Yu et al., 2020; 2021; Sun et al., 2023; Luo et al., 2023; Zhu et al., 2025) due the generalization capabilities of learned models. Additionally, recent works have framed policy learning as a supervised learning and sequence modeling problem using advanced network architectures (Emmons et al., 2021; Chen et al., 2021; Janner et al., 2022). However, due to the pessimistic constraints for reliable learning, several hyperparameters need to be determined before employing offline algorithms, most of which are empirical (Paine et al., 2020). Furthermore, most current algorithms use a uniform constraint coefficient across all states, despite variations in data density.
Value regularization Value regularization methods penalize the Q-values of OOD actions, thereby reducing the prob- ability of selecting such actions as the policy is derived from the Q-values. Some works address extrapolation error and overestimation by applying regularization constraints to suppress the Q-values of OOD actions. CQL (Kumar et al., 2020) augments the standard Bellman error objective with a Q-value regularizer that provides a lower bound on the value of the current policy. Various CQL variants adjust the constraints or modify the regularizer to avoid excessive pessimism (Lyu et al., 2022; Nakamoto et al., 2024; Mao et al., 2024; Yu et al., 2021). Ensemble-base approaches (An et al., 2021; Nikulin et al., 2022) use a similar Bellman target by estimating uncertainty to mitigate the overestimation of OOD actions. Notably, one study demonstrated the connection between value regularization and one-step RL. (Fysenbach et al., 2023) by modifying the entire policy evaluation formula, while we reveal the link between the regularizer of CQL and explicit policy constraint methods.
Policy constraint Unlike value regularization methods, policy constraint methods maintain the policy evaluation formula but constrain policy updates during the improvement stage. TD3+BC (Fujimoto & Gu, 2021) illustrates the effectiveness of a simple policy constraint applied to the off-policy TD3 algorithm (Fujimoto et al., 2018). Recent works focus on explicit policy constraints for stochastic policies (Wu et al., 2022: Nair et al., 2020) or more expressive policies (Kang et al., 2024; Wang et al., 2022a). Another implementation of policy constraints is the implicit policy constraint derived from the KL-divergence between the learned policy and the behavior policy, known as advantage- weighted behavior cloning (Peng et al., 2019). Many works based on this approach have demonstrated significant performance improvements across various tasks (Nair et al., 2020; Kostrikov et al., 2021b; Wang et al., 2024; Park et al., 2024; Hansen-Estruch et al., 2023). Among them, FamO20 (Wang et al., 2024) also utilizes state-adaptive coefficients, but it is limited to IQL-style methods with implicit constraints and the entire dataset. Moreover, the coefficients are updated by maximizing Q-values, lacking the interpretability offered by our method.
Offinneto-online RL The distribution shift between the offline dataset and that induced by the learned policy is the most challenging issue in offline-to-online RL. Policy constraint methods can mitigate this distribution shift by keeping policy updates close to the replay buffer. For transitioning from offline RL to online settings, some methods adopt the key idea of policy constraints while adjusting constraints for improved online performance, such as decaying the constraint coefficient in a certain way (Beeson & Montana, 2022; Zhao et al., 2022; Wang et al., 2024). Additionally, certain offline methods can be directly applied to 020 RL (Nair et al., 2020; Kostrikov et al., 2021b). However, the use of a uniform constraint coefficient makes difficult to balance stability and speed, inhibiting efficient online fine-tuning. Conversely, transitioning from value regularization offline methods to online poses challenge due the sudden increase in Q-values, resulting from the pessimistic estimation of OOD actions in offline RL. During online fine-tuning, the agent will estimate the -aluuuss of OOD actions optimistically, leading to a rapid increase in overall Q-values and subsequent performance degradation. Some works have relaxed constraints to prevent severe underestimation of OOD actions during offline learning (Lyu et al., 2022; Nakamoto et al., 2024) or utilized Q-ensemble methods to maintain reliable estimations at a higher computational cost (Lee et al., 2022; Mark et al., 2022; Zhao et al., 2023). In contrast, our method assigns different levels of pessimism based on the distance between the output of the learned policy and the reliable sub-dataset, avoiding unreasonable underestimation. Thus,
13

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offline RL
Table Offline performance comparison with prior methods the D4RI. benchmark. The mean-wise hest results among algorithms highlighted in bold.
Dataset
AWAC
IQL
Cal-QL
SPOT
FamCQL
TD3+BC(SA)
CQL(SA)
halfcheetah-m
09.8+0.3
48.1
47.
57.6+10.6
58.1+0.5
56.3+3.77
63.9:+
hopper-m
68.6:1112
66.714.4
64.713.4
71.4177.2
82.3+16.0
101.6:104
89.149.7
walker2d-m
85.1
74.8=1.8
84.
69.6=30.2
87.4
87.9124
84.9
halfcheetah-m--r
45.4
44.5++0.3
4..
52.3+00.7
51.9+
49.6++0.3
53.8++0.4
hopper-m-r
97.8+1.4
89.6#11.9
93.416.6
87.1=23.9
85,4
101.6::0.7
101.442.1
walker2d-m-1
73.2+8.4
80.6+5.88
84.7=1.4
88.9+1.6
88.6=1.5
93.5#2.0
94.7=3.3
ha
95.31009
91.8+1.11
52.71:5.4
92.7+2.3
91.7:1288
94.911.2
102. 1.2 1.2
hopper-m-e
108.642.3
106.317.4
107.6:2.4
102.1
92.9
103.8:6.7
109.6
walker2d-m-e
89.7+33944
111.9#110
109.0
110.3+0.2
110.6+0.4
112.5+1.4
112.2
halfcheetah-e
97.1+0.5
95.1+3.1
96.1+0.9
94.4+10.5
93.4+0.9
95.5+113
105.9
hopper-e
99.7=11.11
111.1
111.910.3
110.1
111.7#1.
109.8:4..3
111.4
walker2d-e
112.740.3
113.010.2
109.1+0.33
110.1
110.1
109.6:0..3
110.2
locomotion total
1023
1033.5
1007.5
1046.6
1064.1
1116.7
1139.1
antmaze-u
63.5+19.2
74.8
1.88
88.8
93.4
2.3
antmaze-u-d
57.8+8.0
52.2:6.44
16.2+20.11
41.545.3
50.0:
80.2.1..9
antmaze-m-p
0.0:00.0
63.8
69.5
63.0=#1..9
49.4#
70.2
antmaze-m-d
0.0
61.2++2.5
64.0
67.0=22..5
47.6+12.1
71.6++9.3
antmaze-l-p
0.0.10.0
37.813.8
41.8
34.0
18.044.6
53.0
antmaze-l-d
0.0
20.811..1
32.819..3
16.2
17.619.8
18.8
antmaze total
121.3
310.6
299.1
330.5
286
406.8
the Q-values estimated during the offline stage will fall significantly below the values evaluated during the online stage, preventing drastic jumps in Q-values. Furthermore, the adaptive coefficients relax the constraints in states and achieve more effective fine-tuning.
B. More Experimental Results
B.1. Comparison with some advanced offline methods
We also compared our method with several common advanced baseline methods. The results shown Table demonstrate competitive performance our method.
B.2. Different initializations for online replay buffer
notable advantage of our method its ability to reduce reliance on offline datasets, as the sate-adaative coefficients can generalize to OOD data encountered during the online fine-tuning process. We conducted 020 experiments by initializing the online replay buffer with four different types: (1) Initialize the buffer with the entire offline dataset akin (Kostrikov al., 2021b). (2) Conduct separate online buffer and sample symmetrically from both the offline dataset and online buffer akin to (Ball et al., 2023). (3) Initialize the buffer with the sub-dataset D used in offline training. (4) Initialize the buffer without any offline data. For simplicity, we denote them as all, half. part and none respectively.
From the results shown in Table 4, we can see that in non-sparse reward tasks, our method achieves good fine-tuning performance even without accessing the offline dataset. challenging sparse reward tasks, fine-tume tuning with offline data performs better due the difficulty in accurately capturing the value function distribution (especially for TD3+BC). In our
14

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offliee RL
implementation, initialized the online replay buffer during fine-tuning with high-value subset of the offline dataset for Mujoco tasks, whereas for Antmaze tasks, the entire offline dataset was used.
Table 4. The performance comparison of online fine-tuning with different sample strategics.
TD3+BC(SA)
CQL(SA)
Dataset
all
half
part
all
half
part
halfcheetah-medium--22
61.2
63.4
82.9
66.6
71.2
70.6
95.3
69.8
hopper-medium-Vy
102.6
103.1
103.5
97.0
102.1
77.7
99.3
89.4
walker2d-medium--22
99.6
101.5
101.6
101.8
86.6
105.3
105.9
104.1
halichetal-mmcdiin---plaay
65.9
69.4
73.1
71.6
59.9
61.8
79.4
62.4
hopper-mecdium--rpplaa--2
100.7
101.4
102.9
100.9
103.2
103.0
103.1
102.8
walker224mmcciuu--repaay-2
96.2
98.0
100.9
101.6
101.5
110.2
116.3
105.9
halfcheetah-mediuu-eeerrrrr2
99.0
98.0
98.5
99.9
117.8
103.3
115.4
106.1
hopper-mncduum--xxprr-22
98.1
111.9
111.2
87.4
110.8
111.3
109.5
110.3
walker2dd-medduu--erppe--
116.4
113.1
115.7
109.7
114.9
112.4
117.5
113.8
halfcheetah-experrt-22
103.4
102.5
102.5
103.3
109.1
108.5
113.3
104.7
hopper-expert-v2
107.6
108.8
112.0
98.9
110.4
110.8
110.8
100.6
walker2d-expert-v2
110.6
111.4
113.8
112.6
111.7
111.6
112.6
112.1
locomotion total
1154.8
1180.9
1218.6
1165.0
1196.0
1192.1
1279.4
1190.1
antmaze-umaze-v2
96.5
95.5
95.8
95.8
99.0
99.2
97.8
97.8
antimaze-umazzediiveee--22
87.2
61.5
78.8
83.5
95.2
93.8
95.0
95.0
antmarz-nmeduun-plyy-422
76.5
59.0
18.8
2.8
88.8
84.2
86.2
83.5
antmaze-mediinn-divreee>
63.0
38.0
38.8
2.0
89.0
89.8
88.0
88.5
autmaze-large-paayy22
35.5
31.2
14.0
5.8
66.5
57.8
58.2
53.0
uutaazo-arree-iiveeee-222
30.5
35,2
18.2
9.2
56.8
42.5
26.5
26.5
antmaze total
389.2
320.4
264.4
199.1
495.3
467.3
451.7
444.3
B.3. Computational cost
Our method introduces a lightweight network with minimal overhead. Below is the training time on 2080 GPU, excluding IQL-style pretraining, showing a slight increase in time cost.
Algorithm
TD3+BC
TD3+BC(SA)
CQL
CQL(SA)
Time Cost(h)
2.25
2.69
7.22
8.05
Table The computational time required by TD3+BC, CQL, and the integration of our proposed method,
C. Proof
Derivation of Eq. (12) Revisiting the interaction process TD3, when the policy takes an action, the exploration noise sampled from Gaussian distribution N(00 8) and added to the output IT()) the policy. Thus, the action can viewed as a sample from stochastic policy N (s)(s), 6), denoted Mact-
15

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offliee RL
We explicitly compute the logarithmic value of the policy Foct as follows
log (ale( -log 2T0 (a Ts)))
(16)
262
By plugging it to Eq. (9), can get
Lg(ø) = E (s.aa) flo (ece(als) (a))) Cn(s)]Bo(s)
(+,))= log V2rd (a xs)))²
n²80
log V2Tō
282 1B((s)
(17)
282
20 B (era)= (n20²
- (a D(
[[s))2||((())
where (s) = min flog Fatt (I() + nō|s). ss, log Tact (T(s) - no(s)) = log Tact (TT(S) Tact(
+ nōls).
Omit the coefficient 1/20?, we can get Eq. (12).
Proof of Proposition 3.1 Revisit the original formula of CQL(p) (Eq. (3) in (Kumar et al., 2020) with the regularizer of
KL-divergence)
- B - 8 - ( (18)
min maxx((BrDDaarrpallll00 - Es,an +
Consider the inner optimization problem for a more general form
max ((EorIff(z)) DKL(HlP)) s.t. Em(z) 1, mzx) 2 0 v
(19)
The optimal solution is equal to
p(x) exp (f(x))
1º (z) (x)
(20)
Σ ((rr exp (f(z))
If we set f(x) = ()s,aa/oo a)/a, then we can get
min B a E.-D Σ explQ(s,a)/a) _ EaLD BBeDDD((ssa)))
-Bo - 8 (
(21)
regularizer
As the policy is modeled as Boltzmann distribution, get
exp exp(Q(saa))
( A, x(als) =
(22)
Σ expp(Q(saa)/o
Then
1 Q(s a) log (als)
V(s a) E S A, log Σ exxp(Q(s,a)/o)
(23)
Select actions that follow the distribution the behavior policy, and plug Eq. (23) into the regularizer R in Eq. (21)
R 3 EAND Ean 1 Q(,, - log [ags() 1 Ba-
a
(24)
= 3E..~ D. log (a[s))
D. pseudo code
16

---

Learning to Trust Bellman Updates: Selective Statc-Adaptive Regularization for Offline RL
Algorithm 1 state-adaptive Regularization for offline and offline-to-online RL
Input:
the return threshold Gr. the initial value Nstorts the end value Rends update interval Tines offline total steps T, online decay steps Nend
Initialize Q network Q. V network V with random parameters for IQL-style update
Initialize critic network Qu, actor network Tw. coefficient network S& with random parameters for offline policy learning, replay buffer D with the offline dataset # Pre-training
Compute the returns G for all trajectories or train Q and V by Eq. (11)
Obtain the sub-dataset D by G Gr or Q(8, a) V(s) >
Offfine training
for iteration 1, 2, ........................ do
Sample mini-batch B (s, a, s', d) from D, where d is the done flag Update nw by Eq. (2) (Eq. (13) Eq. (14) for TD3+BC)
Update Q by Eq. (7) or Eq. (10) (Eq. (3) for TD3+BC)
Update 3/ by Eq. (9) (Eq. (12) for TD3+BC)
if % Tine = then
if (a.D)) [log 7 (a]s) (s) then Increase n by Eq. (8)
else
Terminate the update of
end if
end if
end for
# Online training
Reset the replay butfer D according to the given strategy for iteration 2, do
Interact with the environment and store the new transition in the replay buffer D
Obtain the annealed coefficients 3on by Eq. (15)
Update Tw by Eq. (2) (Eq. (13) for TD3+BC)
Update Q by Eq. (7) (Eq. (3) for TD3+BC)
end for
E. Implementation details E.1. Baseline implementation
We reproduce all results of baseline algorithms, except for FamCQL, using the deep offline RL library CORL (Tarasov al., 2022) http:://gituu...ommtioofffff.ooo For FamCQL, reproduce the results according the official code http://hhbbbbboommmooooooo//tt//
All hyperparameters are kept consistent with the official implementations.
E.2. Network structure of the coefficient generator
We employ three-layer MLP architecture with 512 neurons per hidden layer to parameterize 3(s). The sigmoid activation function is adopted in the final output layer, with output values constrained to the interval (0, 1.58mmat), where Bini denotes the regularization coefficient from the original implementations of both CQL and TD3+BC algorithms.
Additionally, to prevent catastrophic value overestimation, we applied the trick of Layernorm on the critic networks.
E.3. Return thresholds
Since the critic induced by the deterministic policy is less robust, the return thresholds of TD3+BC are slightly lower than that of CQL. The return thresholds are shown in Table 6. For the dataset involving replay data, we use the IQL method to filter valuable actions. The value of 7 set as 0.7, the same as the original implementation IQL.
Furthermore, in Antmaze tasks, substantial fluctuations were observed in the number of valuable actions acquired through
17

---

Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
IQL pre-training. Therefore, we construct the sub-dataset D by selecting trajectories that complete the task successfully.
Table 6. The return thresholds for different tasks.
Dataset
CQL(SA)
TD3+BC(SA)
aeficheetah-mediume
6000
5200
hopper-medium--v2
2500
1800
walker2d-mecium--2
3600
2500
halfchhectah-expert-22
11000
10500
hopper-expert-v2
3500
3500
walker2d-expert-v2
4800
4500
E.4. Distribution-wware threshold setting
For the distribution-awann threshold, in Mujoco tasks, set Natart 1 and Nend primarily to 3 (1.5 for expert datasets to better mimic high-quality actions). In Antmaze tasks, we set Wend to 5 for more efficient exploration. Other details about the hyperparameters can be found in our open-source code.
E.5. Offline-to-online setting
After offline training, the policy interacts with the environment for 5,000 warm-up steps without any parameter updates. The decay steps Nend is set to 400,000, while the total number of interaction steps in our experiments is limited to 250,000. For hard Antmaze tasks in TD3+BC(SA), we keep the coefficients unchanged during the online fine-tuning because the coefficients are relatively small after the offline process. And we set the interval for policy updates to a larger value to achieve stable updates.
18


Reviews:
[{"review_id": "D3a6IVmEuT", "reviewer": "Reviewer_i2Gp", "summary": "This paper proposes to learn a state-adaptive function to dynamically judge how reliable the Bellman update is. This method starts with CQL and transforms the hyperparameter used to modulate how much the value regularizer (i.e. the term responsible for making the value function more conservative) into a parameterized function which relaxes constraints around areas near the dataset and raises them in areas with high uncertainty. The constraint is only applied to state-action pairs in a sub-set of the full dataset to counter issues with low-value data. They apply this value constraint method to policy constraint methods. Finally, they compare empirically to TD3+BC (policy constraint method) and CQL (value constraint method) and show improvement.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks you for the high praise and the comprehensive review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.\n\n**Q1: Lack of statistical significance** \n\n**A1:** To ensure statistical rigor, we have now included 95% confidence intervals (CIs) for all domain tasks in Tables 1 and 2, consistent with prior works.  The revised tables are as follows:\n\n**Table 1 (offline performance)**\n|Dataset |TD3+BC | TD3+BC(SA)| CQL |CQL(SA) | Base| Ours | \n|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n|...|...|...|...|...|...|...|\n|locomotion total |1000.8 | **1116.7**|1030.4 |**1139.1** |1015.6| **1128.0**|\n|95% CIs|917.9~1083.7|1096.2~1137.3|990.4~1070.1|1111~1167.3|937.5~1078.6|1093.2~1162.8|\n|...| ...|...|...|...|...|...|\n|antmaze total | 131.8|**276.0** |294.1|**406.8**| 213.0| **341.4**| \n|95% CIs | 78.2~185.5|246.5~305.5| 230.9~357.3|334.9~478.7| 130.1~295.8|273.7~409.1| \n\n**Table 2 (offline-to-online performance)**\n|Dataset |IQL |SPOT|FamCQL |TD3+BC |CQL|TD3+BC(SA)|CQL(SA)\n|:---:| :---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|...| ...|...|...|...|...|...|...|\n|locomotion total | 1057.4|  1107.1| 1178.3|1064.5 | 1069.4| 1218.6| **1278.4**| \n|95% CIs| 981.5~1133.2|1093.1~1121.4|1165.3~1191.5|1039.8~1089.2|1058.9~1080.1|1165.4~1248.9|1254.9~1303.5| \n|...| ...|...|...|...|...|...|...|\n|antmaze total |363.2 |  458.1| - |76.8 |405.5 | 389.2|**494.3** | \n|95% CIs |302.8~423.7 | 384.0~534.0| - |79.5~146.1 |327.9~483.1 |316.1~462.4|415.6~573.4| \n\nThese results confirm that our method achieves statistically significant performance improvements across different algorithms and task domains.\n\n**Q2: About hyperparameters** \n\n**A2:** The hyperparameters used in our implementation are provided in Appendix D. For fair comparisons, we adopt the same hyperparameter configurations for CQL and TD3+BC as in the CORL benchmark [1].  Additionally, for the distribution-aware threshold, we set $n_{start}$ to 1 and $n_{end}$ primarily to 3 (1.5 for expert datasets to better mimic high-quality actions). We will include a detailed listing of these parameters in the revised version for greater clarity.\n\nWe also deeply appreciate your constructive suggestions, including pointing out typographical errors, which we have now corrected.  Thank you again for your thoughtful review and for helping us improve our work.  We hope our responses sufficiently address your concerns, and we remain open to any further questions or clarifications.\n\n[1] Tarasov, Denis, Alexander, Nikulin, Dmitry, Akimov, Vladislav, Kurenkov, Sergey, Kolesnikov. \"CORL: Research-oriented deep offline reinforcement learning library\". _Advances in Neural Information Processing Systems_ 36. (2023): 30997–31020.", "claims_and_evidence": "The main claim of the paper is that a state-dependent regularizer coefficient in CQL will outperform a fixed parameter. This claim is supported through a series of empirical comparisons between their method and a fixed parameter. While some work needs to go into strengthening their statistical claims (see below), they mostly provide compelling evidence for their approach.", "methods_and_evaluation_criteria": "Making a parameter state-dependent is generally a great way to make a method more flexible and easier to use. The regularization parameter in CQL is a good candidate for this process. The criteria used for the optimization is reasonable and laid out in an intuitive manner. Overall, I believe the method proposed does a good job at improving other approaches with a fixed coefficient.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "W-1. While the design of the experiments are good in theory, I have noticed some odd patterns in the results presented in table 1. Typically a bold-faced result means it is the best performing method w/ statistical significance. But there are several tasks which have overlapping confidence intervals between bolded and non-bolded methods (for instance TD3+BC hopper-expert-v2). While overlapping CIs don't always suggest non-statistical significance, without more details the reader is left to assume these are the standard error, meaning a lack of statistical significance in many of the results of Table 1. This is compounded in Table 2 where baselines don't have confidence intervals reported.\n\nThere are several papers you can find that discuss these issues. Here are a few:\n- [Deep Reinforcement Learning that Matters](https://ojs.aaai.org/index.php/AAAI/article/view/11694)\n- [Empirical Design in Reinforcement Learning](https://www.jmlr.org/papers/v25/23-0183.html)\n\nI would recommend re-thinking how you bold results in your tables, and make sure results you are highlighting are statistically significant. Just having one mean larger than another does not prove it is performing better.\n\n## Hyperparameters\n\nYou discuss in detail the hyperparameters added by your approach, but neglect the standard hyperparameter choices. It is possible I missed these, but this should be discussed somewhere in the main text, or at least expanded on in the appendix.", "relation_to_broader_scientific_literature": "This approach very clearly fits into the literature by making the regularization parameter of CQL a state-dependent function.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper is relatively strong. There are some issues, such as the lack of confidence intervals in table 2, which make the paper less appealing. But if the issues in W-1 could be solved (which I believe should be straightforward), then I think this paper is ready to be accepted.", "other_comments_or_suggestions": "- The phrasing for line 025 in the abstract is a bit odd. The sentence \"On the one hand,...\". This sentence just lists the new things done by the paper, not an either or (which the phrasing suggests).\n- Line 133 second column, you reference equation 4 and 5. I think you meant to reference 3 and 4.\n- $\\mu$ and $\\sigma$ are not defined in the context of equation 6. I believe you mean the mean and standard deviation, as using the initial state distribution here doesn't make sense.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5V49xdWyoC", "reviewer": "Reviewer_ofHf", "summary": "The paper introduces a selective state-adaptive regularization method for offline RL to address the challenge of extrapolation errors caused by varying data quality. Unlike existing methods that apply uniform regularization across all states, the proposed approach learns adaptive regularization coefficients and selectively applies regularization only to high-quality actions, preventing performance degradation from over-constraining low-quality data. Extensive experiments on the D4RL benchmark show that this approach significantly outperforms state-of-the-art offline and offline-to-online RL methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and positive recognition of our work. We appreciate your thoughtful feedback and are pleased that you found our paper to be \"well-written, methodologically sound, and of practical value.\" We also appreciate the questions you raised and are committed to delivering a comprehensive response to address your concerns in detail.\n\n**Q1: Code provision** \n\n**A1:** We have carefully verified that the code has been included in the supplementary material.\n\n**Q2: The collection of offline data** \n\n**A2:** Our approach is evaluated on the publicly available D4RL benchmark [1], which provides diverse offline datasets. These datasets are collected using various strategies, including hand-designed controllers, human demonstrators, multi-task agents performing different tasks in the same environment, and policy mixtures. Given that prior works on offline RL have typically used these datasets without explicitly describing their collection process, we followed the same convention. Further details regarding dataset collection can be found in the original D4RL paper.\n\n**Q3: Relatively incremental contribution and lack of theoretical contribution** \n\n**A3:** Existing approaches employ global regularization to constrain policy updates across the entire dataset. However, this can lead to excessive pessimism, limiting the agent's ability to harness the full benefits of Bellman updates. Building on the core insight of pessimism in offline RL (Proposition 3.1), we propose a selective state-adaptive regularization mechanism that dynamically adjusts regularization coefficients based on the likelihood of high-quality dataset actions under the learned policy. This allows the agent to assess Bellman update confidence at the state level. Notably, the update mechanism is algorithm-agnostic, making it a broadly applicable enhancement that can be integrated with various offline RL algorithms.Another key contribution of our approach is demonstrating that state-adaptive regularization mitigates the excessive pessimism of fixed regularization, enabling RL agents to better exploit Bellman updates. Given the heterogeneous quality distribution in offline datasets, this state-wise adaptation offers a promising direction for improving offline RL performance.\nAlthough our paper does not provide a rigorous theoretical guarantee for the proposed method, given that the regularization encourages the learned policy to mimic dataset actions, the performance lower bound of our method can be approximately ensured by the behavior policy. Furthermore, as our method solely modifies the regularization coefficient, it does not compromise the convergence guarantees of base algorithms such as CQL under standard assumptions.\n\nThank you for your review again. We hope our response sufficiently addresses your concerns, and we remain available for any further clarifications.\n\n[1] Fu, Justin, Aviral, Kumar, Ofir, Nachum, George, Tucker, Sergey, Levine. \"D4rl: Datasets for deep data-driven reinforcement learning\". _arXiv preprint arXiv:2004.07219_. (2020).", "claims_and_evidence": "The main claims seem reasonable and supported by evidence.", "methods_and_evaluation_criteria": "The methods seem good for this type of work. The experiments are done on many standard RL test environments, and the baseline algorithms seem reasonably chosen. The only problem, unless I missed something, the code has not been provided. Moreover, there is lacking information on how the offline data has been collected, this seems important given the focus of the paper.", "theoretical_claims": "There is just one theoretical statement, Proposition 1. It seems quite straight forward characterization of the regularization term, similar as is done in similar papers, e.g., (Kumar et al., 2020). I check the main steps in the proof, it is short and easy to follow, I think it is correct. One small feedback to the author is that the text in Proposition 1 could be improved, maybe split into two sentences, I felt it was difficult to read.", "experimental_designs_or_analyses": "Seems reasonable.", "relation_to_broader_scientific_literature": "Seems reasonable.", "essential_references_not_discussed": "No  that I am aware of.", "other_strengths_and_weaknesses": "One of the main weaknesses of the paper is that its contribution appears relatively incremental. While the idea of more refined regularization is useful, it feels like a minor extension of existing work, which limits its overall impact. Additionally, the paper lacks theoretical contributions, unlike related work that provides convergence guarantees, sample complexity bounds, or similar results. It would be beneficial to establish similar theoretical results to strengthen the paper’s contribution. Alternatively, providing simple illustrative examples where the benefits of the proposed method are clearly demonstrated could help justify its significance.\n\nOn the positive side, the paper is well-written and presents a methodologically sound approach. The evaluation is rigorous, and the results show clear improvements over baseline algorithms. These empirical gains support the practical value of the method, even if the theoretical foundation could be stronger.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kUk2FKfe5A", "reviewer": "Reviewer_wb2q", "summary": "This paper introduces an offline RL method that balances regularization strength conditioned on the provided state. This state-adaptive form of regularization is applied to CQL and explicit policy constraint methods demonstrating improved performance in both offline and offline-to-online settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review and positive recognition of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.\n\n**Q1: Questions about claims** \n\n**A1:** Our main claim is derived from Proposition 3.1 and can be succinctly stated as balancing pessimism and optimism by constraining the divergence between the likelihood of high-quality dataset actions under the learned policy and a predefined threshold. This motivates the development of our state-adaptive mechanism, which adjusts regularization dynamically as divergence varies across different actions. Since the learned policy's distribution is directly used to update the regularization coefficients, we propose distinct methods tailored for stochastic and deterministic policies. However, the core insight remains the same for both, as detailed in the derivation of Eq. (12) in Appendix C. Additionally, due to the variability in dataset distributions and differences in reward scales, some manual hyperparameter tuning is unavoidable.\n\n**Q2: Questions about methods** \n\n**A2:** 1.) For the use of Proposition 3.1, as discussed in A1, it reveals that the regularizer in CQL inherently increases the likelihood of dataset actions under the learned policy. Building on this insight, we formulated Eq. (5) to strike a balance between pessimism and optimism.\n2.) For $G_T$ and the schedule used to update $n$, a pre-defined metric is essential to select  high-value actions. In contrast, the schedule is not strictly necessary. If training time is sufficient, $n$ can be updated based on some metrics, such as the loss. Such pre-defined hyperparameters are common in RL, and the adaptivity in our approach is primarily reflected in the **state-dependent regularization coefficients** across varying states.\n3.) For the different objectives in our approach, as shown in Fig. 2, a sub-dataset of high-return trajectories may not cover most of the offline dataset, leading to over-optimism in uncovered regions. To address this, we pre-train IQL critics (independent of the learned policy) to filter a reliable high-value sub-dataset that covers most of the offline dataset.\n4.) Lastly, for the \"speculative lanugage\", our claims are empirically supported by the ablation study and the experimental results presented in Table 4.\n\n**Q3: Weakness 1 about the separate learnable parametrizations** \n\n**A3:** As discussed in A1, Eq. (5) applies to the stochastic policy, while Eq. (12) is for the deterministic policy (since the deterministic policy lacks an explicit policy distribution); however, both share the same key insight.\n\n**Q4: Weakness 2 about construction of the paper** \n\n**A4:** We introduced the technique of IQL to address the sub-dataset selection problem, which is independent of CQL. \nRegarding the paper structure, our goal is to make adaptive adjustments to the constraints across different states in order to maximize the potential benefits of Bellman updates. We begin by proposing a state-level regularization updating mechanism in Section 3.1, stemmed from Proposition 3.1. To automatically select the appropriate threshold for this mechanism, we introduce a method based on distributed perception in Section 3.2. To further ensure that constraints are valid and to fully harness the benifit of the RL form, we propose selective regularization in Section 3.3. Additionally, we extend the method to deterministic policy algorithms that lack explicit policy distributions.\nWe will provide a summary at the end of each sub-section and emphasize that Eq. (5) and Eq. (12) are fundamentally the same but apply to different policy formulations. Thank you for your constructive suggestions.\n\n**Q5: The definition of $\\mu$ and $\\sigma$ in Eq.(6)**\n\n**A5:** $\\mu$ is the mean of a stochastic policy and $\\sigma$ is the standard deviation. \n\n**Q6: Details of Fig.1** \n\n**A6:** We trained the policy both with and without selective regularization, and then used the trained policy along with all data to compute the values presented in Fig. 1. In Fig. 1, we highlight the advantage of selective regularization: it helps to avoid the imitation of low-value actions.\n\n**Q7: Selection of $G_T$** \n\n**A7:** The selection of $G_T$ depends on the dataset, similar to previous works like DT and RvS, as reward scales vary significantly. Since the dataset's reward information is available, it's easy to determine a proper value.\n\n**Q8: Two-phase training paradigm** \n\n**A8:** Yes, we first pre-select the high-value sub-dataset before initiating offline training.\n\n**Q9: $n$ used  in Eq.(12)** \n\n**A9:** Yes, details are in Appendix C.\n\n**Q10: The region of policy learning** \n\n**A10:** The policy is updated across all states, but regularization related to the policy (if applied) is restricted to the sub-dataset.\n\nThanks for your review again. We hope our response sufficiently addresses your concerns, and we remain available for any further clarifications.", "claims_and_evidence": "It was difficult to clearly articulate what claims this paper proposes as they are not stated anywhere in the paper. The paper claims to unify value and policy regularization approaches using state-adaptive regularization but still resorts to developing separate methods for each class of offline RL algorithm. This was fairly disappointing. Additionally there was a lot of narrative strength given to creating a wholly adaptive approach yet I was disappointed to find a lot of hand-tuned or scheduled parameterizations of the underlying objectives.", "methods_and_evaluation_criteria": "The paper attempts to unify the two major forms of regularization among contemporary offline RL approaches by bridging between value regularization and policy constraint methods. This appears to be begun by proposition 3.1 but the result of the proposition goes unused in the development of the objective set out in Equations 6 and 7.  \n\nI am generally supportive of creating state-adaptive approaches to dealing with the inherent partial observability problem. However, this paper seems to shift the “global” parametrization of regularization to the selection of the hyperparameter $n$ which defines the shape of the trust region. Even with a linear annealing of $n$ it’s not totally satisfying that so much attention was paid to set up prior regularization methods as inflexible or limited when there is a similar (yet downplayed) assumption being made in this work. The distributionally aware notion of the threshold is nice in principle but it rests on a pretty strong a priori assumption (line 185-188). As the threshold is updated on a set schedule with an early stopping condition, I am not comfortable calling this an adaptive scheme. As implemented and discussed in this paper, this threshold is exactly task/state/data distribution-agnostic. This same complaint exists for the selective threshold $G_T$ introduced in Section 3.3 as well as the annealing of the regularization coefficient in online fine-tuning of the offline policy (equation 15). \n\nThere also doesn’t seem to be a whole lot of consistency about what the specified objectives are for the proposed approach. There is fairly little organization around what objectives are used and when and how they all relate together. There appears to be a CQL version of the state-adpative regularization as well as a TD3+BC version, both with pretrained value functions from IQL? With different formulations depending on the quality of the dataset (e.g. differences between Eqts 13 and 14?)\n\nThere is a lot of speculative lanugage about the scale of the regularization coefficient reflecting confidence in the quality of the dataset. It would be nice to have this demonstrated with actual analyses rather than speculative language. This is continued when talking about the advantages of the proposed approach over IQL (lines 308-314). It would be nice to have these assumptions clearly identified because the advantage of the proposed approach really rests on the quality of the trained regularization coeffcient network. Assuming it’s accurate and generalizes to the online setting is great but perhaps is wishful thinking? This is further exacerbated by the claim at the end of Section 3 (lines 326-329); there is no evidence for this, only willful speculation.", "theoretical_claims": "Not closely. Ultimately, I felt that the paper eschewed its theoretical claims pretty early on and simply compiled various components from prior literature when forming its own proposed method.", "experimental_designs_or_analyses": "Yes, I felt that the experiments were well set up. There is however a lot of analysis to support the claims of the scale and direction of state-adaptive regularization that should have been included. I am reduced my final score as a result of this.", "relation_to_broader_scientific_literature": "I feel that the paper is well oriented among the relevant offline RL literature. There is fair discussion of prior work and its limitations as well as extensive baseline comparisons (mostly included in the appendix).", "essential_references_not_discussed": "I felt that the paper fairly covered the offline RL literature. I would however recommend that the authors consider `Moskovitz, et al (NeurIPS 2021) “Tactical optimism and pessimism for deep RL”` which presents an approach to dynamically select between optimism and pessimism based on the task presented to the algorithm. I think it would help anchor the current work despite not being wholly aligned to the offline setting.", "other_strengths_and_weaknesses": "## Strengths\n\nThe conceptual framing of the work is well grounded in limitations of current offline RL methods. I’m not overly convinced of the originality of the ideas but it’s clear that the authors have attempted to unify regularization techniques and have thought through various challenges in doing so by adapting several approaches to combine in forming their proposed method.\n\nI also commend the authors for their efforts to develop a method that works in standard offline RL as well as in the offline-to-online setting.\n\nThe empirical results presented in Section 4 are compelling and cover a wide range of problem complexity and distributional settings (here, discussing the formation of the offline dataset). \n\n## Weaknesses\n\nIt’s unclear how the different objectives combine as the separate learnable parametrizations between the policy and state-adaptive regularization coefficient are not consistently parameterizied.\n\nThere are a lot of moving pieces and definition of objectives to establish the proposed state-adaptive regularization. As the paper is currently written, it is difficult to follow everything. There are a lot of “stream of consciousness” declarations which belies a poorly composed work. Specifically, so much of the paper has been talking about CQL up until late in Section 3.3 where suddenly “we utiliize the approaches from IQL” crops up. The paper would greatly benefit from a top down restructuring around what the proposed contributions are and how they’re acheived. The lack of clarify and construction of the paper also led me to lower my score of this paper.", "other_comments_or_suggestions": "It seems that Equation 5 is misreferenced in the paragraph before the equation is introduced? \n\nIt’s unclear where the quantities $\\mu$ and $\\sigma$ are drawn from in the definition of Equation 6. The term “trust region” is used but the specific computation of this distributional region is not specified.\n\nI think that Section E in the appendix should be referenced far more prominently in the paper. It isn’t super clear but it does help unify the various techniques used in the paper.\n\n## After rebuttal and reviewer discussion periods\n\nI apologize for the lack of engagement the authors rightfully deserved from this submission. I believe that the authors did a nice job responding to the various requests made in the collective reviews. As such, I feel that all reviewers agreed that this is a paper worthy of acceptance, I did not feel inclined to increase my score after the author's rebuttal and in respect to the other reviews (and accompanying rebuttals). I urge the authors to certainly include all of their promised changes in the event that the paper is formally published.", "questions_for_authors": "The analysis around Figure 1 is unclear. What determines the values of the histograms? What data is used? How early in the training is this analysis drawn from? I found the concept aroud Section 3.3 interesting but the presentation and writing is quite poor. Figure 1 should probably not be referenced until more detail is shared about what is being presented. (Porbably after Equation 10). Ultimately, there is not enough detail in place to describe what’s happening in Figure 1 to fully follow what is being presented and the ultimate value of the analysis.\n\nWhat dictates the choice of $G_T$ in Section 3.3? Is this task-dependent?\n\nUltimately Section 3.3 is extremely unclear… Is there a two-phase training paradigm? First for the high-value sub-dataset and then a more general training over all data?\n\nIs the $n$ in Equation 12 the same $n$ used to define the trust region?\n\nIs policy learning only done on the sub-datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "su4YyUFEaz", "reviewer": "Reviewer_578t", "summary": "This paper proposes selective state-adaptive regularization method for offline RL, addressing the limitations of fixed-strength regularization. It involves state-adaptive regularization coefficients through learning and regularization on a subset of high-quality data. Experiments on the D4RL benchmark show some performance improvements in both offline and offline-to-online scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review and positive recognition of our work. We are glad that you consider our work \"well-supported, simple yet effective, logically rigorous and coherently structured.\" We are glad to answer all your questions.\n\n**Q1:More comparative results** \n\n**A1:** We compare our approach with two strong baselines, MCQ and MISA. MCQ is a CQL variant achieving mild conservatism, while MISA unifies CQL and TD3+BC via mutual information. The results  below demonstrates our competitive performance.\n|Dataset |MCQ|MISA|TD3+BC(SA) |CQL(SA) \n|:---:|:---:|:---:|:---:|:---:|\n|ha-m|62.7|47.4|56.5|**63.9**|\n|ho-m|79.2|67.1|**101.6**|89.1|\n|wa-m|**90.5**|84.1|87.9|84.9|\n|ha-m-r|**56.6** |45.6|49.6|53.8|\n|ho-m-r|100.7|98.6|**101.6**|101.4|\n|wa-m-r|91.4| 86.2|93.5|**94.7**|\n|ha-m-e|88.2| 94.7|94.9|**102.1**|\n|ho-m-e|**110.6**|109.8|103.8|109.6|\n|wa-m-e|**114.0**|109.4|112.5|112.2|\n|ha-e|96.6|95.9|95.5|**105.9**|\n|ho-e|110.9|111.9|109.8|**111.4**|\n|wa-e|107.4|109.3|109.6|**110.2**|\n|total|1108.8|1060|1116.7|**1139.1**|\n\nSince we do not employ any specialized techniques in the online phase, we compare our approach with some other advanced unified methods across O2O phases, as shown below. Our method still leads in performance.\n|Dataset |SPOT|Cal-QL|TD3+BC(SA) |CQL(SA) \n|:---:|:---:|:---:|:---:|:---:|\n|ha-m|58.6|61.6|82.9|**95.3**|\n|ho-m|99.9|97.1|**103.5**|99.3|\n|wa-m|82.5|83.4|101.6|**105.9**|\n|ha-m-r|57.6|51.1|73.1|**79.4**|\n|ho-m-r|97.3|99.3|102.9|**103.1**|\n|wa-m-r|86.4|91.9|100.9|**116.3**|\n|ha-m-e|91.9| 95.9|98.5|**115.4**|\n|ho-m-e|106.5| **111.4**|111.2|109.5|\n|wa-m-e|110.6|110.9|115.7|**117.5**|\n|ha-e|94.1|97.0|102.5|**113.3**|\n|ho-e|111.8|**112.2**|112.0|110.8|\n|wa-e|109.9|109.5|**113.8**|112.6|\n|total|1108.8|1121.3|1218.6|**1278.4**|\n\n**Q2: Experiments on hyperparameter sensitivity** \n\n**A2:** We conduct hyperparameter sensitivity experiments on  $n_{end}$​, as shown below. The results demonstrate the robustness of our method, as the early stopping mechanism in the update of $n$ ensures that $n$ increases to an appropriate value and then stabilizes.\n|$n_{end}$ |1.0|2.0|3.0|4.0|5.0|\n :---:|:---:|:---:|:---:|:---:|:---:|\n|ho-m |82.6±26.7|79.4±25.6|89.1±9.7|90.6±9.7|84.1±10.5|\n|ho-m-r |101.7±1.4|101.3±1.8|101.4±2.1|100.0±3.1|97.1±9.4|\n\n**Q3:The meaning of $\\mu$ in Eq.(6)** \n\n**A3:** Apologies for the omission. $\\mu$ represents the mean of the stochastic policy (i.e., the learned policy), and $\\sigma$ denotes its standard deviation. \n\n**Q4:Explanation regarding threshold** \n\n**A4:** The threshold $n$ in Eq.(5) is initially set to a small value (1.0 in our experiments), which results in a large $C_n(s)$. At this stage, the loss value of Eq.(5) is negative, leading the regularization coefficient to increase and encouraging the learned policy to closely imitate dataset actions during the early learning phase. As $n$ gradually increases, the loss value of Eq.(5) eventually becomes positive, indicating that the learned policy has sufficiently approximated the high-value dataset actions in most states. In this trust region, certain state constraints can be relaxed while ensuring that a substantial portion of constraints remain enforced.\n\n**Q5:Fine-tuning coefficients online** \n\n**A5:** Since a single online step cannot directly determine action value, directly fine-tuning regularization online is difficult.\n\n**Q6:Computational cost** \n\n**A6:** Our method introduces a lightweight network with minimal overhead. Below is the training time on a 2080 GPU, excluding IQL-style pretraining, showing a slight increase in time cost.\n|Algo|TD3+BC|CQL|TD3+BC(SA) |CQL(SA)| \n|:---:|:---:|:---:|:---:|:---:|\n|Time Cost(h)|2.25|2.69|7.22|8.05|\n\n**Q6:Comparison with FamO2O** \n\n**A6:** Both methods adjust state-wise regularization, but key differences include:\n1. Regularization Parameterization: FamO2O employs a hierarchical architecture, using an intermediate variable to learn policies under different constraints. In contrast, our method directly parameterizes the regularization coefficients with a neural network, explicitly modeling constraint strength across different states.\n2. Policy Representation: FamO2O learns a family of policies, where the optimal policy is selected by maximizing the Q-value with respect to the intermediate variable. This requires passing through two networks to obtain the final policy. In contrast, our method updates the constraints based on the relationship between the learned policy and dataset actions, directly learning the final policy without additional selection steps. \n3. Interpretability: The intermediate variable in FamO2O lacks interpretability, meaning its value does not directly indicate constraint strength. In our approach, the parameterized regularization coefficients directly reflect the constraint strength, providing better interpretability.\n\nThank you for your insightful review again. We hope we have resolved your concerns. We are always willing to answer any of your further concerns.", "claims_and_evidence": "The claims are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed state-adaptive coefficient and selective regularization are simple yet effective. The use of D4RL benchmarks aligns standard settings in RL.", "theoretical_claims": "The overall derivation is logically rigorous and coherently structured, and the proof of Proposition 3.1 is theoretically correct.", "experimental_designs_or_analyses": "The experimental designs are clear and the analyses are complete. Through a variety of comparisons and ablation experiments, it demonstrates the performance of the proposed method in different scenarios. However, the experimental part lacks sensitivity analysis of hyperparameters.", "relation_to_broader_scientific_literature": "This paper addresses the limitations of prior fixed-strength regularization works. It includes the comparison results with recent state-adaptive approaches like FamO2O.", "essential_references_not_discussed": "Yes. The references contain the background required to understand the key contributions of the paper. No essential literature has been omitted in the paper citations, and relevant research findings/results/etc. have been cited and discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n1) The proposed selective state-adaptive regularization method is simple yet effective.\n2) Sufficient theoretical derivations are provided.\n\nWeaknesses:\n\n1) The comparative results are not sufficiently reliable as lacking the performance of some latest offline and offline-to online methods. There is no mention of hyperparameter sensitivity and setting experiment.\n2) Current explanations regarding threshold calculation are confusing and hard to follow, requiring further clarification.", "other_comments_or_suggestions": "There are problems with the expression, for example, \"we provide a unified framework to unified framework...\"", "questions_for_authors": "1) What does $\\mu$ represent in Eq.6?\n2) Why not fine-tune regularization coefficients during online training? Would adaptive updates improve stability?\n3) Does updating regularization coefficients introduce significant computational overhead?\n4) Could the authors compare the proposed method with FamO2O in detail, especially in terms of interpretability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After careful evaluation of the reviews, author rebuttals, and the paper itself, I recommend accepting this submission.  This paper presents a novel method for addressing a key challenge in offline reinforcement learning (RL): balancing between extrapolation errors and behavior cloning in value estimation through state-adaptive regularization. Rather than relying on a global, fixed regularization coefficient, the authors propose a state-adaptive mechanism that dynamically adjusts regularization strength at the state level, informed by the reliability of Bellman updates. Additionally, the paper introduces selective regularization over a subset of high-quality actions, mitigating the degradation caused by low-quality data. The method is unified across both value regularization (e.g., CQL) and explicit policy constraint approaches, and achieves significant improvements in both offline and offline-to-online settings on the D4RL benchmark. The inclusion of ablation studies, and comparisons with recent baselines like FamO2O further bolster the experimental robustness. 


There are two limitations the authors could further discuss or address in the final revision. 1. Limited theoretical guarantees: While the paper provides an intuitive and empirical justification for state-adaptive regularization, it lacks a formal theoretical analysis or convergence guarantees under the proposed method. 2. Evaluation beyond continuous control: The experiments are confined to continuous control benchmarks (MuJoCo, AntMaze); evaluating other domains (e.g., discrete control) would further validate generality and make the paper more impactful.
Decision: Accept (poster)