Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
Authors: Yixiu Mao
Yun Qu
Cheems Wang
Xiangyang Ji
Venue: NeurIPS 2025 spotlight ()
Keywords: offline reinforcement learning
Primary area: reinforcement_learning
TL;DR: This work proposes a flexible neighborhood constraint for offline RL that mitigates the over-conservatism of density and sample constraints, while approximating the least restrictive support constraint without behavior policy modeling.

Abstract: Offline reinforcement learning (RL) suffers from extrapolation errors induced by out-of-distribution (OOD) actions. To address this, offline RL algorithms typically impose constraints on action selection, which can be systematically categorized into density, support, and sample constraints. However, we show that each category has inherent limitations: density and sample constraints tend to be overly conservative in many scenarios, while the support constraint, though least restrictive, faces challenges in accurately modeling the behavior policy. To overcome these limitations, we propose a new neighborhood constraint that restricts action selection in the Bellman target to the union of neighborhoods of dataset actions. Theoretically, the constraint not only bounds extrapolation errors and distribution shift under certain conditions, but also approximates the support constraint without requiring behavior policy modeling. Moreover, it retains substantial flexibility and enables pointwise conservatism by adapting the neighborhood radius for each data point. In practice, we employ data quality as the adaptation criterion and design an adaptive neighborhood constraint. Building on an efficient bilevel optimization framework, we develop a simple yet effective algorithm, Adaptive Neighborhood-constrained Q learning (ANQ), to perform Q learning with target actions satisfying this constraint. Empirically, ANQ achieves state-of-the-art performance on standard offline RL benchmarks and exhibits strong robustness in scenarios with noisy or limited data.

Adaptive Neighborhood-Constrainee Q Learning for Offline Reinforcement Learning
Yixiu Mao', Yun Qu', Qi Wang', Xiangyang Ji Departmeet of Automation, Tsinghua University myx210mails tsinghua. edu. en, xyji@tsinghua edu. cn
Abstract
Offline reinforcement learning (RL) suffers from extrapolation errors induced by -iitt---ittiitttonn (OOD) actions. To address this, offline RL algorithms typically impose constraints on action selection, which can be systematically categorized into density, support, and sample constraints. However, show that each category has inherent limitations: density and sample constraints tend to be overly conserva- tive in many scenarios, while the support constraint, though least restrictive, faces challenges in accurately modeling the behavior policy. To overcome these limita- tions, we propose new neighborhood constraint that restricts action selection in the Bellman target to the union of neighborhoods of dataset actions. Theoretically, the constraint not only bounds extrapolation errors and distribution shift under certain conditions, but also approximates the support constraint without requiring behavior policy modeling. Moreover, it retains substantial flexibility and enables pointwise conservatism by adapting the neighborhood radius for each data point. In practice, we employ data quality as the adaptation criterion and design an adaptive neighborhood constraint. Building we develop a simple yet effective algorithm, Adaptive grighborhood-costtainng learning (ANQ). to perform Q learning with target actions satisfying this constraint. Empirically, ANQ achieves statc-of-the- performance on standard offline RL benchmarks and exhibits strong robustness in scenarios with noisy or limited data.
efficient bilevel optimization framework,
Q
1 Introduction
Reinforcement learning (RL) tackles sequential decision decision-making problems and has gained consid- erable attention recent years [58, 70, 67, 14]. Despite its promise, RL faces practical challenges, notably the high data collection costs [38] and exploration risks [24]. Offline RL offers compelling alternative by learning from a static dataset collected by behavior policy [43, 44]. enables the use of existing large-scale datasets [33, 52, 63] and reduces the dangers of unsafe exploration. However, it also introduces key challenge: evaluating outtt--distribution (OOD) actions leads to extrapolation errors [23], which further causes value overestimation and significant performance degradation [44].
To address this issue, offline RL. approaches typically impose constraints on the action selection process. A common strategy is to align the probability densities of the trained and behavior poli- cies [83, 21], enforcing a density constraint. This is usually achieved using divergence metrics such as reverse Kullback-Leible (KL) [83, 32], forward KL (i.e., behavior cloning) [21, 61], Fisher [39], or the implicit CQL divergence [42]. While straightforward, these methods can overly restrictive both theoretically and empirically [41], in cases where the overall quality of the behavior policy is low. To overcome this limitation, recent work has explored the most relaxed support constraint, which only keeps the selected actions within the support of the behavior policy [82, 53, 93]. Accom- plishing this generally necessitates high-fidelity estimation of the behavior policy through advanced generative modeling techniques [26, 82, 93, 9]. However, such modeling challenging due the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: A brief' summary of constraint types in offline RL research.
Constraint type
Description
Algorithms
Key characteristics
Enforce density proximity between the trained and behavior policies
BRAC [83],
Straightforward but heavily
Density
TD3BC [21],
limited by the overall quality
CQL [42]
of behavior policy
[QL[40],
Avoid extrapolation error but
Restrict action selection dataset actions
Sample
XQL[25].
lack action generalization
SQL[88]
beyond the dataset
BCQ[23]
Least restrictive but require accurate behavior
Restrict action selection to behavior policy's 's support
Support
BEAR[41]
SPOT [82]
policy modeling
Restrict action selection to certain neighborhoods of dataset actions
Flexible and approximate support constraint without
Neighborhood
ANQ (Ours)
behavior modeling
sigh-dimensional and multi-modal nature of real-world data [93], subjecting these methods to height- ened error susceptibility and increased computational overhead. Alternatively, the sample constraint has emerged, formulating the Bellman target exclusively using actions in the dataset [6, 40, 92, 88]. These methods are easy to implement and effectively avoid extrapolation errors [40]. However, their performance inherently limited by a lack of action generalization beyond the offline dataset, often resulting in overly conservative policies when near-optimal actions are rare in the dataset.
This work aims to address the over-conservatism of the density and sample constraints while avoiding complex behavior modeling required by the support constraint. To this end, we introduce new neigh- borhood constraint that restricts action selection in the Bellman target the union of neighborhoods of dataset actions. Theoretically, the constraint not only bounds extrapolation errors and distribution shift under certain conditions, but also approximates the least restrictive support constraint without behavior modeling. Moreover, it maintains high flexibility and can achieve pointwise conservatism by adapting the neighborhood radius for each data point. In light of real-world data patterns, we adopt data quality as the adaptation criterion and develop adaptive neighborhood constraint in practice. It assigns larger neighborhood radii to low-advantage dataset actions to promote a broader search, and smaller neighhorhood radii to high-advantage dataset actions to limit the overall extrapolation error.
To enforce the proposed constraint, we introduce an efficient bilevel optimization framework and, based on it, develop a simple yet effective algorithm, Adaptive weighborhooddccosttriipp Q learn- ing (ANQ), which performs Q learning with target actions constrained accordingly. Specifically, in the inner optimization, we maximize the Q function separately within each dataset action's 's neighborhood; in the outer optimization, we implicitly maximize the Q function over all available neighborhoods via expectile regression [40]. With the Q function being trained, the policy is independently extracted by weighted regression toward optimized actions within the neighborhoods, obtained in the inner maximization. Empirically, ANQ achieves mate-offtheearr performance on standard offline RL. bench- marks [20], including Gym locomotion tasks and challenging AntMaze tasks. Moreover, benefiting from the flexible constraint without behavior modeling errors, ANQ attains superior performance in both noisy and limited data scenarios compared to algorithms with other types of constraints. The code is available at https://github.com/ http:///ggthbbbbomm/mm/////////
2 Preliminaries
RI. In RL. the environment is typically modeled as a Markov Decision Process (MDP) M (S,A, P, 7, da), with state space S, action space A, transition dynamics P S x A + A(S), reward function R. : S - [0, Rmax], discount factor 7 E [0, 1). and initial state distribution do [73]. The agent seeks policy π S A(A) that maximizes the expected return:
oc x R(st. an)
TT(T) = Eso (r ( (())
(1)
1=
2

---

For given policy x, the state value function is defined as (*(s) = En E- " ae)lsu s). and the state-action value function is defined as (x, = E Et== 7 R(ss, 04)100 = 8, 0, = a].
Offline RL. In offline RI., the agent is able access fixed dataset D {(xi, ai (i))) == collected by some behavior policy Tg, and aims to learn an optimal policy without further data collection [43, 44]. Standard Q learning methods seek to learn the optimal Q function by minimizing:
(o = E1 [Qo((,, - R(s,a) a) - max Qu (s)
where Qo(s, a) denotes a parameterized function, and Q (s, a) represents a target Q function with parameters updated using Polyak averaging [58].
A central challenge in offline RL is the presence of out-of-distribution (OOD) actions that fall outside the support of the behavior policy. These OOD actions often lead to inaccurate Q value estimates due to extrapolation errors [23]. As result, maximizing the estimated Q functions tends to favor OOD
actions with overestimated values, resulting in significant performance degradation [44].
3 Adaptive Neighborhood-Constral
Q Learning for Offline RL
This section focuses on developing action-selection constraints to address the OOD issue in offline RL. First, we provide a systematic categorization of existing approaches and analyze the strengths and limitations of each category. To overcome the limitations, we propose a new neighborhood constraint, supported by theoretical analyses that elucidate its properties. Furthermore, we design an adaptive variant of this flexible constraint, achieving pointwise conservatism in practice. Finally, we develop a simple yet effective algorithm to facilitate Q learning and policy extraction under the constraint.
3.1 A Categorization and Analysis of Constraints in Offline RL
To address the OOD issue, offline RL algorithms impose various constraints to prevent either the learned policy (in actor-critic training) or the Bellman target (in Q learning) from selecting OOD actions. In this context, many approaches inherently align the probability density of the trained policy with that of the behavior policy, either explicitly, through divergence measures such as reverse KL [32, 83], forward KL (i.e., behavior cloning) [21, 61]. and Fisher divergence [39], or implicitly, via value penalties that reduce the Q values of trained policy's actions and while increasing those of dataset actions [42, 12]. We formalize this concept as the density constraint in Definition 1.
Definition 1 (Density constraint). The trained policy satisfies the density constraint D(x, R3) S C, where D represents a divergence measure between the trained policy a and the behavior policy Tg. e.g., KL, total variation (TV), or Fisher divergence.
While straightforward, this density constraint can be overly restrictive in many scenarios. Lemma 1 provides a theoretical upper bound on policy performance under various forms of density constraints.
Lemma (Performance bound under density constraints). If any of the conditions DKL (π||TT) < 2c, (πA(TT 2e, DTv(m, TB) S Ve holds, then the policy performance nj is bounded as follows:
2Rmax
n)x) S (((T)) + (1 (1- Vc.
Lemma demonstrates that policy performance under the density constraints is affected by the overall quality of the behavior policy. Consequently, even if the optimal behavior is present in the dataset, the learned policy can still remain highly suboptimal if the overall behavior policy is of low quality.
To address this limitation inherent in the density constraint, recent studies have explored the more relaxed support constraint, which only requires selected actions to be within the support of the behavior policy [82, 26, 53, 93]. Prior to the formal definition, we introduce a general loss function for constrained Q learning in Eq. (4), where C(s) denotes conditional set of actions for given state.
Le (0) = EL (Qo(s,a) R(s, a) - max Qo (0 a EC(('))
(4)
3

---

Definition 2 (Support constraint). The selected action in the Bellman target is restricted to the support of the behavior policy. which is defined as Csupp (s) := fa E A [T(aas) > e), where T is the behavior policy and is threshold that determines the support.
This support constraint is generally considered the least restrictive for offline RL [82], as the quality of actions outside the behavior policy's support cannot be reliably assessed. To enforce this constraint, existing approaches typically rely on behavior policy modeling, using techniques such as conditional variational autoencoders (CVAEs) [23, 41, 94, 82], autoregressive models [26], flow-GANS [93], and diffusion models [9]. Specifically, these methods either use pre-trained behavior density estimators to explicitly constrain the policy within the behavior support [41, 82, 93], or employ pre-trained behavior policy samplers to generate in-support actions and select the one with the highest Q value [23, 26, 94, 9]. However, their effectiveness is fundamentally limited by the accuracy of behavior policy modeling [26], which is well well-known to be challenging due to the high- siggh-imessional and multi-modal nature of worldw data [93]. Moreover, these methods incur extra computational costs due to behavior model training and, in some cases, extensive action sampling per state.
A parallel research direction has introduced the sample constraint, which constructs the Bellman target exclusively using actions in the dataset [40, 85, 92, 88), and derives policies via weighted behavior cloning [61, 81], thereby avoiding querying any out-of-dataset actions.
Definition 3 (Sample constraint). The selected action in the Bellman target is restricted to the sample set Csamp (a A (s,a) E D), consisting of actions in the dataset for a given state s E DD.
Sample constraint methods are compurationally efficient, easy to implement, and effective in avoiding extrapolation errors [40]. However, their performance is inherently constrained by the inability to generalize beyond the offline dataset. This over-conservatism becomes especially problematic when the dataset lacks coverage of near-optimal actions, common issue in environments with large or continuous action spaces, or when the dataset exhibits low quality or limited diversity. Moreover, to ensure computational stability, these methods often struggle adequately suppress the impact of suboptimal dataset actions [88], diminishing their efficacy when such actions dominate the dataset.
For extended discussions on related work, we refer the reader to Appendix A.
3.2 Neighborhood Constraint for Offline RL
This work aims to mitigate the over-conservatism inherent in the density and sample constraints, while circumventing the behavior modeling requirement posed by the support constraint. To this end, we introduce a flexible neighborhood constraint for offline RL, which restricts action selection the Bellman target to the union of neighborhoods of dataset actions on a given state.
Definition 4 (Neighborhood constraint). The selected action in the Bellman larget is restricted to the neighborhood set (s) == {ā E A |ā all S e, (s,a) D), which comprises actions located within the t-neighborhoods of all dataset actions on a given state 8 E D.
In contrast to the sample constraint, this neighborhood constraint offers greater freedom, as allows for seeking better actions beyond the dataset, within a flexible range. As shown in the following Theorem 1, the neighborhood constraint can serve as a viable approximation to the least restrictive support constraint, with the benefit of being achievable without behavior policy modeling. To establish the theorem, we introduce the standardness assumption commonly used in geometric measure theory [13, 8J, which ensures that the measure does not exhibit "holes" at small scales.
Assumption 1 (Standardnesss) Let S C Rd be the support of a probability distribution v, and B(x, r) be the closed ball of radius r centered x E Rd. There exist constants To > O and Co > 0 such that:
Vz E S, Vr S ro. r(B(z, r)) 2 20o Co.r". Td
(5)
Theorem 1 (Support approximation via neighborhoods). Let S C Rd be the compact support of distribution v, and let XI....,X, Xx, be independent and identically distributed samples from v. Define Un.e- U'_ e();,)) as the union of closed balls of radius e centered at the samples. Let (S, (S,e/2) denote the covering number of S, ie., the minimal number of e/2-balls required to cover S. Under the standardness Assumption with constants ro, Co 0, for any 5 (0, 1) and 2ro, if
n Σ cu(e/2ja (log (S, e/2) + log (1/8))
(6)
4

---

then with probability at least 1 1- 8, the Hausdorff distance between S and Un.e satisfies
(H Un.e) == max
sup inf d(z, u), sup inf d(z, u) wES ueUn..
Se.
ueUn, #ES
Since both the support and neighborhood constraint sets are defined over actions conditioned on a given state, Theorem I analyzes their relationship at fixed state, focusing their difference in the action space. In Theorem 1, V represents the behavior policy distribution at a state, and S is defined as its support. X1, X1...... Xn are i.i.d. samples from v, corresponding to dataset actions at that state. This theorem ensures that the union of sample neighborhoods Un,c approximates the support within controlled Hausdorff distance, capturing the trade-off between sample size n, neighhorhood radius c, and the geometric complexity of support 5 (as reflected by (SS, c/ e/2)). Note that this Hausdorff distance is sensitive outliers [29], making it well-suited for evaluating approximation quality in setting, where outlier actions can significantly affect Q function optimization.
In the following, we further investigate several properties of the proposed neighborhood constraint the context of controlling extrapolation and distribution shift. The definition of this constraint closely related to the concept of extrapolation, and Lemma 2 provides a theoretical characterization of the extrapolation behavior of deep Q functions under this constraint.
Lemma 2 (Extrapolation behavior). Under the neural tangent kernel (NTK) regime [30]. for any in-saplle state-action puir (8. a) E 'D and in-neighborhood state-action puir (s. () such that ||ā all S 6, the value difference of the deep Q function can bounded as:
[|Qa ū) Qo(s, a)ll S Clymin(|ls @ all, ||8 @ all) - 26);
(8)
where @ denotes the vector concatenation operation, and C' is a finite constant.
Lemma 2 is direct corollary of Theorem in [45], specialized to the case of action extrapolation. It demonstrates that, for any unseen action à, its Q value (s ā) can be effectively controlled by a dataset action's 's Q value Qa(s, a) and the distance ||ä al. Specifically, a smaller neighborhood radius yields tighter control over the output of deep functions.
Furthermore, Proposition demonstrates that, under mild continuity conditions on the transition dynamics [15, 87]. the neighborhood constraint also helps to bound the degree of distribution shift.
Proposition 1 (Distribution shift). Let nı be a deterministic policy that satisfies the neighborhood constraint with threshold Assume that the transition dynamics P is Ky-Lipschit continuous: Vs E S, Yai, a2 E A. [|''ssss, a1) P(s' (sss|s,,oo a2)ll |I I Kplla1 azll. Then, there exists a policy T2
satisfying the sample constraint such that:
DrV (dt) ())dd)))))) < 7KpE
2(1 -7)²
where d= (s)
(1 Ens E, [
s|] is the state occupancy induced by policy
Adaptive neighborhoods. The neighborhood constraint highly flexible and can, in practice, achieve pointwise conservatism by adapting the neighborhood radius for each data point, enabling the design of an adaptive neighborhood constraint. Considering real-world data patterns, expert data typically clusters within a narrow distribution [3, 18], necessitating tighter constraints to mitigate extrapolation errors, while suboptimal data tends to be more dispersed and thus benefits from looser constraints that facilitate policy improvement. Inspired by this idea, we propose a concrete instantia- tion of adaptive neighborhoods Definition where par-sample radius is set as exp( (aaA(s, a)).
Definition 5 (Adaptive neighborhood constraint). The selected action in the Bellman target is re- stricted to the adaptive neighborhood sei CAN (s) == {ā E A ||ā -a]l S e eexp(-aA(s a)), (s, a) E D}, where A denotes the advantage function and an inverse temperature parameter that modu- lates the sensitivity of the neighborhood radius to advantage values.
This adaptive neighborhood constraint assigns larger neighborhood radii to dataset actions with low advantage, thereby promoting broader search over the action space and further mitigating the impact of low-quality data. Conversely, dataset actions with high advantage are assigned smaller neighborhood radii to more effectively reduce the overall extrapolation error. In practice, advantage
5

---

estimation errors are typically not concern for two reasons: (1) In-distribution estimation: the advantage is computed only on dataset points (s, a) E D, where estimates are relatively reliable; (2) Qualitative use: the purpose of using advantage is to distinguish actions qualitatively, and the exponential form merely a soft heuristic to bias the radius, without requiring precise values.
3.3 Adaptive Slighboorhood-Cnstrriiine Q Learning
In the following, we develop an efficient bilevel optimization framework to achieve Q learning under the adaptive neighborhood constraint. Specifically, we aim minimize the following Q learning loss:
[ (Ro(s..)- R(ss max
(0)() 27
(0) = )
(10)
d'ECAN(5")
Bilevel optimization. The primary challenge in constrained Q learning lies in enforcing rex.)) in the Bellman target. While the support constraint CSupp typically necessitates accurate modeling of the behavior policy, we demonstrate that the adaptive neighborhood constraint CAN can be effectively enforced by decomposing the objective into bilevel optimization structure:
Q(s,a), E = max Q(s, a + Ssa), Vs E D aED(s)
maa (ECAN(()
s.t. Sen
argmax [óllSeexp(- (((sa))
Q(s,a+ v(s,a) ED (11)
where we use (ss) to denote the empirical action set observed in the dataset for given state D.
The inner maximization in Eq. (1I) optimizes the Q function separately within each dataset action's neighborhood. To this end, we introduce an auxiliary policy Pww that takes state-action pairs (s, a) from the dataset as input and outputs action variations 5. This formulation enables straightforward enforcement of the adaptive neighborhood constraint by restricting the norm of pw (s, a) to stay within the bound e exp(-axA(x, a)). Practically, we multiply both sides of the constraint inequality {))l (8, a)ll < e. Conse- by exp(oxA(ss a)) to maintain a constant constraint threshold: exp(axA(s; (x, a)
quently, we optimize the Q function with respect to fuu to seek the optimal action within the adaptive neighborhood of each dataset action, according to the following objective:
max Qo(s, a + (s, s.t. exp(aA(s, (s, a)ll S E Mw
(12)
We reformulate the constrained optimization problem into an unconstrained one using Lagrange multiplier À E +R+ In addition, we introduce a state value function (s), whose training objective will be specified later, and use the difference Qø -V-VV V to compute the advantage function. Accordingly, we optimize the following objective for the inner maximization:
max E (s,a)D [Qo(s, + (s) Acxx(a((Q Hw
(s)
(13)
The outer maximization in Eq. (11) searches over all dataset actions on given state and seeks the one whose corresponding neighborhood yields the highest Q value. To achieve this objective, we first sample state-action pairs from the dataset and refine the actions by adding the outputs of the trained auxiliary policy, thereby simulating the sampling of the optimized actions across the neighborhoods. We then employ expectile regression [40] to implicitly maximize the Q function over these optimized actions. Specifically, we fit a V function with the following asymmetric squared error loss, treating the Q values of the optimized actions as regression targets:
min E. Ve (s,0)~D
[Tg (Qur (8, a + (s, - V (*)),,
(14)
where L5(x) = - 1(x < 0) (0, 1), Qo and pwr are the target Q function and target auxiliary policy, whose parameters are updated via Polyak averaging [58].
For ~ 1, (s) captures the maximum Q value within the adaptive neighborhood set CAN (s). By substituting max Qo (s', a') in Eq. (10) with Vsp(s), adaptive neighborhoood-consrrraeed
Q learning is achieved based on the following loss:
min (, (Qo(s;a) a) - (Vvo
(15)

---

A radius-agnostic framework. Although our Q learning algorithm is presented specifically for the adaptive neighborhood in Definition 5, i.e., using the per-sample radius e eexp(-aA(s, a)), the overall framework is general and can accommodate arbitrary radius schemes. Specifically, one can simply replace e eexp(-aAfe (s, a)) Definition 5 (and Eq. (11) with of a) to define a generic per-sample neighborhood radius, where f S x A - I is an arbitrary function that modulates the radius. Correspondingly, in Eq. (12), exp(a a)) becomes 1/ fs, a), and Eq. (13) becomes:
max E B(sa)) a a (s. a)) (w o))l/f(f
(16)
With all other equations unchanged, the resulting algorithm supports arbitrary neighborhood schemes.
3.4 Policy Extraction via Weighted Regression Toward Optimized Actions
While our algorithm enables Q learning under the adaptive neighborhood constraint, it does not explicitly derive the corresponding policy, thereby requiring separate policy extraction step. To this end, we employ the weighed behavior cloning method [17. 11] and, rather than imitating the actions in the dataset, we instead imitate the actions that have been refined through the auxiliary policy, which represents the optimal actions within the adaptive neighborhoods. Moreover, we set the weights as the exponentiated advantage function [79, 61, 60, 81]. Consequently, the final policy T S + A is extracted according the following loss:
min (s,a) exp(3(Qer) (s, a + pw (s, - Vv((s)))lla + (s, (
(17)
where B is an inverse temperature and Qa V computes the advantage function.
Remark. This policy extraction step, which does not interfere with the Q learn-
Algorithm ANQ
ing process described in Section 3.3, also constitutes a key distinction from exist-
1: Initialize policy Na. auxiliary policy Huss tar-
get auxiliary policy pw', Q-network Qa, tar-
ing regression-basse
policy learning objec-
get Q-network Qur, and V-network V.-
tives, such as those employed in AWR [61].
2: for each gradient step do
AWAC [60], CRR [81], 10% BC [10],
3: Update vi by minimizing Eq. (14)
IQL [40], and SQL [88]. all of which per- form weighted regression toward the dataset actions. In contrast, our policy learning objec- tive performs weighted regression toward the optimized actions within the adaptive neigh- borhoods. This enables the trained policy to select actions superior to those in the dataset, while also significantly mitigating the adverse effects of suboptimal dataset actions.
4: Update 0 by minimizing Eq. (15)
5:
Update by maximizing Eq. (13)
6:
Update & by maximizing Eq. (17)
7: Update target networks: 0' + (1 E)0' +
EA, w' + (1 {)w/ + Ew
8: end for
Integrating all components, we present our final algorithm in Algorithm 1.
Experiments
We conduct experiments to evaluate the performance and properties of the proposed approach ANQ. Experimental details and extended results are provided in Appendices C and D, respectively.
4.1 Benchmark Results
Tasks. We assess ANQ on two distinct task suites from D4RL [20]: the Gym-MuJoCo locomotion domains and the challenging AntMaze domains. The AntMaze tasks involve sparse rewards and require the ant agent to combine segments of suboptimal trajectories to reach the maze's goal.
Baselines. Our offline RL baselines span various constraint categories. For density constraints, we compare to TD3BC [21], CQL [42], and AWAC [60], where CQL and AWAC essentially enforce a density constraint as analyzed in Theorem 3.5 of [42] and Section 3 of [53], respectively. For support constraints, we include BCQ [23], BEAR [41], and SPOT [82]. For sample constraints, we compare against OneStep RL [6] and IQL [40]. We also include the sequence modelln approach DT [10].
7

---

Table 2: Averaged normalized scores on Gym locomotion and Antmaze tasks over five random seeds. == medium, m-r medium-replay, m-e = medium- expert, e expert, = random; = umaze, u-d umaze-diverse, m-p n-p- medium- play, m-d medium-diverse, 1-p- large-play, I-d = large-diverse.
Dataset-v2
BCQ
BEAR
DT
AWAC
OueStep
TD3BC
CQL
IQL
SPOT
ANQ (Ours)
halfcheetah-m
46.6
43.0
42.6
47.9
50.4
48.3
47.0
47.4
58.4
6.
hopper-m
59.4
67.6
59.8
87.5
59.3
53.0
66.2
86.0
100.9:10.6
walker2d-m
71.8
-0.2
74.0
83.1
84.8
83.7
73.3
78.3
86.4
12.9
halfcheetah-m--r
42.2
36.3
36.6
44.8
42.7
44.6
45.5
44.2
52.2
55,5:114
hopper-m-r
60.9
52.2
82.7
69.8
98.5
60.9
88.7
94.7
100.2
101.5+2.77
walker2d-m-r
57.0
7.0
66.6
78.1
61.7
81.8
81.8
73.8
91.6
92.7++.88
halfcheeiah-m--e
95.4
46.0
86.8
64.9
75.1
90.7
75.6
86.7
86.9
94.2
hopper-m-e
106.9
50.6
107.6
100.1
108.6
98.0
105.6
91.5
99.3
107.0:44,9
walker2d-m-e
107.7
22.1
108.1
110.0
111.3
110.1
107.9
109.6
112.0
111.7:10.2
halfchectah-e
89.9
92.7
87.7
81.7
88.2
96.7
96.3
95.0
94.8
95.9++04
hopper-e
109.0
54.6
94.2
109.5
106.9
107.8
96.5
109.4
111.0
111.4+2.5
walker2d-e
106.3
106.6
108.3
110.7
110.2
109,9
111.8:1011
halfcheetah-r
2.3
11.0
17.5
13.1
25.4
24.9-11.0
hopper-r
7.8
3.9
5.4
9.2
5.6
8.5
7.9
7.9
23.4
31.1-1022
walker2d-r
4.9
12.8
2.2
0.2
6.9
1.6
5.1
5.4
2.4
11.2449.5
locomotion total
968.0
581.7
972.6
975.3
1041.2
1013.2
1010.2
1033.1
1139,9
1194.5
antmaze-u
78.9
73.0
54.2
80.0
54.0
73.0
82.6
89.6
93.5
96.
antmaze-u-d
55.0
61.0
41.2
$2.0
57.8
47.0
10.2
65.6
40.7
80.2.1188
antmaze-m-p
0.0
0.0
0.0
0.0
59.0
76.4
74.7
76.2+33.3
antmaze-m-d
0.2
0.2
46.6
72.8
79.1
7..2+6.1
antmaze-I-p
0.0
0.0
0.0
0.0
16.4
42,0
35.3
56.2.14.9
antmaze-l-d
2.2
0.0
0.0
0.0
0.2
0.0
3.2
46.0
36.3
55.8:1..0
antmaze total
142.8
142.0
95.4
132.2
112.6
120.2
218.0
392.4
359.6
441.6
Comparisons. Aggregated results reported in Table 2. On the Gym locomotion tasks, ANQ outperforms existing methods on most tasks and achieves the highest overall score. the challenging AntMaze tasks, ANQ surpasses the baselines by considerable margin, particularly in the most complex large maze settings. The learning curves are provided in Appendix D.5. We also extend our evaluation in Appendix D.2 by comparing ANQ with additional recent SOTA algorithms.
Runtime. We test the runtime of ANQ and some baseline methods GeForce RTX 3090. As shown in Appendix D.I, ANQ is among the fastest tier of offline RL algorithms, on par with efficient baselines such as AWAC, IQL, and TD3BC, with detailed analysis provided in the same section.
4.2 Noisy Data Results
We examine the robustness of various constraint types under noisy data conditions. Specifically, we construct noisy datasets by mixing the random and expert datasets at varying ratios, thereby simulating real-world scenarios such imperfect demonstrations in robotics or suboptimal data collection autonomous systems. We then evaluate performance of representative algorithms, including CQL. (density), IQL. (sample), SPOT (support), and ANQ (neighborhood).
As presented Figure (a), ANQ generally outperforms the other algorithms across expert ratios, and its performance advantage becomes more pronounced as the proportion of expert data decreases. As analyzed in Section 3.1, the density constraint is sensitive to the overall quality of the behavior policy, which tends to be low in noisy datasets, while the support constraint often struggles with modeling the moltil behavior policy distribution inherent in such datasets. contrast, the adaptive neighborhood constraint employed by ANQ exhibits greater robustness to noisy data. Moreover, we evaluate ANQ on such noisy datasets with varying inverse temperature a that controls the adaptiveness of neighborhood radius. The results in Figure ((b) demonstrate that, compared with the uniform neighborhood constraint (a = 0), the adaptive neighborhood constraint further mitigates the adverse effects of low-quality data in the datasets and exhibits greater robustness to such data.

---

(Neighborhood)
SPOT (Support)
(Sample)
CQL (Density)
+-
Hopper
Walker2d
Halfcheetah
ANO
120
120
100
Norallzzd Return
100-
100
80-
Normalized Retum
s0-
60
60-
60
40
60
20-
20
20
40
1.0
0.6 0.2
0.6
0.2
0.8
0.4
0.0
0.6
0.4
Expert Ratio
Expert Ratio
Export Ratio
Expert Ratio
(a) Evaluation algorithms with various constraint types on noisy datasets
(b) ANQ with varying a
Figure 1: (a) Evaluation on noisy datasets over five random seeds. (b) Evaluation of ANQ on noisy datasets with varying inverse temperature a that determines the adaptiveness of neighborhood radius.
ANQ (Neighborhood)
SPOT (Support)
IQL (Sample)
A-2 - A=5 + A-10
Attiazz-Mediuu-Play
Antmazz--agge-lay
Autmaze-I Lagge-Dvvese
ANQ Antmazo-M-P
Normalized Retur
80
60
Normalized Retum
%0
50
50
60
40
20
20
40
10
10
0.0
0.4
0.0
0.2
0.5
0.2
0.8
Discard Ratio
Diseard Ratio
Discard Ratio
Discard Ratio
(a) Evaluation algorithms with various constraint types on reduced datasets
(b) ANQ with varying
Figure 2: (a) Evaluation on reduced datasets over five random seeds. (b) Evaluation of ANQ on
reduced datasets with varying Lagrange multiplier À that controls the overall radius of neighborhoods.
4.3 Limited Data Results
We also investigate the robustness of these constraint types limited data settings. To this end, we create reduced datasets by randomly discarding some portion of transitions from the AntMaze datasets. This setup mimics practical scenarios in which data is rare or partially missing, such as in healthcare applications. Again, evaluate the performance of IQL, SPOT, and ANQ, omitting CQL due to its consistently inferior performance on Antmaze tasks as reported in Table 2.
As shown in Figure 2(a). ANQ demonstrates superior performance across nearly all discard ratios, with the performance gap widening as the amount of available data decreases. In such limited data settings, support constraint methods face even more difficulties in modeling the behavior policy due to sample scarcity, whereas sample constraint methods risk being overly conservative because of reduced coverage of near-optimal actions. In contrast, ANQ bypasses the need to model the behavior policy and leverages generalization to attain superior performance beyond the offline dataset. Furthermore, we evaluate ANQ reduced datasets with varying Lagrange multiplier X, which is inversely proportional to the overall radius of the neighborhoods. The results in Figure 1(b) show that an appropriately large neighborhood is crucial for achieving good performance in such limited data scenarios, further showcasing the benefit and flcxibility of ANQ over sample constraint methods.
4.4 Ablation Study
Lagrange multiplier X. The Lagrange multiplier À controls the overall neighborhood radius in ANQ. We vary and present the learned values and performance across various tasks in Figure 3. As À decreases from sufficiently large value, ANQ enables larger neighborhoods, resulting in higher and possibly divergent Q values, and performance also exhibits a rise-then-fall trend. Note that ANQ with X = 0 and X = approximately corresponds to Q learning without any constraint and with the sample constraint, respectively. Therefore, the results provide evidence that ANQ not only effectively suppresses extrapolation error, but also mitigates the over-conservatism of the sample constraint.

---

halEchhcttab-meddun
halfcheetahh halfchorrab-meccfrm-cc medium-replaaa
balfchectah--zzdom
hopper -eedium
Nomalned ketum
16
362
Nomalies Ketim
280
Komaliest Ketm
Nomslies
Heo
Sabie
8 5
Retus
viloo
Reuu
Retua
Qu
Reau
00EE5O0 03
10 Lagrange multiplier
00060F0T Lagrange multiplier
15
1o 36
60L6350T Logrange multiplier
50 10
L0 30 75 25 Lagrange multiplier
Figure 3: Performance and Q values of ANQ with varying Lagrange multiplier À over five random seeds. The crosses X mean that the value functions diverge some seeds. As decreases, ANQ enables larger overall neighborhood radii, resulting in higher and probably divergent learned Q values. moderate À (neighborhood constraint) is crucial for achieving superior performance.
hulfchertah mediium
hulifcheciah mediumm ruplay
hulfchettah-mmdumm
hupper medium
Narraiaa Itrum
12 12
Narraa bieed Netem
200
Nomratee Nerm
480
135
625
Nomzlied
12
8
Ressan
L0 10 Renmn
Renmn
Renmn
Quata:
05 Invense lomperature
01 S.
00
L0
20
Inverse lempurature
Invone lempurature
Inverse lemperature O
Figure 4: Performance and Q values ANQ with varying inverse temperature a over five random seeds. An appropriately large (adaptive neighborhoods) yields enhanced performance.
Inverse temperature a. The inverse temperature a determines how the neighborhood radius adapts to the action advantage, where = 0 corresponds to fixed neighborhood radius. The results in Figure demonstrate that an appropriately large a leads to enhanced performance, validating our design of advantage-based adaptive neighborhoods. However, excessively large (a 20) may degrade performance, likely due to the increased variance of the learning objective.
5 Conclusion and Limitations
This work focuses on developing To overcome the identified limitations of existing approaches, we propose the flexible neighborhood constraint and the corresponding algorithm ANQ, which mitigates the over-conservatism inherent in the density and sample constraints, and approximates the least restrictive support constraint without challenging behavior modeling. Empirical results demonstrate that ANQ achieves SOTA performance on standard offline RI. benchmarks and exhibits enhanced robustness noisy limited data.
constraints to address the OOD issue in offline RL.
At the algorithmic level, this work develops a general framework for achieving pointwise conservatism by adapting the neighborhood radius for each data point. In particular, the practical algorithm ANQ represents one instantiation of adaptive neighborhoods, using data point quality as the adaptation criterion. However, this design is not necessarily optimal; incorporating additional information, such as uncertainty quantification, could potentially lead to more effective neighborhood construction.
Acknowledgment
We thank the anonymous reviewers for feedback on an early version this paper. This work was supported by the National Key R&D Program of China under Grant 2018AAA010200..
References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base offline reinforcement learning with diversified q-ensemble. Advances neural information processing systems, 34:7436-7441 2021.
10

---

[3] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-445 2009.
[4] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparaneetrrized two-layer neural networks. International Conference Machine Learning, pages 322-332. PMLR, 2019.
[5] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi- Zhi-Hong Deng, Animesh Garg. Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty- offline reinforcement learning. In International Conference on Learning Representations, 2022. URL htttps::/opeerevviiii
het/forummid-iee id=Ydcssisss 1d=Y4c1133Hqq.
[6] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933-4944 2021.
[7] Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural semporal-differene learning converges to global optima. Advances Neural Information Processing Systems, 32, 2019.
[8] Frédéric Chazal, Mare Glisse, Catherine Labruère, and Bertrand Michel. Convergence rates for persistence diagram estimation topological data analysis. In International Conference on Machine Learning, pages 163-171. PMLR, 2014.
[9] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fielliyy generative behavior modeling. In The Eleventh International Con- ference on Learning Representations, 2023. URL https:://po// http::://ppeeeviee... formm?id- 42zz3qa2kpy.
[10] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14:45585-,,,,7, 2021.
[11] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353-18363, 2020.
[12] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3872-3878. PMLR, 2022.
[13] Antonio Cuevas. Set estimation: Another bridge between statistics and geometry. Bol. Estad. Investig, Oper, 25(2):71-88 2009.
[14] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414-419 2022.
[15] Francois Dufour and Tomas Prieto-Rumeau. Finite linear programming approximations of constrained discounted markov decision processes. SIAM Journal on Control and Optimization, 51(2):1298-133 1324, 2013.
[16] Francois Dufour and Tomas sion processes using empirical distributions and concentration incqualities. Stochastics An International Journal of Probability and Stochastic Processes, 87(2):273-307, 2015.
Approximation of average cost markov deci-
[17] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751 2021.
[18] Anders Ericsson, Ralf T Krampe, and Clemens Tesch-Römer. The role of deliberate practice in the acquisition of expert performance. Psychological review, 100(3):363, 1993.
11

---

[19] Jianqing Fan, Zhaoran Wang. Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learming. In Learning for dynamics and control. pages 486-489. PMLR, 2020.
[20] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint urXiv:2004.02/19 2020.
[21] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances neural information processing systems, 34:20132-20145, 2021.
[22] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International conference on machine leurning, pages 1587-1596. PMLR, 2018.
[23] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[24] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[25] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations. 2023. URL https: / //p///ppeeeeeeeeeeeeeeeeeeeeeeee net seee//oouum/dddd..ddddtttt.
[26] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaqq Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682-3691. PMLR, 2021.
[27] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
[28] Shengchao Hu, Ziqing Fan, Chaoqin Huang. Li Shen, Ya Zhang, Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement learning. In International Conference on Machine Learning, pages 19165-19181. PMLR, 2024.
[29] Daniel P Huttenlocher, Gregory A. Klanderman, and William Rucklidge. Comparing images using the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence, 150))885--66, 1993.
[30] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[31] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-basee policy optimization. Advances in neural information processing systems, 32, 2019.
[32] Natasha Jaques. Asma Ghandeharioun. Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.000 197,000977000466 2019.
[33] Alistair EW Johnson, Tom Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[34] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model- based reinforcement learning for atari. arXiv preprint arXiv: arXiv:/903.000 1903.00374, 2019.
[35] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference Machine Learning. Citeseer, 2002.
12

---

[36] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 13:21810-21822 2020.
[37] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[38] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 32(11):1238-1277, 2013.
[39] lly Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5783. PMLR, 2021.
[40] Uya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https: /oopenreview
net/forummid d
[41] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping emor reduction. Advances in Neural Information Processing Systems, 32, 2019.
[42] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1 1179-1 1199-1191.11 1191, 2020.
[43] Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning, Reinforce- ment learning: State-of-the-art, pages 45-73, 2012.
[44] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0/644 2020.
[45] Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https: //openreiiew weet/forrmmdd:11071700 cuuh.
[46] Boyi Liu, Qi Cai, Zhuoran Yang. and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy, Advances in neural information processing systems, 32, 2019.
[47] Xiaoteng Ma, Yigin Yang, Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint arXiv:21 2021.
[48] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34:19235-19247, 2021.
[49] Yi Ma. Hongyao Tang. Dong Li, and Zhaopeng Meng. Reining generalization in offline rein- forcement learning via representation distinction. Advances in Neural Information Processing Systems, 36:40773-40785, 2023.
[50] Yi Ma. Jianye Hao. Xiaohan Hu, Yan Zheng, and Chenjun Xiao. Iteratively refined behavior regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 57:562115-56253 2024.
[51] Yi Ma, Jianye Hao, Hebin Liang, and Chenjun Xiao. Rethinking decision transformer via hierarchical reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, pages 33730-33744 2024.
[52] Will Maddern, Geoffrey Pascoe, Chris I Linegar, and Paul Newman, 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3-15, 2017. 2017.
13

---

[53] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pages 23829-23851 PMLR, 2023.
[54] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:40587 40609, 2023.
[55] Yixiu Mao, Qi Wang, Chen Chen, Yun Qu, and Xiangyang Ji. Offline reinforcement learn- ing with ood state correction and ood action suppression. Advances in Neural Information Processing Systems, 37:93568- 77:93568-93601, 2024.
[56] Yixiu Mao, Qi Wang, Yun Qu Yuhang Jiang, and Xiangyang Ji. Doubly mild generalization for offline reinforcement learning. Advances in Neural Information Processing Systems, 37: 11436-51475 2024.
[57] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efficient reinforcement learning via model-based offline optimization. In In- termational Conference on Learning Representations, 2021. URL https:/ sst/forummid-3hhllqph
[58] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 518(7540):529-533 2015.
(59] Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1))1-118, 2023.
[60] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
[61] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- regression. Simple and scalable off off-policy reinforcement learning. arXiv preprint ar11:119/00001177 2019.
[62] Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. survey of temporal credit assignment in deep reinforcement learning. arXiv preprint arXiv:2312.01072, 2023.
[63] Yun Qu, Boyuan Wang, Jianzhun Shao, Yuhang Jiang, Chen Chen, Zhenbin Ye, Line Liu, Junfeng Yang. Lin Lai, Hongyang Qin. et al. Hokoff: Real game dataset from honor of kings and its offline reinforcement learning benchmarks. In Thirth-seventh Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, 2023.
[64] Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao, Cheems Wang, Chang Liu, and Xiangyang Ji. Latent reward: Lim-empowered credit assignment episodic reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 20095-20103, 2025.
[65] Yun Qu, Cheems Wang, Yixiu Mao, Yiqin Lv, and Xiangyang Ji. Fast and robust: Task sampling with posterior and diversity synergies for adaptive decision-makers in randomized environments. In Forty-second International Conference Machine Learning, 2025. URL https://openreviiss wtwp://rwwwiiii.eeeoooooonnaaaae t
[66] Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning, pages 28701-28717. PMLR, 2023.
[67] Julian Schrittwisser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-600) 2020.
14

---

[68] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[69] Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfactual conservative q learning for offline multi-agent reinforcement learning. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https: http:///ppeeeeeeeeeeeeee forum?id=622n 6m2224
[70] David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
[71] Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning. In International Conference on Machine Learning, pages 33177-33194 PMLR, 2023.
[72] Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 2(4):160-163, 1991.
[73] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[74] Abdel Aziz Taha and Allan Hanbury. An efficient algorithm for calculating the exact hausdorff distance. IEEE transactions on pattern analysis and machine intelligence, 37(11)21133-2133, 2015.
[75] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
[76] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[77] Qi Wang and Herke Van Hoof. Learning expressive meta-reprresentations with mixture of expert neural processes. Advances in neural information processing systems, 35:26242-26255, 2022.
[78] Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference on Machine Learning, pages 23055-23077. PMLR, 2022.
[79] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018
[80] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL /tttps://operreeeeee www.aarwwwww..eeeooore/oeeeaeeee hett/forrrr/dd---WBD-----------
net
[81] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[82] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: http:////pppevveeee net fott/ffrummidd (ct.:rrmdddxco0o0
[83] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint
2019.
[84] Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in neural information processing systems, 16:12222--8850 2023.
15

---

[85] Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in ia-maele soft- max for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URI. https:://////ttt /ttp////ppeeeeeeeee/ee/eeeeeeeee net./ iiterrmmmmiiiidde -
[86] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-6694 2021.
[87] Huaqing Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang. Deterministic policy gradient: Convergence analysis. In Uncertainty in Artificial Intelligence, pages 2159-2169. PMLR, 2022.
[88] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization, In The Eleventh International Conference on Learning Representations, 2023.
URL https: htttpp:///peereviese net/ foofororuuue forumm Wee
[89] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline reinforeement learning via conservative smoothing. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: /ttp////ppeeeeeeeee///eee wwt//net/oouuee http//formmddddoomm _a_J_a___________ KE
[90] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14144 2020.
[91] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances neural information processing systems, 34:28954 28967, 2021.
[92] Hongchang Zhang, Yixiu Mao, Boyuan Wang. Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In The Eleventh International Con- ference Learning Representations, 2023. URL https://openeeviee..et// wtw.s//wwnee//ooooiiiie.e dfDvoWU853R.
[93] Jing Zhang, Chi Zhang, Wenjia Wang. and Bingyi Jing. Constrained policy optimization with explicit behavior density for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[94] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719-1735. PMLR, 2021.
[95] Heming Zou, Yunliang Zang, and Xiangyang Ji. Structural features of the fly olfactory circuit mitigate the stabilit--plasticiy dilemma in continual learning. arXiv preprint arXiv:2502.0142) 2025.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 5,
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
Please refer to Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: Please refer to the code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt S ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results in the paper accompanied by standard deviations across multiple seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix D.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://meee //tpp/////////p////eeeee cc cc/public//thhicciuiiee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code is well documented and anonymized.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https http://ies...//eeeeeeeeee httpe//ris..e/eeoeeeeeeeeeeeeeee c for what should or should be described.
23

---

A Extended Related Work
Model-free offline RL. Offline RL aims to learn policies from a fixed dataset without any further interaction with the environment [43, 44]. Standard off-policy methods often struggle in this setting due to extrapolation errors induced by OOD actions [23]. To address this issue, a variety of algorithms have been developed, broadly falling into model-free and model-based categories. Within the model- free family, value regularization methods promote conservative value estimation, either by explicitly penalizing overestimated Q values [42, 39, 48, 86, 12, 69, 54], or by employing ensembles to capture epistemic uncertainty [2, 5, 89]. Policy constraint approaches, by contrast, aim to keep the learned policy close to the behavior policy, accomplished either explicitly through divergence penalties [83, 41, 32, 21, 82], implicitly via weighted behavior cloning [11, 61, 60, 81, 53), or through carefully designed policy parameterizations [23, 26, 94]. More recently. these methods have increasingly shifted toward learning the optimal policy within the support of the behavior policy, offering theoretical guarantees and reduced sensitivity to the overall quality of datasets [82, 53, 54, 50]. Another direction emphasizes in-sample learning, which avoids estimating values of unobserved actions by constructing Bellman targets solely from dataset samples [6. 47, 40, 85, 88, 25, 92]. For example, OneStep RL [6] employs SARSA-style value updates [73] and performs single round of policy improvement. IQL [40] extends this idea using expectile regression within the SARSA update, enabling multi-step dynamic programming. In contrast to strict in-sample learning, some methods propose to exploit mild generalization beyond the dataset further boost performance [56, 49]. Orthogonal to the above, another influential branch of work formulates offline RL as conditional sequence modeling [10]. These methods employ causal transformers conditioned on the desired return, past states, and actions to autoregressively generate future actions [10, 84, 51, 28], bypassing the need for bootstrapping in long-term credit assignment [62, 64].
Model- offline RL. Model-based offline RL approaches construct learned model of the environment dynamics, which is subsequently used to generate synthetic trajectories for policy improvement [72, 31, 34]. Given the challenges of distributional shift in offline settings, algorithms such as MOPO [90] and MOReL [36] incorporate mechanisms to quantify the model uncertainty and penalize high-uncertainty state-action pairs, thereby discouraging the exploitation of unreliable model predictions. MOBILE [71] further advances this line of work by proposing an uncertainty quantification approach based on inconsistencies in Bellman estimations within an ensemble of learned dynamics models. Some algorithms bridge the gap between model-based and model-free paradigms by incorporating similar regularization techniques. For instance, COMBO [91] integrates value penalization into model-based rollouts, while BREMEN [57] emphasizes behavior-regularizz policy optimization. More recently, SCAS [55] proposes a generic model-based regularizer for model-free offline RL that unifies OOD state correction and OOD action suppression. Despite their promise, model- based methods often entail substantial computational costs [31], and their efficacy is
highly contingent on the fidelity of the learned dynamics model [59].
Neighborhood in offline RL. This work formalizes the neighborhood constraint for offline RL and demonstrates its key properties through theoretical analyses. Some offline RL methods implicitly involve the concept of neighborhood by incorporating regularization terms that guide the trained policy toward dataset actions [21, 75, 66, 56]. However, due to the multi-modal nature of dataset action distributions [9] and the limited expressiveness of policy architectures [80], these policy iteration approaches cannot ensure that the trained policy's 's output actions for the Bellman target remain within the union of dataset action neighborhoods. In contrast, we adopt a value iteration framework and introduce an auxiliary policy to effectively enforce the proposed neighborhood constraint. BCQ [23] employs a perturbation model related to our auxiliary policy. Specifically, BCQ samples multiple candidate actions per state from a pre-trained generative behavior policy model, perturbs them using the perturbation model, and selects the one with the highest Q value. The perturbation model constructs neighborhoods around the generated actions and is intended to avoid "sampling from the generative model for prohibitive number of times [23]". Subsequent works such as EMaQ [26] and SIBC [9] show that this perturbation model can be omitted without sacrificing performance. By contrast, our auxiliary policy constructs neighborhoods directly around dataset actions and is designed to support Q learning under the proposed constraint, without requiring complex behavior policy modeling. Based on the proposed neighborhood constraint, this work provides a general constrained Q learning framework for adaptive distribution-shif control, with the potential to extend to broader RL paradigms such as safe RL [1, 27, 24] and meta RL [78, 77, 65].
24

---

B Proofs
Lemma I in the main text characterizes the performance difference between two policies subject to divergence constraints, which is related to the results in TRPO [68] and CPO [1]. To prove Lemma 1, we begin with the performance difference lemma [35], which decomposes the difference in policy performance, 17(=²) (T)),, into expectation of advantages.
Lemma 3. For any two policies ni and the following equality holds:
1
((T') (
1-7 Edz dx)(1) (((
(18)
where d' (s) (s)= Ei=o ET [T [st = >|] the state occupancy induced by the policy m.
Proof. Please refer to the proof of Lemma 6.1 in Kakade and Langford [35].
Lemma 4 (Performance bound under density constraint, Lemma 1), If any of the conditions DKL(T [g)) S 2e, ( I 2c, or (T TT) S ve holds, then the policy performance a bounded as follows:
n()) S (TTT)) + 2
(1 7)2 Ve.
(19)
Proof. By Lemma 3, we have
IT(TT) [1(TT]
Σ dro Σ (T)( (s,
(20)
Σ dra (s)
[ (s))
(21)
1
Σ dsr
Σ (s. VT(s))
(22)
1
Σ (s)
Σ T=(a|s)( (s. - Σ (saal))
(23)
1
Σ Edro(s) Σ (TT(a)s) - (=als))
(24)
1
Σ
drp (8)
|xt(a|s) (Q=
(25)
Rmax
Σ d (s)
Σ [TT(aas) - [<a|s)]
(26)
7)²
2mmax
Σ
drp (s)rv(T,TT)))
(27)
(1 (1-
22Rmaxx
Edng(s) (s) min { (RI,(TTIT)) v (To|TT
(28)
7)
where the last inequality holds by Pinsker's inequality.
By substituting the TV condition, Drv(m, T8) & Vc, into Eq. (27), the KL condition, (KL(T]|TT)) S 2c or ( & 2e, into Eq. (28), we derive the final performance bound:
2Rmax Ve.
In(x)) - ((TT) 5
(29)
(1 7/2
25

---

Lemma 2 in the main text analyzes the impact of neighborhood constraint deep Q functions under the neural tangent kernel (NTK) regime [30]. The result a direct corollary of Theorem [45], specialized to the case of action extrapolation. The NTK assumption is presented in Assumption 2. Assumption 2 (NTK assumption). The Q function approximators are two-layer fally-connected ReLU neural networks with infinite width and are trained with an infinitesimally small learning rate.
While not applicable to some practically advanced architectures [95], NTK remains one of the most influential theoretical frameworks for analyzing the generalization of deep neural networks. Assumption 2 is common in previous analyses on the generalization of deep neural networks [30, 4] and the convergence of deep RL [7, 46, 19].
Lemma 5 (Extrapolation behavior, Lemma 2). Under the neural tangent kernel (NTK) regime [30], for any ia-salle state-action pair (s,a) E and in-neighhorhood state-action pair (s, â) such that ||ā all < the value difference of the deep Q functiion can be bounded as:
[[Qu(s, ā) Qu(s, a)ll v [|] @ all. ||s @ all)ve+ 2c). where @ denotes the vector concatenation operation, and C is finite constant.
(30)
Proof. The lemma follows directly from Theorem or Lemma in [45]. Please refer to [45] for detailed proofs.
Proposition the main text assumes that the transition dynamics is Kp-Lipschtzz continuous, which is a common assumption in the theoretical RL studies [15, 16, 87, 66). A small Kp is also empirically supported by observations in physical systems, where small action changes often lead to limited next-state variation [76].
Proposition 2 (Distribution shift, Proposition 1). Let œ be a deterministic policy that satisfies the
neighborhood constraint with threshold e. Assume that the transition dynamics P is K p p-Lisschitz continuous: Vs E S, Vai, 02 A. || a1) - P(s'|s, U2)1l S Kplla azl-- Then, there exists policy 22 satisfying the sample constraint such that:
3Kpe
DrV (d ()),)² S
(31)
2(1 2(1-7)²
where d'
[ [st = s]] is the state occupancy induced by policy n.
Proof. We analyze the difference in state occupancy measures induced by two deterministic policies Ti and n2, leveraging the Lipschitz continuity of the transition dynamics.
The state occupancy measure d" satisfies the Bellman-flon constraints.:
dt (s) (s) Σ P(s 8', (())
(32)
where ds(s) is the initial state distribution.
For policies Ti and n2, their corresponding occupancy measures d and d satisfy: (5') P(s s (s(
(s) Σ
(33)
We decompose the difference into two terms:
d= 2TT Σ ((s)ss (s)
Let Ass) ld (s) - dda (s) and apply the triangle inequality:
A(8) 5 EP(s (' + Σ (P(s s')) P(s Is (2(') d=)
21 Σ P(s (()))()))))) + Σ P(ss s',)1((s)) - P(s | s',)2(5'))
d's (34)
26

---

Sum the above inequality over states
| = (ss)
$ ΣΣ
E P(s 10,1, ((5()((((6)))A(3) + ΣΣ IP(s s'mm(ss) P(s s' m2('s)) d=)))))=)())))))))))))))))))))))
y Σ A(')) Ed Σ P(s s(,=(('))))))) P(s s2)))())))
> Σ (s' Σ s P(s ss,))(ss))1-
(35)
Using the Kp -ipLisschitz property for transition dynamics P:
( P(s s',me(s))) (pllT((s)) m2(s')l
(36)
Substitute Eq. (36) into the TV bound in Eq. (35):
|| & A|4 + Kp
Σ d (5')[ - 23('l|
(37)
$ bا + KK max || (s) - T2(1))|-
-
(38)
Rearranging terms to isolate ||]| we obtain the following expression for the total variation distance between the state occupancy measures:
1 Drv(d" d = 2 Alll S TKp
2(1 max (( - T2()))
(39)
Finally, based on the definitions of the neighborhood and sample constraints, we conclude that for any deterministic policy æı that satisfies the neighborhood constraint (Definition 4), there exists a deterministic policy #2 satisfying the sample constraint (Definition 3) such that for any 8 E D,
||x (s) x2(s)] < For & D, We define (s) == (s). Thus, max, || (s) x2()] S c.
Therefore, the distance between the state occupancies of such policies πi and I2 satisfies:
Kp
Drv d 2(1 7)
(40)
Theorem in the main text uses the standardness assumption, which is common in geometric measure theory [13, 8). Assumption 3 (Standardness,, Assumption 1). Let S d be the support of probability distribution v. and Bzx, r) be the closed ball of radius centered at x E Rd. There exist constants To >> 0 and
Co such that:
Vx E S, Vr S To. B(B(z, r)) 2 Co rd.
(41)
It ensures that the measure y/ does not exhibit "holes" at small scales, preventing v from being too sparse near any S. imposes a lower bound the measure of small balls but does not explicitly require the existence of a density.
Theorem (Support approximation via neighborhoods, Theorem 1). Let S C gd be the compact support of distribution v, and let XI...., Xx, be independent and identically distributed samples from v. Define Un.c Uh_1 B(X,, c) as the union of closed balls of radius centered the samples. Let N(S, e/2) denote the covering number of S, i.e., the minimal number of e/2-balls required to cover S. Under the standardness Assumption 1 with constants ro, Co > O, for any (0,1) and < 2ro. if
Cu(c/2ja (log (S,, + log(1/8)),
(42)
then with probubility at least 1 - 5, the Hausdorff distance between and Un.e satisfies
d (S, Un.)) == max
sup inf d(x, u), sup inf d(xx u) Sc.
(43)
#eS ueUn.
ueUn. #ES
27

---

Proof. By the compactness of S, for any > O, there exists a finite set of points fyi...., VN C S such that:
N SC B(y,,
(44)
j=1
where Blyj, e/2) denotes the closed ball of radius e/2 centered at yj, and N = (N,, denotes the covering number of S for a given radius e/2, i.e., the minimal number of e /2-balls required to cover S. The existence of such finite cover follows directly from the compactness of S.
For each yy, the probability that none of the samples
Xn lies in (y,,, is:
IP (B(yj, (X X 0) (1 (y f/2)))
(45)
Under the standardness assumption, there exist constants ro and Co > 0 such that Yz E S, Vr S ro. v(B(z, r)) 2 Co rd.
(46)
For S 2ro, we have c/2 S ro, and thus
(B(5,, (0)),/22)))))))))))) 2 Co (c/2)
(47)
Using the inequality x S 6- it follows that
(1 -1- e(=,,, s <e-n)((())) S enCole/))
(48)
Applying the union bound over all N covering balls:
IP (3j s.t. B(ys e/2) n (X1, X 0) ( (B(y,, S NCol//()/
(49)
To ensure this probability is most 6, require
Ne Ce = -e(s)e/2)ee-nCaffe S.
(50)
Solving for n, we obtain
log ( Colc/2)d
(51)
which guarantees that, with probability least 1 5, every B(yj, contains at least one sample X;
The Hausdorff distance is metric used to quantify the discrepancy between two sets and is widely applied in shape comparison and analysis across various domains [29, 74J. captures the maximum mismatch between the sets by measuring how far one must travel from point in one set to reach the nearest point in the other. Mathematically, for two non- non-empty sets and in metric space (e.g., Euclidean space) with distance function d, the Hausdorff distance du (A, B) is defined
dn () max sup inf d(a, b), sup inf d(a,b) (oEA BER
(52)
be BeBaCA A
That is, for each point in set A, we find the closest point in set B, and take the maximum of these minimum distances. Similarly, for each point B. we find the closest point in A, and take the maximum of these minimum distances. The Hausdorff distance is the larger of these two values.
This distance is sensitive to outliers, of the distance. This property renders it particularly appropriate for evaluating approximation effects in our setting, where we aim optimize the Q function over approximated constraint set. Outliers in such a set can exert a substantial influence on the Q function's 's value.
single distant point significantly increase the value
The Hausdorff distance between S and Un,E U B(X,,e) is:
d (S,UU))))) max sup inf d(x, u), sup inf d(x; u) aES ueUn..
(53)
weU,,, ZES
where d' is the Euclidean distance.
28

---

We now analyze each term separately.
(I) Bounding the term suPxes infueu... dx, u):
By Eq. (44), for any x E S, there exists y/ such that a E Blys, e/2). By the coverage guarantee under Eq. (51) (with probability least 1 1- every Bly,, contains at least one sample X;), follows that Byy c/2) contains some X; Thus,
d(z,X,, S (z) (()):X)) S </
(54)
This implies x E B( _ Un,e. Since I E S is arbitrary, it holds that _ Un,es yielding supzes infueU,.. d(x, u) O.
(2) Bounding the term suPHEt ins d(æ, u):
By definition, any u E Un.e belongs to B(X,, e) for some X, E S. Therefore, for any u E Un,es the following holds:
inf d(z,, u) S d(X,, u) S c.
(55)
*E8
Thus, suPueU... inf d(x,, u) S Combining both terms, we conclude
( Un,e) S max (0.e) S.
(56)
Therefore, when the sample size satisfies Eq. (51), follows that, with probability at least 8, the Hausdorff distance between S and Un,, satisfies (S, Un.e) S.
C Experimental Details
C.1 Experimental Details on D4RL Benchmarks
We adopt evaluation criteria consistent with those employed in prior studies. For the Gym locomotion tasks, performance is assessed by averaging returns over 10 evaluation trajectories and 5 random seeds. In the AntMaze suite, returns are averaged over 100 evaluation trajectories across the same number of seeds. Following the D4RI. benchmark guidelines [20], we subtract 1 from the rewards in the AntMaze datasets. Additionally, in line with established practices [21, 40, 82, 88). we normalize the states in all Gym locomotion datasets. Our implementation builds upon TD3 [22], optimizing deterministic policy. Consequently, we employ mean squared error in the weighted behavior cloning objective in Eq. (17), instead of a log-likelihood term. This approach is commonly used in RL algorithms, where a maximum likelihood problem is transformed into regression problem when dealing with Gaussians with fixed variance [21]. The architectures of the auxiliary policy and the final policy are identical, with the former's ' output range (i.e., max action) set to twice that of the latter account for the most extreme cases. Performance reported using normalized scores provided by the D4RL benchmark [20], which quantify the quality of the learned policy relative to both random and expert baselines:
D4RL score = 100 x learned policy return random policy return
(57)
expert policy return random policy return
ANQ introduces two key hyperparameters: the Lagrange multiplier X, which controls the overall neighborhood radius, and the inverse temperature a, which controls the adaptiveness of neighborhood radius to advantage values. For the inverse temperature, we fix a = across all tasks, which yields consistently strong performance. For the Lagrange multiplier, we tune À within small set of values (0.1,5.0) 5.0} for all tasks. Given that part of our algorithm incorporates techniques from IQL [40], such as expectile regression and weighted behavior cloning, we adopt the hyperparameters suggested in their work: r = 0.7 and B 3 for Gym locomotion tasks, and T = 0.9 and B = 10 for AntMaze tasks. To ensure that the constrained neighborhood is neither too large nor too small, we clip the exponentiated advantage weight exp(a(Qer (s, - (s)) in Eq. (13) to [0. in the Gym locomotion domains and (0.01, 10] in the Antmaze domains. Following prior practices, we also clip the exponentiated weight exp(3(Qu (8, (s, a)) Vo (s))) in Eq. (17) to [0,3] in the Gym locomotion domains and (0, 100) in the Antmaze domains to avoid instability. comprehensive list of hyperparameter settings for ANQ is provided in Table 3.
29

---

Insights into hyperparameter selection. For selecting the Lagrange multiplier X, general principle is as follows: for narrow-distribution datasets (typically expert or demonstration data), larger À can be used to induce smaller neighborhoods, thereby helping to more effectively control extrapolation error. For more dispersed datasets (often containing noisy or mixed-quality data), the impact of extrapolation error is relatively weaker; hence, smaller X (i.e., larger neighborhoods) can be adopted to promote broader optimization over the action space and mitigate the impacts of suboptimal actions. Additionally, using a larger inverse temperature in such cases can further enhance this effect by downweighting low-advantage actions more aggressively.
Table 3: Hyperparameters of ANQ.
Hyperparameter
Value
Optimizer
Adam [37]
Critic learning rate
x 10-4
Actor learning rate
3 x 10 with cosine schedule
Discount factor
0.99 for Gym, 0.995 for Antmaze
Target update rate
0.005
ANQ
Policy update frequency
2
Number of Critics
4
Batch size
256
Number of iterations
106
Lagrange multiplier À
(0.1,
Inverse temperature a
Expectile т
0.7 for Gym, 0.9 for Antmaze
IQL Specific
Inverse temperature 3
3.0 for Gym, 10.0 for Antmaze
Actor Critic
input-256-256-ouppll input-256-256-1
Architecture
C.2 Experimental Details of Noisy Data Experiments
In the noisy data experiments, we construct noisy datasets by mixing the random and expert datasets in D4RL at various expert ratios, thereby simulating real-world scenarios such as suboptimal data collection in autonomous systems or imperfect demonstrations in robotics. The total size of the combined dataset fixed x 10 In certain environments, the sizes of the random or expert datasets in D4RL are slightly smaller than 1 x 10 so we directly use the corresponding D4RL dataset when the expert ratio 0 or
We evaluate the performance of several representative algorithms with different constraint types, including CQL (density constraint), IQL (sample constraint), SPOT (support constraint), and ANQ (neighborhood constraint). The hyperparameters of ANQ follow those used in the default benchmark datasets, detailed in Table and we fix À 5 for all the mixed datasets. In addition, we clip the exponentiated ad advantage weight in Eq. (13) to [0.1, 100) for Hopper and Walker2d, and to [0.1, for Halfcheetah.
Hyperparameters of baseline methods. The hyperparameter configurations for the baseline methods are as follows. For CQL, tune its regularization coefficient within the set {5, 10, 20, 30), as it performs relatively well in this range, and report the best results obtained for each dataset. For IQL., adopt the hyperparameters suggested in their work: expectile temperature 3 = 3 on Gym locomotion tasks. For SPOT, we follow the implementation details provided in their paper, tuning its regularization coefficient within (0.05, 0.1, 0.2, 0.5, 2.0) Gym locomotion tasks, and report the best results obtained for each dataset.
0.7 and inverse
C33 Experimental Details of Limited Data Experiments
In the limited data experiments, we generate reduced datasets by randomly discarding some portion of transitions from the default AntMaze datasets. This setup simulates practical scenarios where data is sparse partially missing, such as in healthcare applications.
30

---

We evaluate the performance of IQL (sample constraint). SPOT (support constraint), and ANQ (neigh- borhood constraint), excluding CQL (density constraint) due to its consistently inferior performance on Antmaze tasks, as reported in Table 2. The hyperparameters for ANQ follow those used in the default benchmark datasets, as detailed in Table 3, except that the Lagrange multiplier is set to enable sufficiently large neighhorhood.
Hyperparameters of baseline methods. The hyperparameter configurations for the baseline methods are as follows. For CQI., tune its regularization coefficient within the set {5, 10, 20, 30), as it performs relatively well in this range, and report the best results obtained for each dataset. For IQL, we adopt the hyperparameters suggested in their work: expectile T 0.9 and inverse temperature 3 = 10 on Antmaze tasks. For SPOT, we follow the implementation details provided in their paper, tuning its regularization coefficient within {0.025, 0.05, 0.1, 0.25, 0.5, 1.0} on Antmaze tasks, and report the best results obtained for each dataset.
D
Additional Experimental Results
D.1 Computational Cost
assess the runtime offline RL algorithms on the halfcheetah- hatttheeta--mcciiu-ree--------- dataset using a GeForce RTX 3090. The training time for ANQ and the baseline methods are presented in Figure 5. ANQ completes the task in approximately two hours, achieving competitive efficiency with other fast offline RL algorithms such as AWAC, IQL, and TD3BC,
Several factors contribute this efficiency: (i) ANQ is model-free and ensemble-free, making more efficient than model- -ased (e.g.. MOPO [90]) and ensemble-based methods (e.g., EDAC [2]); (ii) ANQ uses simple three-layer MLPs for policy and value networks (as TD3 [22]). which are more lightweight than complex architectures used in methods like Decision Transformer [10] or Diffusion Q-learning [80]: (iii) ANQ optimizes deterministic policies with a policy update frequency of 2 (as in TD3 [22]), offering slightly better efficiency than algorithms with stochastic policies; (iv) ANQ solves the bi-level objective via alternating updates, requiring only one extra lightweight optimization step for the auxiliary policy compared to IQL [40]. As result, ANQ achieves competitive training efficiency while maintaining strong performance.
Runtime of Offline RL Algorithms
16h
15
15
10
4h
2h
2
DT
MOPO
CQL
OneStopAWAC
ANQ
IQL
TD3BC
Figure 5: Runtime of algorithms haffchetaah-mediin-rrraay---- GeForce RTX 3090.
D.2 Additional Benchmark Comparisons
This section extends our evaluation by comparing ANQ with additional recent SOTA algorithms of other constraint types on the D4RL benchmark [20]. For support constraints, we compare against STR [53], CPI [50], CPED [93], and SVR [54]. For sample constraints, we compare against LAC [92], SQL [88], and EQL [88]. As shown in Table 4, ANQ achieves the best overall score in the Antmaze domain and demonstrates highly competitive performance in the Gym locomotion domain. From methodological design perspective, the proposed neighborhood constraint may be particularly beneficial high-dimensional action spaces such as those in Antmaze. such settings, support constraint methods often struggle to accurately model the behavior policy, while sample constraint
31

---

methods risk being overly conservative due to limited coverage of near-optimal actions. ANQ avoids both issues by enabling controlled generalization without explicit behavior policy modeling.
Table 4: Averaged normalized scores on Gym locomotion and Antmaze tasks over five random seeds. medium, medium-reppe replay,, medium medium-expert, expert, random; umaze, u-d umaze-diverse m-p medium- play, m-d = medium-diverse, 1-p= large-play, I-d = large-diverse.
Dataset-v2
TAC
SQL
FQI.
STR
CPI
CPED
SVR ANQ (Ours)
halfchectah-m
01.6=#.3
48.3=0.2
47.2++0.3
51.800.3
6.,4=1.3
61.8=1.6
60.5+112
1.
hopper-m
14.6
75.5133
70.612..
101.300.4
98.513.0
100.1:2288
103.510.4
100,910.6
walker2d-m
85.2:10.4
84.2.1466
83.244.
85.91111
85.8.0.8
90.221.7
92.4112
82.9
halfcheetah-m-r
0..2-+0.3
44.8++0.7
44.5+0.5
47.5+0022
1..6#1.3
55.8:229
52.5+3.0
55.5+
hopper-m-r
103.2=1100
101.743..
98.1433.6
100.0:1.22
101.7=166
98.1=2.1
103.7+1.3
101.5=2277
walker2d-m-r
93.2.11.8
77.21338
81.614.2
85.71:2.2
91.8:229
91.91009
95.612..5
92.713.8
haffcheetah-m-e
92.9-1077
94.0:10.4
94.610.5
94.91116
94.7=1111
85.4110.9
94.2:12.2
94.2.10.8
hopper-m-e
109.3:+4.0
111.8+222
111.5++2.1
111.9.0.6
106.4+4.3
93.3+
111.2:10.9
107.0::499
walker2d-m-e
110.1=0.1
110.0=0.8
110.2+0.8
110.2+0.11
110.9=04
113.0=14
109.3+0.2
111.7=0.2
halfcheetab-e
94.511..5
95.2.10.3
96,5.0.2
96.1100.7
95.91004
hopper-e
110.6-11.9
111.2:10.3
112.2
111.1-0.4
111.4
walker2d-e
1.2
110.1+0.1
110.6+0.1
110.0++0.2
111.+0.11
halfchectah-r
20.9=112
20.6+1.1
29.7=111
27.2+112
24,9=1.0
hopper-r
01.3==0.3
31.3+0.3
29.5=3.7
31.0=0.3
31.1=0.2
walker2d-r
3.0
4.743.88
5.9117
2.241.5
11.2.19.5
locomation
1142.4
1162.2
1193.2
1200.5
1194.5
antmaze-u
77.6.1..8
92.2.11.4
93.212.
93.614400
98.8
2.6
1.6
antmaze-u-d
71.2=8866
74.0+22.
70.4+2.7
77.47.22
88.6=5.7
55.6=2.2
80.2+ 1.8
antmaze-m-p
72.0=7.6
80,243.7
77.544.
82.615.4
82.415.88
85.1
76.2
antmaze-m-d
74.2:14.1
75.114.2
74.0:3.77
87.014.22
80.48899
72.1=229
77.2:16.1
antmaze-l-p
57.0.174
50,2.14.8
45.6114.2
42.8.18.7
20.6.1.6.3
34,915.3
56.2 4.9
antmaze-l-d
47.24194
52.3+-52
49.5+44.7
46.8477.6
45.2+6.9
32.3=7.4
55.8
antmaze total
399.2
424.0
410.2
430.2
416.0
376.8
441.6
D.3
Effeet of the Auxiliary Policy
This section conducts ablation study that replaces the auxiliary policy μwu in ANQ with random Gaussian noise.
Discussion on exploiting vs. smoothing. Defining the auxiliary policy as Gaussian noise is reminiscent of the target policy noise trick introduced in TD3 [22]. Specifically, TD3, Gaussian noise is added to the policy actions in the Bellman target. Because the trained policy is prone to overfitting to local optima in the Q-function landscape, such noise injection can smooth the value estimates and reduce accumulated eтrors in Q-function training. By contrast, in our method, Pw is applied to fixed in-dataset actions. Therefore, there is no need for Q-function estimate smoothing as in TD3. On the contrary, our method exploits, rather than smooths, the local Q landscape surrounding dataset actions, using it as a guide for the inner optimization. seek better actions within the neighborhood of dataset actions. Replacing pa with Gaussian noise thus turns this directed search into undirected random perturbations.
our algorithm, Paa is optimized to
Empirical results. We compared different choices of the auxiliary policy on the Gym locomotion tasks, with results shown Table 5. The tested variants include: (i) ANQ-Gaussian noise μr: μ is replaced with Gaussian noise, i.e., μ ~ clip(N((0. 0.04), -0.5, 0.5) as used in TD3 and TD3BC; (ii) IQL: correspond ANQ with μ set zero; (iii) ANQ-default: our default ANQ algorithm where Pss is optimized. The results show that using an optimized (i.e., default ANQ) significantly outperforms the other two variants, especially on low-quality datasets, clearly demonstrating the benefit of our design. Additionally, ANQ with Gaussian noise μ performs slightly worse than IQL., likely because the Q values of dataset actions do not require smoothing, and the randomness introduced by Gaussian noise may cause the Bellman target to select actions inferior to the original datasct actions.
32

---

Table 5: Averaged normalized scores on Gym locomotion tasks over five random seeds.
Dataset-v2
IQL
ANQ-Gaussian noise μ ANQ-stochastic pdo
ANQ-default
halfeheetah-m
47.410.2
46,9:0.2
59.8=3.6
61.8+1.4
hopper-m
66.2+5.7
63.2#5,4
94.5=7.9
100.9=0.6
walker2d-m
78.3+8.7
82.9#2.2
84.3=0.9
82,9#1.5
halfcheetah-m-rr
4..2+t1.2
43.1
54.3+11.9
55.5+:1.4
hopper-m-r
94.7+18.6
66.14998
103.3=0.4
101.5+22.7
walker2d-m-r
73.8+7.11
81.2++4.5
93.6++299
92.743388
halfcheetah-m--e
86.71:5.3
85.1
94.6210.5
94.241.88
hopper-m-e
91.5=114.3
60.8+38.4
106.0=5.2
107.014.9
walker2d-me m--e
109.6:11.0
111.910.5
112.1=0.2
111.710.2
halfcheetah-r
13.1+1.3
2.3+0.0
26.5+11.7
24.9+1.0
hopper-r
7.9+0.2
7.2+0.2
31.0+0.2
31.1+0.2
walker2d-r
5.4+1.2
6.0+0.2
8.347.6
11,2+99.
total
718.8
656.7
868.2
875.4
D.4
Stochastic Policies
In practice, our algorithm is compatible with stochastic policies. Both policies introduced in the method (the auxiliary policy μ and the final policy л) can replaced by stochastic counterparts without modification to the overall framework.
For optimizing stochastic auxiliary policy phu (typically Gaussian), Eq. (13) can be directly opti- mized using the reparameterization trick:
max (, [Qa(,, a + pa Her
(m (e;;8,)) (exp(a(Qo a) - ())
For extracting stochastic final policy Ty (typically Gaussian), the regression problem in Eq. (17) can be reformulated maximum likelihood objective:
min E (e exp(B(Qo (e (+())) log ग Ta (u(e) xg
Empirically, we tested stochastic version the algorithm, denoted as ANQ-stochastic ,&T, where the both policies are Gaussian. The results, reported in Table 5, show that it performs comparably to default ANQ across most Gym locomotion tasks, though slightly worse on a few.
D.5 Learning Curves
Learning curves of ANQ on Gym-MuloCo locomotion tasks and Antmaze tasks are presented in Figure and Figure 7, respectively. The curves averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
E Broader Impact
Offline reinforcement learning (RL) holds significant potential for expanding RL's real-world appli- cations in fields like robotics, recommendation systems, healthcare, and education, particularly in scenarios where data collection is expensive or risky. Nevertheless, it essential to be aware of the potential negative socictal impacts that could arise from the deployment of offline RL systems. One concern is that the data used for training may contain inherent biases, which could then be reflected in the resulting policies, potentially exacerbating existing inequalities reinforcing harmful stereotypes. Another issue is the potential impact offline RI. on employment, as it may lead to the automation of jobs that currently performed by humans, such in manufacturing autonomous driving. To ensure the responsible use of offline RL, it is crucial to address these challenges, striking a balance between fostering innovation and mitigating adverse societal consequences.
33

---

balfdhectb dacehhheeaaaaoooo
100 dodalides eplaa
balccheub andem--2
Epioadd R-aan
Egionde itmum
Equonde mumu
Egisond
50
12
1150
8100
810
Steps (x10%)
Steps (x109)
Steps
Stajs (x109)
bargen-capedo222
hurpe: mnduum-elloeenete
bopoz: madistt-22
100
Epiinddo EpiandeRRurs Rourz
Upsande Retunt
Lppond RonoodoRuuuu Munan
Iprondo
Igrond
12
50
11:3 1.50 B.7 11b
0.00
023
07.5 1700
6mm
0.25
1130 1125 7mh
n21 1:30 1.75 1mh
8mm
1:25 1:30 075
Grodien Steps (x10%)
Giadeen Steps (x10f)
Grodiest Stepa (x10%)
Grudian Steys (x10%)
Gradient Steps (x 109)
wulterill endfam-erslaaee
wolkeral-mmddeme
100
100
100
Eplande Roazz
Epiank Raarn
Uppoode
Eppoade
Uiponde
20
30
8co
82o
0.00
8mm
8o
Stepo (X10%)
Stegs
Stepe
Stepe [X10%)
Stepe
Figure 6: Learning curves of ANQ Gym locomotion tasks during offline training. The curves are averaged over random seeds, with the shaded area representing the standard deviation across seeds.
100
antraaae um2ze-v2
100
aptmaz: umaze diverse-12
100
antmaze mediom: m2dyum--pay-v22
Episode Retum
Eppssdee Ratur
W
Epiiode Renum
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Gradient Steps (x10%)
Gradiem: Steps (x10%)
Gradieut Sceps (x109)
100
matmaz-mollun-diiveeeee
100
antmaze- banezzee-rreeppay--22
100
sninuze-baro--ie bnntrree-lbeedivvee->>
Epizede Return
Epiisoee Retum
Rpirode Retum
50
50-
0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.0
0.75
1.00
Gradient Steps (x109)
Gradient Steps (x109)
Gradicnt Steps (x109)
Figure 7: Learning curves of ANQ Antmaze tasks during offline training. The curves averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
34


Reviews:
[{"review_id": "ZNsQDCIyfZ", "reviewer": "Reviewer_JKyo", "summary": "The paper introduces Adaptive Neighborhood-Constrained Q Learning (ANQ), a method designed to mitigate extrapolation bias in offline reinforcement learning (RL) by applying an adaptive neighborhood constraint to restrict action selection within a set of actions that are near the ones seen in the dataset. This constraint is enforced using a bi-level optimization framework, where the inner optimization updates the Q-function and the outer optimization updates the policy parameters.", "questions": "1. The bi-level optimization framework introduces significant complexity by requiring four separate updates per mini-batch. This could lead to instability or slow convergence, especially in environments with high variance or limited data. Could the authors provide a more detailed analysis of how this affects training stability and convergence in practice? Are there any strategies for mitigating potential instability, particularly when training with limited data or in environments with high-dimensional state-action spaces?\n\n2. Given the increased complexity of bi-level optimization, does the method show diminishing returns as the training progresses? How does the computational cost of this approach compare to other state-of-the-art offline RL methods, especially in terms of training time and memory usage? Could the method be optimized further to reduce the computational burden?\n\n3. The Lagrange multiplier λ and the inverse temperature α play important roles in controlling the neighborhood radius and enforcing the constraint. Could the authors provide more insight into how these hyperparameters are tuned in practice? Does their choice significantly impact training stability or the final policy performance? Is there an automatic way to tune these parameters based on the environment or data?\nScalability to Complex and High-Dimensional Tasks:\nWhile the method performs well in standard benchmark tasks, how does it scale to high-dimensional action spaces or environments with partially observable states? Does the method retain its robustness and performance in such scenarios, or are modifications needed to accommodate the complexity of real-world tasks?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The introduction of an adaptive neighborhood constraint is a novel and effective approach for balancing conservatism and exploration in offline RL. \n\n2. The method addresses a significant problem in offline RL, providing a flexible solution to extrapolation bias without the need for complex behavior policy modeling.\n\n3. The paper is generally well-written, with clear motivations, method descriptions, and results. The authors effectively explain the problem of extrapolation bias in offline RL and introduce the adaptive neighborhood constraint as a solution.\n\n**Weaknesses:**\n\n1. Bi-level optimization introduces significant complexity to the method. Since each mini-batch requires four separate updates to the parameters, the computational cost becomes expensive. This results in increased training time per iteration, which could be a significant drawback, especially in high-dimensional or real-world applications.\n\n2. The instability introduced by the bi-level optimization process is not fully explored. Given that each mini-batch requires multiple updates to the parameters, this could lead to issues with convergence or training stability, particularly in environments with high variance or limited data. \n\n3. A more detailed discussion of the potential instability and its mitigation would provide clearer guidance for users and researchers adopting this method.", "was_revised": true, "final_justification": "The rebuttal discussion solves me concerns.", "rebuttal": "We appreciate the time and effort you have dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: About the cost and complexity of bi-level optimization.**\n\n(1) Runtime efficiency.\n\nWe briefly describe the training time results in the Benchmark Results section, with detailed comparisons provided in Figure 5 of Appendix D.1. The proposed ANQ is among the fastest tier of offline RL algorithms, on par with efficient baselines such as AWAC, IQL, and TD3BC. Several factors contribute to this efficiency:\n\n- ANQ is model-free and ensemble-free, making it more efficient than model-based (e.g., MOPO [1]) and ensemble-based methods (e.g., EDAC [2]);\n- ANQ uses simple three-layer MLPs for policy and value networks (as in TD3), which are more lightweight than complex architectures used in methods like Decision Transformer [3] or Diffusion Q-learning [4];\n- ANQ optimizes deterministic policies with a policy update frequency of 2 (as in TD3), offering slightly better efficiency than algorithms with stochastic policies;\n- ANQ solves the bi-level objective via alternating updates, requiring only one extra lightweight optimization step for the auxiliary policy compared to IQL.\n\nAs a result, ANQ achieves competitive training efficiency while maintaining strong performance.\n\n(2) Memory usage.\n\nAll four networks used in ANQ are three-layer MLPs, resulting in low memory overhead as well. We test GPU memory usage on the halfcheetah-medium-replay-v2 dataset using a GeForce RTX 3090.  As shown in the table below, the memory consumption of ANQ is also comparable with other efficient offline RL algorithms.\n\nTable 2: GPU memory usage of offline RL algorithms.\n\n| Algorithm | CQL | TD3BC | EDAC | ANQ |\n| --- | --- | --- | --- | --- |\n| GPU Memory (GB) | 1.4 | 1.4 | 1.9 | 1.4 |\n\n**Q2: About potential instability from bi-level optimization.**\n\nThanks for the thoughtful comment. As shown in the training curves (Appendix D.2, Figures 6 and 7), ANQ generally demonstrates stable learning and convergence across most tasks. We observe no notable signs of instability such as divergence or diminishing returns.\n\n> this could lead to issues with convergence or training stability, particularly in environments with high variance or limited data.\n\n(1) High-variance environments. Hopper and Walker2d are commonly used to evaluate robustness in high-variance environments due to their sensitive dynamics. Even small action changes can cause the agent to fall, leading to highly variable returns. Our results show that ANQ maintains relatively stable training progress and strong convergence performance on both tasks, suggesting that the bi-level optimization framework does not compromise stability in these high-variance environments.\n\n(2) Limited data settings. It is worth noting that Section 4.3 specifically evaluates performance under limited data. As shown in Figure 2, ANQ consistently outperforms baselines with more stable final performance across varying data discard ratios, supporting its robustness in limited data scenarios.\n\n> A more detailed discussion of the potential instability and its mitigation would provide clearer guidance for users and researchers adopting this method.\n\nWhile our results demonstrate ANQ's overall stability across tasks, we agree that guidance on mitigating potential instability can be valuable. Below are two practical suggestions based on our analysis.\n\n(1) Increase $\\lambda$: Ablation results show that too small $\\lambda$ (i.e., too large neighborhoods) can lead to increased return variance. Therefore, increasing $\\lambda$, which tightens the neighborhood constraint, may help enhance stability if necessary, though potentially at the cost of reduced policy flexibility.\n\n(2) Decrease $\\alpha$: Although ANQ remains stable across a wide range of $\\alpha$ in our experiments, we hypothesize that setting a smaller $\\alpha$ may further enhance stability by preventing low-quality data points from having excessively large neighborhood radii.\n\n**Q3:  Effects and more insights of hyperparameter tuning.**\n\n> Does their choice ($\\lambda$ and $\\alpha$) significantly impact training stability or the final policy performance?\n\n(1) Effects of $\\lambda$ and $\\alpha$. We have conducted an ablation study in Section 4.4 on the Lagrange multiplier $\\lambda$ and inverse temperature $\\alpha$, analyzing learned Q-values and performance. ANQ generally exhibits stronger and more robust performance with intermediate values of $\\lambda$ and $\\alpha$. Since $\\lambda=0$ and $\\lambda=\\infty$ correspond to no constraint and sample constraint respectively, the results indicate that the proposed neighborhood constraint outperforms both across a range of hyperparameters. Similarly, as $\\alpha=0$ corresponds to a fixed neighborhood radius, the results show that advantage-based adaptive neighborhoods generally outperform uniform neighborhoods.\n\n> Could the authors provide more insight into how these hyperparameters are tuned in practice? \n\n(2) Hyperparameter details. The ANQ algorithm involves relatively little hyperparameter tuning. As detailed in Appendix C1, we fix $\\alpha=1$ across all tasks, which yields consistently strong performance. For $\\lambda$, we use only a small set of values: $\\lambda = 5$ for Antmaze, and $\\lambda \\in \\\\{0.1, 5\\\\}$ for Gym locomotion, involving less tuning compared to baselines like CQL and SPOT.\n\n(3) Insights into hyperparameter selection. For selecting $\\lambda$, a general principle is as follows: for narrow-distribution datasets (typically expert or demonstration data), a larger $\\lambda$ can be used to induce smaller neighborhoods, thereby helping to more effectively control extrapolation error. For more dispersed datasets (often containing noisy or mixed-quality data), the impact of extrapolation error is relatively weaker; hence, a smaller $\\lambda$ (i.e., larger neighborhoods) can be adopted to promote broader optimization over the action space and mitigate the impacts of suboptimal actions. Additionally, using a larger $\\alpha$ in such cases can further enhance this effect by downweighting low-advantage actions more aggressively.\n\n> Is there an automatic way to tune these parameters based on the environment or data?\n\n(4) Automatic tuning. As with many offline RL algorithms, these parameters primarily control the trade-off between conservatism and exploration. Automatically tuning such parameters remains a well-recognized and important open problem in offline RL. Research progress in this direction could be integrated into our method by leveraging the above hyperparameter selection insights. While automatic tuning is not the main focus of this work, we believe it is a promising direction for future research.\n\n**Q4: While the method performs well in standard benchmark tasks, how does it scale to high-dimensional action spaces or environments with partially observable states?** \n\nThanks for the good question.\n\n(1) Partially observable environments. To our knowledge, partially observable settings are not commonly featured in standard offline RL benchmarks. To simulate partial observability, we modify the Hopper and Walker2d environments by removing the agent's z-axis position, mimicking a height sensor failure. We compare representative algorithms with different constraint types, including TD3+BC, IQL, SPOT, and ANQ. Results averaged over 5 random seeds are shown in the table below. While all methods exhibit performance drops under partial observability, ANQ outperforms the baselines on most datasets and achieves the highest total score, demonstrating its robustness in such settings.\n\nTable 1: Normalized scores in partially observable environments without height information, averaged over 5 random seeds.\n\n| Dataset | TD3+BC | IQL | SPOT | ANQ |\n|---|---|---|---|---|\n| hopper-m-e | 85.3 | 65.7 | 82.3 | **99.6** |\n| hopper-m-r | 43.8 | 66.7 | 86.4 | **88.9** |\n| hopper-m | 55.9 | 54.1 | 50.6 | **67.0** |\n| hopper-r | 8.1 | 8.2 | 22.9 | **31.5** |\n| walker2d-m-e | 109.2 | 103.5 | 109.9 | **110.4** |\n| walker2d-m-r | 70.5 | 65.5 | **84.4** | 77.5 |\n| walker2d-m | 76.8 | 72.2 | **79.1** | 78.8 |\n| walker2d-r | 2.0 | 6.8 | 2.8 | **13.9** |\n| total | 451.5 | 442.5 | 518.3 | **567.6** |\n\n(2) Complex tasks and high-dimensional action spaces. \n\nAntMaze represents one of the most challenging offline RL benchmarks, with high-dimensional action space and sparse-reward, goal-reaching nature. The agent must not only coordinate multiple joints for locomotion but also compose suboptimal trajectory fragments to reach the goal. Empirical results show that ANQ significantly outperforms baseline methods on these tasks, especially in the most complex large-maze variants, demonstrating its effectiveness in such complex settings.\n\nFrom a method design perspective, the proposed neighborhood constraint may be particularly beneficial in high-dimensional action spaces. In such settings, support constraint methods often struggle to model the behavior policy accurately, while sample constraint methods risk being overly conservative due to limited coverage of near-optimal actions. ANQ avoids both issues by enabling controlled generalization without behavior policy modeling.\n\n**Reference**\n\n[1] Yu et al. Mopo: Model-based offline policy optimization. NeurIPS 2020.\n\n[2] An et al. Uncertainty-based offline reinforcement learning with diversified q-ensemble. NeurIPS 2021.\n\n[3] Chen et al. Decision transformer: Reinforcement learning via sequence modeling. NeurIPS 2021.\n\n[4] Wang et al. Diffusion policies as an expressive policy class for offline reinforcement learning. ICLR 2023."}, {"review_id": "z83Al38Bbo", "reviewer": "Reviewer_DsUf", "summary": "This paper addresses the challenge of extrapolation errors in offline reinforcement learning (RL), which occur when evaluating out-of-distribution (OOD) actions not present in the static training dataset. The authors propose a novel neighborhood constraint that restricts action selection in the Bellman target to the union of neighborhoods around dataset actions, offering a flexible middle ground between overly conservative density/sample constraints and the computationally demanding support constraint (which requires accurate behavior policy modeling). The proposed constraint bounds extrapolation errors and distribution shift while approximating the least restrictive support constraint without requiring behavior policy modeling. The constraint is made pointwise adaptive by adjusting the neighborhood radius for each data point based on its advantage (quality). Empirically, ANQ achieves state-of-the-art results on standard offline RL benchmarks (D4RL), outperforming existing methods like CQL, IQL, and SPOT on both Gym locomotion and challenging AntMaze tasks.", "questions": "* Could u please provide an additional ablation that changes the $\\mu_{\\omega}$ to a random variable sampled from gaussion?  As the performance improvement could also be derived from the Q-landscape smoothness despite theoretical supports.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strength\n* The paper introduces a neighborhood constraint that restricts Bellman targets to the union of neighborhoods around dataset actions. Theoretical analyses (Lemma 2, Proposition 1, Theorem 1) rigorously establish that the neighborhood constraint bounds extrapolation errors under mild continuity assumptions, approximates the support constraint without needing behavior policy modeling and\ncontrols distribution shift via Lipschitz continuity of transition dynamics.\n* The proposed bilevel optimization (inner maximization + outer expectile regression) efficiently enforces the constraint during Q-learning.\n* ANQ achieves top-tier results on D4RL benchmarks, including Gym locomotion and challenging AntMaze tasks. Robustness is validated in noisy data and limited data scenarios, where ANQ outperforms density (CQL), sample (IQL), and support (SPOT) constraints.\n\n## Weakness\n* The performance of ANQ might be sensitive to the choice of hyperparameters. A more detailed analysis of hyperparameter sensitivity would be beneficial to understand the robustness of the method.\n* Some new sota support/sample constraint baselines are suggested such as [1][2].\n\n\n[1] Iteratively Refined Behavior Regularization for Offline Reinforcement Learning\n\n[2] Supported trust region optimization for offline reinforcement learning", "was_revised": true, "final_justification": "I think the authors have fully addressed my concerns. I recommend to accept the paper.", "rebuttal": "We appreciate the time and effort you have dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: A more detailed analysis of hyperparameter sensitivity would be beneficial to understand the robustness of the method.**\n\nThank you for the suggestion. We analyze the hyperparameter sensitivity from the following three aspects:\n\n(1) Superior performance within moderate parameter ranges. As shown in the ablation study in Section 4.4, ANQ generally exhibits stronger and more robust performance with intermediate values of Lagrange multiplier $\\lambda$ and inverse temperature $\\alpha$. Since $\\lambda=0$ and $\\lambda=\\infty$ correspond to no constraint and sample constraint respectively, the results indicate that the proposed neighborhood constraint outperforms both across a range of hyperparameters. Similarly, as $\\alpha=0$ corresponds to a fixed neighborhood radius, the results show that advantage-based adaptive neighborhoods generally outperform uniform neighborhoods.\n\n(2) Relatively limited hyperparameter tuning. The specific ANQ algorithm involves relatively little hyperparameter tuning. As detailed in Appendix C1, we fix $\\alpha=1$ across all tasks, which yields consistently strong performance. For $\\lambda$, we use only a small set of values: $\\lambda = 5$ for Antmaze, and $\\lambda \\in \\\\{0.1, 5\\\\}$ for Gym locomotion, involving less tuning compared to baselines like CQL and SPOT.\n\n(3) Insights into hyperparameter selection. For selecting $\\lambda$, a general principle is as follows: for narrow-distribution datasets (typically expert or demonstration data), a larger $\\lambda$ can be used to induce smaller neighborhoods, thereby helping to more effectively control extrapolation error. For more dispersed datasets (often containing noisy or mixed-quality data), the impact of extrapolation error is relatively weaker; hence, a smaller $\\lambda$ (i.e., larger neighborhoods) can be adopted to promote broader optimization over the action space and mitigate the impacts of suboptimal actions. Additionally, using a larger $\\alpha$ in such cases can further enhance this effect by downweighting low-advantage actions more aggressively.\n\n**Q2: Some new sota support/sample constraint baselines are suggested such as [1][2].**\n\nThanks for the suggestion. The table below reports the overall performance of STR, CPI, and ANQ. ANQ achieves the best results in the Antmaze domain and demonstrates competitive performance with CPI in the Gym locomotion domain.  A full comparison will be included in the revised manuscript.\n\nTable 2: Total scores on Gym locomotion and Antmaze tasks, averaged over 5 random seeds.\n\n| Dataset-v2 | STR | CPI | ANQ |\n|---|---|---|---|\n| locomotion total | 1162.2 | 1193.2 | **1194.5** |\n| antmaze total | 430.2 | 416.0 | **441.6** |\n\n**Q3: Ablation that changes $\\mu_\\omega$ to a random variable sampled from Gaussian.**\n\nThanks for the insightful question. \n\n(1) Clarification on exploiting rather than smoothing. Defining the auxiliary policy $\\mu$ as Gaussian noise is reminiscent of the target policy noise trick introduced in TD3. Specifically, in TD3, Gaussian noise is added to the policy actions in the Bellman target. Because the trained policy is prone to overfitting local optima in the Q-function landscape, such noise injection can smooth the value estimates and reduce accumulated errors in Q-function training. By contrast, in our method, $\\mu_\\omega$ is applied to fixed in-dataset actions. Therefore, there is no need for Q-function estimate smoothing as in TD3. On the contrary, our method exploits, rather than smooths, the local Q landscape surrounding dataset actions, using it as a guide for the inner optimization. In our algorithm, $\\mu_\\omega$ is optimized to seek better actions within the neighborhood of dataset actions. Replacing $\\mu_\\omega$ with Gaussian noise essentially turns this directed search into undirected random perturbations. Thus, our method is not intended to smooth the Q landscape, but rather to exploit its local structure for improvement.\n\n(2) Empirical results. We compared different choices of the auxiliary policy $\\mu$ on the Gym locomotion tasks. Results are shown in the table below. The tested variants include:\n\n- ANQ w/o $\\mu$: $\\mu$ is set to zero, equivalent to IQL.\n- ANQ w/ Gaussian noise $\\mu$: $\\mu$ is replaced with Gaussian noise, i.e., $\\mu \\sim \\text{clip}(\\mathcal{N}(0, 0.04), -0.5, 0.5)$ as used in TD3 and TD3+BC.\n- ANQ w/ optimized $\\mu$: our default ANQ algorithm where $\\mu_\\omega$ is optimized.\n\nTable 1: Averaged normalized scores on Gym locomotion tasks over 5 random seeds.\n\n| Dataset-v2 | ANQ w/o $\\mu$ (IQL) | ANQ w/ Gaussian noise $\\mu$ | ANQ w/ optimized $\\mu$ |\n|---|---|---|---|\n| halfcheetah-m | 47.4$\\pm$0.2 | 46.9$\\pm$0.2 | **61.8$\\pm$1.4** |\n| hopper-m | 66.2$\\pm$5.7 | 63.2$\\pm$5.4 | **100.9$\\pm$0.6** |\n| walker2d-m | 78.3$\\pm$8.7 | 82.9$\\pm$2.2 | **82.9$\\pm$1.5** |\n| halfcheetah-m-r | 44.2$\\pm$1.2 | 43.1$\\pm$0.2 | **55.5$\\pm$1.4** |\n| hopper-m-r | 94.7$\\pm$8.6 | 66.1$\\pm$9.8 | **101.5$\\pm$2.7** |\n| walker2d-m-r | 73.8$\\pm$7.1 | 81.2$\\pm$4.5 | **92.7$\\pm$3.8** |\n| halfcheetah-m-e | 86.7$\\pm$5.3 | 85.1$\\pm$6.2 | **94.2$\\pm$0.8** |\n| hopper-m-e | 91.5$\\pm$14.3 | 60.8$\\pm$38.4 | **107.0$\\pm$4.9** |\n| walker2d-m-e | 109.6$\\pm$1.0 | **111.9$\\pm$0.5** | 111.7$\\pm$0.2 |\n| halfcheetah-r | 13.1$\\pm$1.3 | 2.3$\\pm$0.0 | **24.9$\\pm$1.0** |\n| hopper-r | 7.9$\\pm$0.2 | 7.2$\\pm$0.2 | **31.1$\\pm$0.2** |\n| walker2d-r | 5.4$\\pm$1.2 | 6.0$\\pm$0.2 | **11.2$\\pm$9.5** |\n| total | 718.8 | 656.7 | **875.4** |\n\nThe results show that using an optimized $\\mu$ significantly outperforms the other two variants, especially on low-quality datasets, clearly demonstrating the benefit of our design. Additionally, ANQ with Gaussian noise performs slightly worse than ANQ without $\\mu$, likely because the Q values of dataset actions do not require smoothing, and the randomness introduced by Gaussian noise may cause the Bellman target to select actions inferior to the original dataset actions."}, {"review_id": "Vn1QK6gcPW", "reviewer": "Reviewer_5EYf", "summary": "This paper attempts to address the extrapolation error issue resulting from OOO actions in offline RL by restricting action selection for the Bellman target within some neighborhood of actions in the dataset. The authors provide theoretical guarantees for this choice of constraints. Practically, the authors propose an efficient algorithm by decomposing the objective into a bilevel optimization problem.", "questions": "See question above on stochastic policies.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n- The paper is very well written and easy to read.\n- I particularly like the summary of different constraint types used in modern offline RL algorithms, the authors did an excellent job in summarizing and categorizing recent works with different constraint types, as well as addressing their strengths and limitations.\nThe idea of using neighborhood constraints is straightforward and yet powerful, the authors did a good job at motivating this.\n- I think the theoretical analysis is also one of the key strengths of the paper, with the authors showing through Theorem 1 that the union of sample neighborhoods can approximate the true support.\n- The bilevel optimization algorithm proposed is sound. Using the advantage function for the adaptive constraints is probably the most intuitive choice here and appeared to show good results. I agree with the authors comments in the limitations section that it is worth exploring different instantiations of adaptive neighborhoods in future works.\n- Experiments are very well done with an extensive set of environments and baselines, as well as very detailed comparisons in terms of performance and compute time with each. Empirical results are quite strong\n\nWeaknesses\n\n- The theoretical analysis as well as the practical algorithm assumes a deterministic policy, could the authors comment how/whether it is possible to extend to stochastic policies?\n- The algorithm itself seems to be quite sensitive to the choice of $\\lambda$ and $\\alpha$, however the authors are very up front about this and have performed extensive ablation studies on these hyperparameters, as well as giving nice interpretations on the values for $\\lambda$ and $\\alpha$.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you have dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: Could the authors comment how/whether it is possible to extend to stochastic policies?**\n\nThank you for the meaningful question.\n\n(1) From a practical perspective, our algorithm is fully compatible with stochastic policies. Both policies introduced in the method (the auxiliary policy and the final policy) can be replaced by stochastic counterparts without modification to the core framework.\n\n- For optimizing a stochastic auxiliary policy $\\mu_\\omega$, Eq. (13) can be directly optimized using the reparameterization trick:\n\n$\\max_{\\mu_\\omega} \\mathbb{E}_ {(s,a)\\sim \\mathcal{D},\\epsilon\\sim\\mathcal{N}(0,1)} \\left[ Q_\\theta(s,a+\\mu_\\omega(\\epsilon;s,a)) - \\lambda \\exp(\\alpha (Q_{\\theta'}(s,a) - V_\\psi(s))) \\\\|\\mu_\\omega(\\epsilon;s,a)\\\\| \\right].$\n\n- For extracting a stochastic final policy $\\pi_\\phi$, the regression problem in Eq.(16) can be reformulated as a maximum likelihood problem:\n\n$\\min_{\\pi_\\phi} \\mathbb{E}_ {(s,a)\\sim \\mathcal{D},\\epsilon\\sim\\mathcal{N}(0,1)} \\exp (\\beta(Q_{\\theta'}(s,a+\\mu_\\omega(\\epsilon;s,a)) - V_\\psi(s))) \\log \\pi_\\phi(a+\\mu_\\omega(\\epsilon;s,a)|s)$\n\n(2) Empirically, we tested a stochastic version of the algorithm, denoted as ANQ-Gaussian, where both policies are Gaussian. The results, reported in the table below, show that ANQ-Gaussian performs comparably to the original ANQ across most Gym locomotion tasks, though slightly worse on a few.\n\nTable 1: Averaged normalized scores on Gym locomotion tasks over 5 random seeds.\n\n| Dataset-v2 | ANQ-Gaussian | ANQ |\n|---|---|---|\n| halfcheetah-m | 59.8$\\pm$3.6 | **61.8$\\pm$1.4** |\n| hopper-m | 94.5$\\pm$7.9 | **100.9$\\pm$0.6** |\n| walker2d-m | **84.3$\\pm$0.9** | 82.9$\\pm$1.5 |\n| halfcheetah-m-r | 54.3$\\pm$1.9 | **55.5$\\pm$1.4** |\n| hopper-m-r | **103.3$\\pm$0.4** | 101.5$\\pm$2.7 |\n| walker2d-m-r | **93.6$\\pm$2.9** | 92.7$\\pm$3.8 |\n| halfcheetah-m-e | **94.6$\\pm$0.5** | 94.2$\\pm$0.8 |\n| hopper-m-e | 106.0$\\pm$5.2 | **107.0$\\pm$4.9** |\n| walker2d-m-e | **112.1$\\pm$0.2** | 111.7$\\pm$0.2 |\n| halfcheetah-r | **26.5$\\pm$1.7** | 24.9$\\pm$1.0 |\n| hopper-r | 31.0$\\pm$0.2 | **31.1$\\pm$0.2** |\n| walker2d-r | 8.3$\\pm$7.6 | **11.2$\\pm$9.5** |\n| total | 868.2 | **875.4** |\n\n(3) Theoretically, while Proposition 1 relies on a deterministic policy, Lemma 2 and Theorem 1 hold independently of policy determinism, as they concern properties of the neighborhood constraint itself. However, in practice, a stochastic policy may occasionally generate actions that violate the neighborhood constraint, making it only probabilistically satisfied. Investigating the theoretical properties of probabilistic neighborhood constraints presents an interesting direction for future work.\n\n**Q2: Sensitivity to $\\lambda$ and $\\alpha$.**\n\nWe appreciate the comment and will discuss the hyperparameter sensitivity in three aspects:\n\n(1) Superior performance within moderate parameter ranges. As shown in the ablation study, ANQ generally exhibits stronger and more robust performance with intermediate values of Lagrange multiplier $\\lambda$ and inverse temperature $\\alpha$. Since $\\lambda=0$ and $\\lambda=\\infty$ correspond to no constraint and sample constraint respectively, the results indicate that the proposed neighborhood constraint outperforms both across a range of hyperparameters. Similarly, as $\\alpha=0$ corresponds to a fixed neighborhood radius, the results show that advantage-based adaptive neighborhoods generally outperform uniform neighborhoods.\n\n(2) Relatively limited hyperparameter tuning. The specific ANQ algorithm involves relatively little hyperparameter tuning. As detailed in Appendix C1, we fix $\\alpha=1$ across all tasks, which yields consistently strong performance. For $\\lambda$, we use only a small set of values: $\\lambda = 5$ for Antmaze, and $\\lambda \\in \\\\{0.1, 5\\\\}$ for Gym locomotion, involving less tuning compared to baselines like CQL and SPOT.\n\n(3) Insights into hyperparameter selection. For selecting $\\lambda$, a general principle is as follows: for narrow-distribution datasets (typically expert or demonstration data), a larger $\\lambda$ can be used to induce smaller neighborhoods, thereby helping to more effectively control extrapolation error. For more dispersed datasets (often containing noisy or mixed-quality data), the impact of extrapolation error is relatively weaker; hence, a smaller $\\lambda$ (i.e., larger neighborhoods) can be adopted to promote broader optimization over the action space and mitigate the impacts of suboptimal actions. Additionally, using a larger $\\alpha$ in such cases can further enhance this effect by downweighting low-advantage actions more aggressively."}, {"review_id": "daARYFLkFH", "reviewer": "Reviewer_EzpC", "summary": "This paper proposed to tackle offline RL problems by enforcing neighborhood constraint rather than imposing density/support/sample constraints. Perturbations are added to action values $Q(s, \\tilde{a})$ from an adaptive neighborhood such that $||a-\\tilde{a}||\\leq \\epsilon\\exp(-\\alpha A(s,a))$. The authors formulated a bi-level optimization problem to first obtain $Q$ value from dataset actions then extract the policy from the perturbed Q value.", "questions": "Please refer to the weaknesses section for questions.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\\\nThe motivation of developing a new type of constraint beyond the listed three classes is appreciated. I find the proposed method interesting and has potential links to several existing approaches.\n\n**Weaknesses:**\\\nI found the paper quite hard to follow. Specifically, the first half of the technical sections (before Definition 5) and the second seem to be detached. I can understand that the motivation is to build a neighborhood set  $||a-\\tilde{a}||\\leq \\epsilon$ such that (1) their value/distributional difference can be controlled; (2) generalization capability is guaranteed. To this end, the authors introduced several lemmas and propositions. However, these lemmas are either from existing works or already well-known. For example, lemma 1 is a very standard result and can be seen from e.g. [Achiam et al. 2017].  lemma 2 comes from [43]. I guess there are some issues with proposition 1 because by definition $D_{TV} \\leq 1$ but the authors showed an upper bound depending on $|\\mathcal{S}|$, which seems to be of little use even for small-sized problems.\n\nThe second half also needs more elaboration: Theorem 1 defines covering numbers in terms of $\\mathcal{S}$. But shouldn't it be about actions since we care about neighbors and generalization capability of actions? If it is a typo and the theorem is indeed about actions, then for the assumption of \"independent and identically distributed samples from $\\nu$\" to hold, the policy $\\nu$ has to be fixed? \n\n\nDefinition 5 does not seem to be very well coupled with previous developments : the authors did not sufficiently motivate the radius  $epsilon\\exp(-\\alpha A(s,a))$ nor explained how it connects to the covering number theorem. More importantly, this quantity relies on accurate estimates of advantage in the offline context. But if we can have such accurate estimates free of OOD issues, in principle we don't have to employ this perturbation scheme?", "was_revised": true, "final_justification": "The paper presents a new constraint method to avoid OOD issue. I found the initial submission quite unclear and ambiguous, but the authors have addressed these concerns and promised will clean up the paper later. In summary, I believe these issues do not overweigh the paper's contributions. I therefore have raised my score.", "rebuttal": "We appreciate the time and effort you have dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: The bound in Proposition 1 depends on $|\\mathcal{S}|$.**\n\nThank you for the valuable comment. We first clarify the original bound and then provide a tighter version that eliminates the $|\\mathcal{S}|$-dependency.\n\n(1) Original bound: The bound in Proposition 1 is mathematically valid, but potentially loose. It scales linearly with the neighborhood radius $\\epsilon$, which makes it meaningful when $\\epsilon$ is small. However, the dependence on $|\\mathcal{S}|$ can indeed make the bound less practical in large-scale settings. This linear dependence on $|\\mathcal{S}|$ originates from the pointwise Lipschitz assumption on the transition dynamics, i.e.,\n\n$|P(s'|s,a_1) - P(s'|s,a_2)| \\leq K_P \\\\|a_1 - a_2\\\\|, \\quad \\forall s,s' \\in \\mathcal{S},\\forall a_1,a_2 \\in \\mathcal{A}$\n\nIn the proof (Eq. (35)), this pointwise difference is summed over all possible next states, thus introducing the factor $|\\mathcal{S}|$.\n\n(2) Tighter bound without $|\\mathcal{S}|$-dependency: \n\nThe $|\\mathcal{S}|$-term can be effectively eliminated by replacing the pointwise Lipschitz assumption with an $L_1$-Lipschitz assumption on the transition dynamics:\n\n$\\\\|P(\\cdot|s, a_1)-P(\\cdot|s, a_2)\\\\| _ 1 \\leq K_{L1} \\\\|a_1 - a_2\\\\|, \\quad \\forall s \\in \\mathcal{S},\\forall a_1,a_2 \\in \\mathcal{A}$\n\nThis assumption is consistent with prior theoretical RL studies [5], where bounding distributional shift under action perturbations is critical for convergence guarantees. A small $K_{L1}$ is also empirically supported by observations in physical systems, where small action changes typically lead to limited next-state variation [6].\n\nIn the proof, we replace Eq. (35) with:\n\n$\\sum_s \\left| P(s \\mid s', \\pi_1(s')) - P(s \\mid s', \\pi_2(s')) \\right| \\leq K_{L1} \\\\|\\pi_1(s') - \\pi_2(s')\\\\|.$\n\nThe rest of the derivation remains unchanged, leading to a revised bound:\n\n$\\mathrm{D_{TV}}\\left(d^{\\pi_1}(\\cdot), d^{\\pi_2}(\\cdot)\\right) \\leq \\frac{\\gamma K_{L1}}{2(1 - \\gamma)} \\epsilon$.\n\nThis new bound is more broadly applicable for large-scale problems, as it is independent of $|\\mathcal{S}|$ and typically yields smaller constants in real-world applications. We thank the reviewer for this observation and will include the new bound in the revision to further strengthen Proposition 1.\n\n**Q2: About the meaning of $S$ in Theorem 1 and fixed $\\nu$.**\n\nWe apologize for the confusion caused by the insufficient explanation of notations. Since both the support and neighborhood constraint set are defined over actions conditioned on a given state, Theorem 1 analyzes their approximation relationship at a fixed state, focusing on their difference in the action space. We adopt following abstract notations to formalize the analysis.\n\n> Theorem 1. Let $S \\subseteq \\mathbb{R}^d$ be the compact support of a distribution $\\nu$, and let $X_1,\\dots,X_n$ be i.i.d. samples from $\\nu$. Define $U_{n,\\epsilon} = \\bigcup_{i=1}^n B(X_i, \\epsilon)$ as the union of closed balls of radius $\\epsilon$ centered at the samples. Let $\\mathcal{N}(S, \\epsilon/2)$ denote the covering number of $S$…\n\nIn the context of offline RL: $\\nu$ represents the behavior policy distribution on a given state, which is generally assumed fixed in offline RL. $S$ (distinct from the state space $\\mathcal{S}$) is defined as the support of behavior policy $\\nu$ at that state. $X_1,\\dots,X_n$ are independent and identically distributed (i.i.d.) samples from the behavior policy $\\nu$, corresponding to dataset actions at that state, consistent with the standard offline RL assumption of i.i.d. data from a fixed behavior policy. Thus, the theorem indeed concerns actions and $\\nu$ is fixed. \n\n**Q3: Lemma 1 is a very standard result and can be seen from e.g. [Achiam et al. 2017]. Lemma 2 comes from [43].**\n\nThank you for pointing this out. We agree that Lemma 1 and Lemma 2 are based on established results. Lemma 1 is derived from the performance difference lemma [1] and is related to results in TRPO [2] and CPO [3]. Lemma 2 is a direct corollary of Lemma 4 in [4], specialized to the case of action extrapolation, as noted in the proof. These lemmas are included not as novel contributions, but to highlight the limitations of existing approaches and to motivate the design of our method. We present them as lemmas (rather than propositions or theorems) to distinguish them from our contributions. However, we acknowledge that their connections to prior work were not sufficiently explained in the main text, which could cause confusion. We will revise the manuscript to clarify these points and properly attribute the prior results. The main theoretical contributions of our work are captured in the proposition and theorem.\n\n**Q4: The first half of the technical sections (before Definition 5) and the second seem to be detached.**\n\nThanks for the comment. The flow of the technical sections is as follows:\n\n(1) The first half (before Definition 5) analyzes the limitations of existing constraint types, proposes neighborhood constraint as a new type, and establishes its theoretical properties under uniform neighborhood radii.\n\n(2) The second half (after Definition 5) develops a tractable optimization algorithm that enables Q learning under the proposed neighborhood constraint. Notably, although the optimization algorithm is presented specifically for the adaptive neighborhood in Definition 5, i.e., using radius $\\epsilon \\exp(-\\alpha A(s,a))$, the framework is general and can accommodate any adaptive radius schemes. Specifically, one can simply replace $\\epsilon \\exp(-\\alpha A(s,a))$ in Definition 5 (and Eq. (11)) with $\\epsilon f(s,a)$ to define arbitrary per-sample neighborhood radii, where $f: \\mathcal S \\times \\mathcal A \\to \\mathbb R^+$ is any function adjusting the radius. Correspondingly, in Eq. (12), $\\exp(\\alpha A(s,a))$  becomes $1/f(s,a)$, and Eq. (13) becomes:\n\n$\\max_{\\mu_\\omega} \\mathbb E_{(s,a)\\sim \\mathcal{D}} [Q_\\theta(s,a+\\mu_\\omega(s,a)) - \\lambda \\\\|\\mu_\\omega(s,a)\\\\| / f(s,a)].$\n\nAll other equations remain unchanged, yielding an algorithm that supports arbitrary neighborhood radius schemes.\n\n(3) Definition 5 serves as a transition point between the theoretical and practical parts. It provides a concrete instantiation of adaptive neighborhoods, where the radius is adjusted based on estimated advantage values, motivated by empirical insights.\n\n**Q5: About Definition 5 and advantage estimates.**\n\n> Definition 5 does not seem to be very well coupled with previous developments: the authors did not sufficiently motivate the radius $\\epsilon \\exp(-\\alpha A(s,a))$ nor explained how it connects to the covering number theorem.\n\n(1) Definition 5 is introduced as a practical instantiation of adaptive neighborhoods, where the sample-wise radius is set as $\\epsilon \\exp(-\\alpha A(s,a))$ based on estimated advantages. As explained in the paragraph preceding Definition 5, this design is motivated by empirical insights: expert-like actions are typically concentrated and require smaller neighborhood to reduce extrapolation, while suboptimal actions tend to be more dispersed and can tolerate and benefit from larger neighborhood to enable broader search. The exponential form of advantages $\\epsilon \\exp(-\\alpha A(s,a))$ heuristically captures this intuition.\n\n(2) Definition 5 is intended as a practical complement based on the theoretical foundation established under the uniform setting. Theorem 1 establishes theoretical guarantees under uniform neighborhood radii, serving as a foundation for understanding geometric coverage. Building on this foundation, Definition 5 extends the formulation by introducing an adaptive radius scheme guided by empirical insights. It serves as an empirically grounded instantiation that fits naturally within our framework and supports the broader goal of enabling flexible, pointwise conservatism.\n\n> More importantly, this quantity ($\\epsilon \\exp(-\\alpha A(s,a))$) relies on accurate estimates of advantage in the offline context. But if we can have such accurate estimates free of OOD issues, in principle we don't have to employ this perturbation scheme?\n\nThanks for the comment. We will clarify from two perspectives:\n\n(1) In-distribution estimation: This quantity (adaptive radius) relies solely on advantage estimates at dataset points $(s,a) \\in \\mathcal D$, which are in-distribution (in fact, in-sample) and therefore more reliable than estimates at OOD points.\n\n(2) No need for precise advantage: The purpose of using advantage is to qualitatively distinguish between good and suboptimal actions, and exact advantage values are not required. The exponential form $\\epsilon \\exp(-\\alpha A(s,a))$ is a soft heuristic to bias the neighborhood radius. Our motivation does not assume or require the radius to be exactly proportional to the exponential of the advantage.\n\n**Reference**\n\n[1] Kakade et al. Approximately optimal approximate reinforcement learning. ICML 2002.\n\n[2] Schulman et al. Trust region policy optimization. ICML 2015.\n\n[3] Achiam et al. Constrained policy optimization. ICML 2017.\n\n[4] Li et al. When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning. ICLR 2023.\n\n[5] Asadi et al. Lipschitz continuity in model-based reinforcement learning. ICML 2018.\n\n[6] Todorov et al. Mujoco: A physics engine for model-based control. IROS 2012."}]

Author rebuttal:

Meta-review: The authors are addressing the important problem of constraining Q learning in offline reinforcement learning to stay within the data support and prevent extrapolation errors. They do so with a bilevel optimization scheme where the Q function is updated in the inner loop and the policy updated in the outer loop. They include a theoretical characterization of the extrapolation behavior of the Q function using NTK assumptions.

Strengths: The authors focus on an important problem of addressing extrapolation and regularization in the offline RL regime and have a novel and interesting approach. There are interesting theoretical results showing they are able to approximate the data support without requiring modeling the behavior policy directly.

Weaknesses: The method seems fairly complicated and analysis of how robust this method is compared to other, simpler offline RL methods is lacking. However, this has been addressed somewhat in the rebuttal with additional results based on hyperparameter sweeps shown. 

Reasons for spotlight: I think this is an interesting direction and use of theoretical tools for addressing the extrapolation issue in offline RL. While there are weaknesses to the method and analysis, I believe the novelty of the approach outweigh these.

Reviewers mainly had concerns about the stability of the bilevel optimization process, the sensitivity to hyperparameters, and clarity of the main contribution and readability of the paper. Most of these concerns were addressed in the rebuttal phase, and I encourage the authors to continue improving on clarity of the contributions for the camera ready.
Decision: Accept (spotlight)