Title: Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: offline reinforcement learning
Abstract: Offline reinforcement learning (RL) suffers from extrapolation errors induced by out-of-distribution (OOD) actions. To address this, offline RL algorithms typically impose constraints on action selection, which can be systematically categorized into density, support, and sample constraints. However, we show that each category has inherent limitations: density and sample constraints tend to be overly conservative in many scenarios, while the support constraint, though least restrictive, faces challenges in accurately modeling the behavior policy. To overcome these limitations, we propose a new neighborhood constraint that restricts action selection in the Bellman target to the union of neighborhoods of dataset actions. Theoretically, the constraint not only bounds extrapolation errors and distribution shift under certain conditions, but also approximates the support constraint without requiring behavior policy modeling. Moreover, it retains substantial flexibility and enables pointwise conservatism by adapting the neighborhood radius for each data point. In practice, we employ data quality as the adaptation criterion and design an adaptive neighborhood constraint. Building on an efficient bilevel optimization framework, we develop a simple yet effective algorithm, Adaptive Neighborhood-constrained Q learning (ANQ), to perform Q learning with target actions satisfying this constraint. Empirically, ANQ achieves state-of-the-art performance on standard offline RL benchmarks and exhibits strong robustness in scenarios with noisy or limited data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to tackle offline RL problems by enforcing neighborhood constraint rather than imposing density/support/sample constraints. Perturbations are added to action values $Q(s, \\tilde{a})$ from an adaptive neighborhood such that $||a-\\tilde{a}||\\leq \\epsilon\\exp(-\\alpha A(s,a))$. The authors formulated a bi-level optimization problem to first obtain $Q$ value from dataset actions then extract the policy from the perturbed Q value.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\\\nThe motivation of developing a new type of constraint beyond the listed three classes is appreciated. I find the proposed method interesting and has potential links to several existing approaches.\n\n**Weaknesses:**\\\nI found the paper quite hard to follow. Specifically, the first half of the technical sections (before Definition 5) and the second seem to be detached. I can understand that the motivation is to build a neighborhood set  $||a-\\tilde{a}||\\leq \\epsilon$ such that (1) their value/distributional difference can be controlled; (2) generalization capability is guaranteed. To this end, the authors introduced several lemmas and propositions. However, these lemmas are either from existing works or already well-known. For example, lemma 1 is a very standard result and can be seen from e.g. [Achiam et al. 2017].  lemma 2 comes from [43]. I guess there are some issues with proposition 1 because by definition $D_{TV} \\leq 1$ but the authors showed an upper bound depending on $|\\mathcal{S}|$, which seems to be of little use even for small-sized problems.\n\nThe second half also needs more elaboration: Theorem 1 defines covering numbers in terms of $\\mathcal{S}$. But shouldn't it be about actions since we care about neighbors and generalization capability of actions? If it is a typo and the theorem is indeed about actions, then for the assumption of \"independent and identically distributed samples from $\\nu$\" to hold, the policy $\\nu$ has to be fixed? \n\n\nDefinition 5 does not seem to be very well coupled with previous developments : the authors did not sufficiently motivate the radius  $epsilon\\exp(-\\alpha A(s,a))$ nor explained how it connects to the covering number theorem. More importantly, this quantity relies on accurate estimates of advantage in the offline context. But if we can have such accurate estimates free of OOD issues, in principle we don't have to employ this perturbation scheme?\n\nQUESTIONS:\nPlease refer to the weaknesses section for questions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to address the extrapolation error issue resulting from OOO actions in offline RL by restricting action selection for the Bellman target within some neighborhood of actions in the dataset. The authors provide theoretical guarantees for this choice of constraints. Practically, the authors propose an efficient algorithm by decomposing the objective into a bilevel optimization problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n- The paper is very well written and easy to read.\n- I particularly like the summary of different constraint types used in modern offline RL algorithms, the authors did an excellent job in summarizing and categorizing recent works with different constraint types, as well as addressing their strengths and limitations.\nThe idea of using neighborhood constraints is straightforward and yet powerful, the authors did a good job at motivating this.\n- I think the theoretical analysis is also one of the key strengths of the paper, with the authors showing through Theorem 1 that the union of sample neighborhoods can approximate the true support.\n- The bilevel optimization algorithm proposed is sound. Using the advantage function for the adaptive constraints is probably the most intuitive choice here and appeared to show good results. I agree with the authors comments in the limitations section that it is worth exploring different instantiations of adaptive neighborhoods in future works.\n- Experiments are very well done with an extensive set of environments and baselines, as well as very detailed comparisons in terms of performance and compute time with each. Empirical results are quite strong\n\nWeaknesses\n\n- The theoretical analysis as well as the practical algorithm assumes a deterministic policy, could the authors comment how/whether it is possible to extend to stochastic policies?\n- The algorithm itself seems to be quite sensitive to the choice of $\\lambda$ and $\\alpha$, however the authors are very up front about this and have performed extensive ablation studies on these hyperparameters, as well as giving nice interpretations on the values for $\\lambda$ and $\\alpha$.\n\nQUESTIONS:\nSee question above on stochastic policies.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of extrapolation errors in offline reinforcement learning (RL), which occur when evaluating out-of-distribution (OOD) actions not present in the static training dataset. The authors propose a novel neighborhood constraint that restricts action selection in the Bellman target to the union of neighborhoods around dataset actions, offering a flexible middle ground between overly conservative density/sample constraints and the computationally demanding support constraint (which requires accurate behavior policy modeling). The proposed constraint bounds extrapolation errors and distribution shift while approximating the least restrictive support constraint without requiring behavior policy modeling. The constraint is made pointwise adaptive by adjusting the neighborhood radius for each data point based on its advantage (quality). Empirically, ANQ achieves state-of-the-art results on standard offline RL benchmarks (D4RL), outperforming existing methods like CQL, IQL, and SPOT on both Gym locomotion and challenging AntMaze tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n* The paper introduces a neighborhood constraint that restricts Bellman targets to the union of neighborhoods around dataset actions. Theoretical analyses (Lemma 2, Proposition 1, Theorem 1) rigorously establish that the neighborhood constraint bounds extrapolation errors under mild continuity assumptions, approximates the support constraint without needing behavior policy modeling and\ncontrols distribution shift via Lipschitz continuity of transition dynamics.\n* The proposed bilevel optimization (inner maximization + outer expectile regression) efficiently enforces the constraint during Q-learning.\n* ANQ achieves top-tier results on D4RL benchmarks, including Gym locomotion and challenging AntMaze tasks. Robustness is validated in noisy data and limited data scenarios, where ANQ outperforms density (CQL), sample (IQL), and support (SPOT) constraints.\n\n## Weakness\n* The performance of ANQ might be sensitive to the choice of hyperparameters. A more detailed analysis of hyperparameter sensitivity would be beneficial to understand the robustness of the method.\n* Some new sota support/sample constraint baselines are suggested such as [1][2].\n\n\n[1] Iteratively Refined Behavior Regularization for Offline Reinforcement Learning\n\n[2] Supported trust region optimization for offline reinforcement learning\n\nQUESTIONS:\n* Could u please provide an additional ablation that changes the $\\mu_{\\omega}$ to a random variable sampled from gaussion?  As the performance improvement could also be derived from the Q-landscape smoothness despite theoretical supports.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Adaptive Neighborhood-Constrained Q Learning (ANQ), a method designed to mitigate extrapolation bias in offline reinforcement learning (RL) by applying an adaptive neighborhood constraint to restrict action selection within a set of actions that are near the ones seen in the dataset. This constraint is enforced using a bi-level optimization framework, where the inner optimization updates the Q-function and the outer optimization updates the policy parameters.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The introduction of an adaptive neighborhood constraint is a novel and effective approach for balancing conservatism and exploration in offline RL. \n\n2. The method addresses a significant problem in offline RL, providing a flexible solution to extrapolation bias without the need for complex behavior policy modeling.\n\n3. The paper is generally well-written, with clear motivations, method descriptions, and results. The authors effectively explain the problem of extrapolation bias in offline RL and introduce the adaptive neighborhood constraint as a solution.\n\n**Weaknesses:**\n\n1. Bi-level optimization introduces significant complexity to the method. Since each mini-batch requires four separate updates to the parameters, the computational cost becomes expensive. This results in increased training time per iteration, which could be a significant drawback, especially in high-dimensional or real-world applications.\n\n2. The instability introduced by the bi-level optimization process is not fully explored. Given that each mini-batch requires multiple updates to the parameters, this could lead to issues with convergence or training stability, particularly in environments with high variance or limited data. \n\n3. A more detailed discussion of the potential instability and its mitigation would provide clearer guidance for users and researchers adopting this method.\n\nQUESTIONS:\n1. The bi-level optimization framework introduces significant complexity by requiring four separate updates per mini-batch. This could lead to instability or slow convergence, especially in environments with high variance or limited data. Could the authors provide a more detailed analysis of how this affects training stability and convergence in practice? Are there any strategies for mitigating potential instability, particularly when training with limited data or in environments with high-dimensional state-action spaces?\n\n2. Given the increased complexity of bi-level optimization, does the method show diminishing returns as the training progresses? How does the computational cost of this approach compare to other state-of-the-art offline RL methods, especially in terms of training time and memory usage? Could the method be optimized further to reduce the computational burden?\n\n3. The Lagrange multiplier λ and the inverse temperature α play important roles in controlling the neighborhood radius and enforcing the constraint. Could the authors provide more insight into how these hyperparameters are tuned in practice? Does their choice significantly impact training stability or the final policy performance? Is there an automatic way to tune these parameters based on the environment or data?\nScalability to Complex and High-Dimensional Tasks:\nWhile the method performs well in standard benchmark tasks, how does it scale to high-dimensional action spaces or environments with partially observable states? Does the method retain its robustness and performance in such scenarios, or are modifications needed to accommodate the complexity of real-world tasks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}