Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
Authors: Tan Jing
Xiaorui Li
Chao Yao
Xiaojuan Ban
Yuetong Fang
Renjing Xu
Zhaolin Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: Offline Reinforcement Learning
Policy Constraint
Behavioral cloning
Primary area: reinforcement learning
TL;DR: We dynamically adjust the policy constraint scale during the training process, enabling the algorithm to adapt to various datasets without manual tuning and surpass SOTA offline RL algorithms

Abstract: Offline reinforcement learning (RL) enables learning effective policies from fixed datasets without any environment interaction. Existing methods typically employ policy constraints to mitigate the distribution shift encountered during offline RL training. However, because the scale of the constraints varies across tasks and datasets of differing quality, existing methods must meticulously tune hyperparameters to match each dataset, which is time-consuming and often impractical. To bridge this gap, we propose Adaptive Scaling of Policy Constraints (ASPC), a second-order differentiable framework that automatically adjusts the scale of policy constraints during training. We theoretically analyze its performance improvement guarantee. In experiments on 39 datasets across four D4RL domains, ASPC using a single hyperparameter configuration outperforms other adaptive constraint methods and state-of-the-art offline RL algorithms that require per-dataset tuning, achieving an average 35\% improvement in normalized performance over the baseline. Moreover, ASPC consistently yields additional gains when integrated with a variety of existing offline RL algorithms, demonstrating its broad generality.

Published as a conference paper at ICLR 2026
ADAPTIVE SCALING OF POLICY CONSTRAINTS FOR OFFLINE REINFORCEMENT LEARNING
Jing Tan
University of Science and Technology Beijing
Beijing, China Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
Xiaorui Li, Chao Yao, Xiaojuan Ban Zhaolin Yuan' University of Science and Technology Beijing Beijing, China mhanzhaolin@ssta edu.cc.eeducc....
Yuetong Fang Renjing Xu*
Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
renjjingxuehkust gz edu.
ABSTRACT
Offline reinforcement learning (RL) enables learning effective policies from fixed datasets without any environment interaction. Existing methods typically employ policy constraints to mitigate the distribution shift encountered during offline RL training. However, because the scale of the constraints varies across tasks and datasets of differing quality, existing methods must meticulously tune hyperpa- rameters to match each dataset, which is time-consuming and often impractical. To bridge this gap, we propose Adaptive Scaling of Policy Constraints (ASPC), a second-ordes differentiable framework that automatically adjusts the scale of policy constraints during training. We theoretically analyze its performance improvement guarantee. In experiments on 39 datasets across four D4RL domains, ASPC using a single hyperparameter configuration outperforms other adaptive constraint methods and state-of-the-an offline RI. algorithms that require per-dataset tuning, achieving an average 35% improvement in normalized performance over the bascline. More- over, ASPC consistently yields additional gains when integrated with a variety of existing offline RL algorithms, demonstrating its broad generality.
INTRODUCTION
Offline reinforcement learning (RL) learns policy exclusively from fixed, pre-collected dataset without further interactions with the environment Levine et al. (2020). This characteristic is particu- larly crucial in real-world applications such as autonomous driving El Sallab et al.) (201 7b; Kendall et al. (2019). healthcare Prasad et al.) (2017); Wang et al. (2018), industry Zhan et al. (2022); Yuan
al. 2024), and other tasks, where interacting with the environment
be expensive and risky.
Despite the potential advantages, a critical challenge offline RL is the distribution shift Levine et al. (2020) between the offline data and the training policies, often leading suboptimal or even invalid policy updates. Many methods have been proposed to mitigate the adverse effects of the distribution shift. A common strategy
(2019); Kumar et al. (2020); Fujimoto & Gu Gul 2021);; Kostrikov et al. (2022). ensuring that the learned policy remains close the behavior policy used collect the dataset. By imposing constraints on policy
impose explicit implicit policy constraints Fujimoto
updates, these methods can effectively mitigate the extrapolation error of the Q value Fujimoto et al.)
(2019) induced by the distribution shift while offering certain performance guarantees.
A central but often overlooked issue in policy constraint methods is the choice of the constraint scale, which crucially governs the balance between the RL objective and the behavior cloning (BC) term. Existing approaches fall into two categories. First, methods that rely dataset-specific hyperparameter tuning can achieve strong results, but their performance collapses once a single
"Corresponding authors

---

Published as a conference paper at ICLR 2026
configuration is applied across tasks or datasets of varying quality, shown in Figure Tb). Second, adaptive variants with fixed hyperparameters/ Peng et al. (2023); Yang et al. (2024) alleviate tuning costs, yet they only reweight actions locally and neglect the global trade-off scale, leaving a significant gap to carefully tuned baselines. In practical offline RI,, where extensive tuning is prohibitively expensive even infeasible, the pressing challenge is how achieve robust performance with single hyperparameter configuration across diverse datasets.
Retable
and cilcttve
Pesces De Sieeece
RiBRAC
Uniilsble collopod
Canservaive
190
Figure I: (a) The RL-BC trade-off in offline RL. ASPC dynamically balances RL and BC, yielding reliable and effective policy (left). Existing methods fail to properly calibrate this trade-off, resulting in suboptimal or collapsed policies (middle and right). (b) Percent difference in performance for ReBRAC and A2PR under a single hyperparameter setting across all datasets. HC HalfCheetah, Hop Hopper, W Walker, random, medium, medium-replay, me medium-expert, Um = umaze, M = medium, I. = large, = diverse, p= play.
To enable single hyperparameter configuration to match or exceed the performance of finely tuned methods across datasets of varying quality and tasks, we propose an adaptive scaling of policy constraints (ASPC) approach that dynamically adjusts the constraint scale during training. The intuition of this method is shown in Figure [a). Our approach leverages a second-order differentiable optimization framework Fiinn et al. (2017) 7) balance the goals RI. and BC. Specifically, we parameterize the scale factor a learnable parameter that balances the RL objective LRL and the BC objective CBC in TD3+BC Fujimoto & Gu (202I). The combined objective C is given by
= a: LRL + LBC,
(1) During training, is dynamically adjusted by
For the full definitions of a, refer to equation constraining the rate of change of the Q-value and the BC loss, enabling the algorithm to discover more stable learning path and exhibit remarkable adaptability across tasks and datasets.
B
We theoretically analyze the performance improvement guarantee ASPC and extensively evaluate on the D4RL benchmark Levine et al. (2020). Our empirical results demonstrate that ASPC outperforms other state-of-the-att offline RL algorithms that depend on meticulously tuned hyper- parameters for each dataset, while adding only minimal computational overhead the original TD3+BC backbone. In addition, ASPC improves a variety of offline RL algorithms beyond TD3+BC, further indicating its generality and broad applicability.
2 RELATED WORKS
2.1 OFFLINE RL
Offline RL aims learn policies purely from static datasets and suffers from distribution shift between the behavior policy and the learned policy, leading to value overestimation and policy collapse. Existing approaches address this challenge from several perspectives. Policy constraint methods explicitly FFujimoto et al. (2019); Fujimoto & Gu (202T) or implicitly Kumar et al. (2020); Kostrikov et al. (2022) regularize the learned policy toward the behavior distribution. Uncertainty- aware approaches penalize actions with high epistemic or aleatoric uncertainty An et al. (202T); Bai et al. 2002);; Zhang et al. (2023). Sequence modeling methods reformulate RL as conditional trajectory modeling using transformers Chen et al. (2021); Janner et al. (2021). Among these, policy constraint methods have emerged as the most direct and widely adopted solution, but their effectiveness crucially depends properly scaling the constraint. This motivates focus developing an adaptive scaling mechanism that eliminates the need for par-dataset tuning while retaining robustness across diverse offline RL benchmarks.
2

---

Published as a conference paper at ICLR 2026
2.2 ADAPTIVE POLICY CONSTRAINTS
Balancing the RL objective against BC is central to offline RL, and the strength of this constraint critically affects both stability and performance. Recent work has explored adaptive ways tune this balance. Trajectory- or sample-weighting methods such as AW Hong et al. (2023), wPC et al. (2023), and OAP et al. (2023) reweight transitions or actions based on estimated value or expert preference, thereby adjusting constraint strength locally. Other approaches introduce auxiliary models to refine constraint scaling, for example PRDC Ran al. (023). GORL Yang et al. (2024), A2PR Liu et al. (2024), and Liu & Hofert (0024). Despite these advances, current approaches either rely on per-dataset hyperparameter tuning for optimal performance, or apply a fixed configuration that yields only limited gains across domains. Our ASPC method addresses this gap by dynamically adjusting the constraint scale during training, enabling robust performance across diverse datasets with single hyperparameter configuration.
PRELIMINARIES
RL problems are formulated as a Markov decision process (MDP), described by the tuple (S, A, P. R. 7). The set of states is S, the set of actions is A. the transition probability function is P(s' a). the reward function R(s, a), and 7 E [o, 1) is the discount factor. The objective is to find a policy 5 - A that maximizes the expected discounted return. This objective is equivalently expressed as maximizing the Q-value Q (sssa) under n, given by:
(s, E Σ S R(ss, a.)
3,00 =
(2)
where and represent the state and action at time t. In practice, RL algorithms update Q-values using the Bellman equation as an iterative rule, seeking converge to the optimal policy x².
A central challenge for offline RL is the distribution shift. When state-action pair (s, a) lies outside the dataset D, directly optimizing the Q-function may cause severe over-estimation. One remedy is to constrain the target policy to stay close to the behaviour policy TT. TD3+BC FFujimoto & Gu (2021)
does so by solving:
π arg max E B(R.a)D(A> a ((s))(()) (a(s) -
a
Σ, (ss. a))
RL
BC
normalizes the RL term to the scale of the BC loss. In vanilla TD3+BC, a is fixed constant. Instead of keeping the scale factor a static, we update throughout training.
4 METHOD
We now present the ASPC algorithm in detail. We begin by introducing its core framework, second- order differentiable optimization that adaptively balances the RL and BC objectives (Section 4411) We then provide theoretical analysis (Section 4.2), which explains the role of the mutual constraint term and establishes single-step and long-term performance guarantees. Finally, we describe practical instantiation of ASPC built on TD3+BC (Section 4.3), which enables its application to standard offline RL benchmarks.
4.1 ADAPTIVE SCALING OF POLICY CONSTRAINTS
To adaptively adjust the relative scaling between the RL and BC objectives, ASPC adopts a meta- learning approach Finn et al. (2017); Franceschi et al. (2018). It converts the scale factor in equa- tion 3 into a learnable parameter and optimizes dynamically via bilevel training, utilizing inner updates and outer updates to maximize RL exploration near the behavior policy.
Inner Update To optimize the policy under offline data, we define the inner objective as
Linner (0; a)) = E((.a)DD X(a) (s,, To()) + [[T)(s)) - all2]
(4)
3

---

Published as a conference paper at ICLR 2026
where X(a) /e///////l/ The inner update is then obtained via gradient descent step with learning rate T)::
((a) = 0 1/0 VoLiner (0;
(5)
and ((a) denotes the updated policy parameters after one inner step.
Outer Update While the inner update optimizes the poliey parameters 0 for given scale a, the outer update is responsible for adjusting a itself so as to dynamically balance the RL and BC objectives. The outer loss is composed of three coordinated components. Li mirrors TD3 BC and steers toward a better balance between RL and BC. Le penalizes abrupt increases in the expected Q-value, while L3 constrains large shifts in the BC loss. Together, L2 and L3 adaptively regulate the step prescribed by L1, preventing either RL or BC from dominating and thereby stabilizing training. Formally, we write:
EAND Q(s,TT(88))
BD [|T -all))
L1
(6)
E (Q((s
C2 EEND (s,) EEDD/ ( | (oss)o(s))) mo(ss)
Ca (L2.detach)
sup (a.a)ED
[TT - all²) |
sup
[| - (112-1(a((s) al²1).
(8)
(a,0)ED
The outer objective is
Couter (B(a)) = l2
Here, Ty and TT denote the policies before and after the inner update, respectively. ,detach indicates stopping gradients. While L1 and C2 are relatively standard, the design of L3 requires clarification. Theoretically, its form follows directly from Theorem with details in Appendix AA.3 Intuitively, L3 combines three factors: the rate of change Q- Q-values, the upper bound of the BC loss, and the variation in BC loss across iterations. Large Q-value fluctuations or high BC-loss bound signal rapid policy change or significant deviation from the behavior policy. In such cases, strengthening the penalty on BC variation helps suppress distributional shift and stabilize training, consistent with intuition. To update a, treat the inner update parameters ((a)
implicit function of a and use second-orde order derivatives. Lets 7a be the learning rate of a. The gradient-desee descent step is
@oouter (((a)) 00(a)
a + a Tla
(10)
00
8c
4.2 THEORETICAL ANALYSIS
We now analyze the theoretical properties of ASPC. We show that the outer objective ensures stable updates and reduces the gap to the optimal policy.
Assumption 4.1. The critic Q(s, a) and the transition kernel P(- s,a) are Lipschitz continuous with respect to the action variable. That is, there exist constants LQ, Lp >> O, independent of s, such that for all E S and all d1, 02 A.
[1Q(s, a1) ((..02)) S Lo |a- a2ll. 8, a.) - P(. 8202) S Lpllai a2ll (11)
Proposition 4.2 (Mutual constraints between TTBC and (AQ)²). Under Assumption 4.1 the change in BC loss BC) and the squared change in Q-values ((AQ)2) mutually constrain each other: (AQ)2 provides lower bound on ALBC, while ALBC provides an upper bound on (AQ)?
This result shows that the two penalties in equation 7 land equation & are inherently coupled rather than independent. It explains why in practice some tasks succeed with only one of them, while others require both for stable training (see Section 5.5). The detailed proof is provided in Appendix A.
Proposition 4.3 (Single-step performance lower bound). For the update step from T4 to T++1, the performance improvement admits the following lower bound:
- -
1-7 |Q
1
((AL BC
22)
(12)
BC 22) is a nonnegative function depending on the BC-loss variation upper bound where ((ALCC
A BC and the BC-loss upper bound 2
4

---

Published as a conference paper at ICLR 2026
This proposition serves the theoretical basis for Theorem 4.4 It also directly motivates the design of the penalty term C3 (equation 8 whose form is derived from bounding ( BC 200) The
detailed derivation of Φ is deferred Appendix A.2
Theorem 4.4 (Single-step performance condition for ASPC). An idealized ASPC update that satisfies the condition AQ 2 + leads to non-decreasing policy performance: J( J(T,) 2 0.
ASPC employs a smooth relaxation this condition via the outer objective, which is designed to guide updates toward this provably stable regime. The detailed proof is given in Appendix AA.3
Theorem 4.5 (Performance gap to optimal). With Theorem 4.4 after T iterations when the single-step gain vanishes (ôr 0), the gap to the optimal policy satisfies:
( ((T) I (((s)) Smin:
(13)
where (EE) is a function of the mismatch εB between the behavior policy and the optimal policy, and min denotes the minimal single-step improvement before convergence.
This theorem shows that ASPC progressively reduces the suboptimality gap until convergence, where the remaining gap is controlled by L (Ea).. The full derivation of (E is given in Appendix A.4
Algorithm Adaptive Scaling of Policy Constraints
Initialize: critic and actor networks, scale factor a, replay buffer D, update intervals kzs ka.
1: for 1 to N do
2: Critic update:
3: Sample minibatch from D: Compute TD targets and update critic networks;
4:
if é mod h = 0 then
5:
Actor update (inner):
6:
Compute Linerr (0; a.) by equation 4 Compute A(a.) by equation
7:
Update actor networks;
8:
if mod (ki, = then
update (outer): 10: Compute Couter (0(a)) by equation 9 Update via equation 10
9:
11: end
12: Soft update critic and actor networks;
13: end
14:
end for
4.3 IMPLEMENTATION ON TD3+BC
To make ASPC practical, we instantiate on top of the TD3+BC backbone with only mod- ifications: (i) redesigned critic network, and (ii) a learnable scale factor All other network components and hyperparameters remain unchanged. See Appendix B.2 for full specification.
Recent studies show that deeper critics Kumar et al. (2022): Lee et al. (2022) and the insertion of LayerNorm between layers Nikulin et al. (2023): Ball al. (2023); Tarasov et al. (2024a) can mitigate Q-value over-estimation and improve stability. Following this evidence, we extend the TD3+BC critic from two to three hidden layers and insert LayerNorm after each layer. An ablation of this
choice is provided in Section 5.5
Algorithm Distts the ASPC procedure. Blue highlights indicate lines that differ from the TD3+BC backbone. Although second-order gradients increase cost, we set the a-update interval ka far longer than the actor-updaa update interval kz which maintains performance while sharply reducing runtime. Section analyses this trade-off in detail.
EXPERIMENTS
In this section evaluate ASPC the D4RL benchmark. Section55.1 compares ASPC with strong baselines to demonstrate its adaptability and overall effectiveness. Section 5.2 analyzes the learning curves of a during training, further illustrating ASPC's adaptive behaviour. Section (53 investigates
5

---

Published as a conference paper at ICLR 2026
Table 1: Average normalized score over the final evaluation across four random seeds. The best performance in each dataset is highlighted in bold, while the second-best performance is indicated with an underline. Blue shading indicates methods with top domain average performance. The symbol + denotes the standard deviation. / denotes fixed hyperparameters, whereas Xdenotes dataset-specific ones. TTo ensure fairness, TD3+BC and wPC employ the robust critie described Section 5.5
Task Name
(V)
A2PR(/)
IQL(X)
wPC'(V)
ReBRAC(X)
ASPC (Ours)(V)
HaltCheetah
Random
10.6 :: 0.7
21.1 1 0.8
19.5 0.8
18.8 0.7
29.5 1 1.5
20.8 0.9
Medium
4.2
56.1 0.3
50.0
54.8 0.2
65.6
1.0
58.7 0.4
Expert
10044 0.4
99.9 3.2
95.5
103.8
105.9 - 1.7
105.1 1.2
Medium- Expert
Mepium-Expert
97
98.9
Medium-Replay
45.8
49.0
4..1
=8.1
$1.0 0.8
50.6
Full-Replay
7
79.5 1.5
7.70
76.7
82.1
79.3 1
Random
8.6
8.5
Medium
6..0
78.3
6.22
81.8 9.8
+
92.7 7.2
Hopper
Expert
108.2
83.9
108.8
26.6
IT2.3 0.4
Medium- Expert
1.3.3499.
110.8 2.6
85.5 29.7
109.1 +.5
107.0
111.0 2.1
Medium-Replay
47.4
98.9 2.0
89.6 13.2
100.8 0.7
98.1 5.3
101.3
Full-Replay
90.3
97.1 17.8
104.4 10.8
105.6 0.6
107.1 1.4
107.2
Random Medium
5.5
1.5
11.3 7.0
12.5 1: 10.6
18.4 4.5
15.6 6.4
Walker2d
6.00
84.2 4.7
80.7 3.4
89.6 0.3
82.5
92.4 5.4
Expert
1.8.2:1:..2
96.9
Medium- Expert
1-3.3+49.2
88.2 18.2--4003 40.7
112.1 0.0.5
110.1 + 0.5
111.6 0.3
111.1 + 0.3
Medium-Replay
76.6 12.7
84.5 12.3
75.4
93.4 3.0
77.3 7.9
97.6
Full Full-Replay
88.3 11.7
102.5
97.5
99.5
102.2 1.7
102.1 0.2
MuJoCo Avg
77.8
Maze2d
Umaze
13.9
102.5
6.1
13.8
106.8
128.1 31.8
Medium
63.3 63.3
90.4 29.6
34.8 1: 2.7
87.4 1: 48.7
105.1 1: 31.6
117.8 17.3
Large
108.9 = 43.6
177,2 = 34.2
61.7 3.5
123.3
70.5
78.3 = 61.7
195.8 31.3
Mazeld Avg
68.9
123.53
46.2
94.6
96.7
147.2
Umaae
100.0 + 0.0
92.5 8.3
83.3 4.5
97.5 5.0
97.8 1.0
92.5 5.0
AntMaze
Umazze-Divers
87.5 17.5.:12.55
32.5 34.9
3.7
88.3
13.0
92.5 9.5
Medium: Play
7-5
40.0 7.1
64.6 4.9
85.0 5.7
84.0
+ 4.2
K5.0 + 12.9
Medium-Diverse
12.5
40.0 25.5
61.7
76.3
13.5
70.0
Large Play
2.5 5.00
5.0 8.7
42.5 6.5
65.0 1: 19.1
60.4
26.1
55.0
5.7
Large-Diverse
2,5 + 5.0
22.5 + 14.8
27.6 7.8
65.0 + 10.0
$4.4 +: 25.1
$2.5 + 18.9
AntMaze Avg
35.4
38.75
58.3
78.7
76.8
74.5
Human
53.8 + 15.7
2.10000
81.5 17.5
39.9 + 12.8
103.5 + 14.1
81.1 8.8.1
Pen
Cloned
71.7 21.5
6.0
77.2 17.7
91.8 21.7
87.2
Expert
126.6 = 24.8
$1.5
133.6 = 16.0
141. 11.8
154.1 = 5.4
141.2 1: 9.4
Door
Human
0.0 0.00+100 0.0
4.2
3.1 2,0
-0.2 0.0
0.0 + 0.0
0.0 + 0.0
0.0
0.0
Expert
81.6 16.3
0.0
2.8
51.4 55.3
104.6
105.6 = 0.4
Relocate Hammer
Human
0.0 0.0
1.110.4
2.5 1.9
0.0 001:0.1
0.2 0.2
2.2 1: 3.2
Cloned
Expert
132.8 0.4
0.3
129.6 71.5
$7.6
133.8 1: 0.7
128.6 1: 0.4
Human
0.0 0.0
0.0
0.1
0.1 0.0
0.0 4.0 0.0
0.1 : 0.2
Cloned
Expert
90.6 18.2
0.0
106.5 +: 2.5
6.7 4.6
106.6 3.2
111.2 = 2.4
Adroit Avg
4.65
$3.4
28.8
58.6
55.7
Total Avg
$7.7
51.2
62.6
64.2
74.8
77.9
soore wore 2
1.00
relocate
halfehectah
Phopper
TD3-BC
Frociin tunu with
0.75
A2PR
walkerrd
0.50
WPC
ReRRAC
0.25
dooc
Inaae2d
0.00
D4RL Normalizad Soore
50
160
150
200
pen Average scoce
Figure Left: performance profiles on 39 datasets of D4RL. Right: radar chart of the mean performance across the nine tasks.

---

Published as a conference paper at ICLR 2026
the necessity of dynamically adjusting (.44iion554/rrpone runtime results to highlight the efficiency
of ASPC. Section |5.5 presents ablation studies on the key components of ASPC. (cction[5.66poove results on integrating the ideas of ASPC with other methods, and Section 5.7 presents the performance of ASPC on the OGBench benchmark.
5.1 COMPARATIVE PERFORMANCE ON BENCHMARK
We evaluate ASPC on 39 datasets spanning four D4RL domains Levine al. (2020): MuJoCo (v2),
AntMaze (v2), Maze2d (v1), and Adroit (v1). Our baselines include TD3+BC Fujimoto & Gu (001))
and IQL Kostrikov et al. (2022) standard policy-constraint methods. wPC Peng et al. (2023)
(SOTA) adaptive policy constraint methods built on TD3+BC. ReBRAC Tarasov et al. (2024a) integrates multiple performance- boosting components into TD3+BC and has achieved SOTA results across a wide range of datasets. TD3+BC, wPC, A2PR, and ASPC are all set as the single hyperparameter set, whereas IQL and ReBRAC rely on dataset-specific hyperparameters found via grid search. We reproduce results for TD3+BC, wPC and A2PR. IQL and ReBRAC results are taken from Tarasov et al. (2024a b). Complete experimental details for each
and A2PR[Liu et al. (2024) are statc-of-the-
algorithm are provided in the appendix B.2
The performance comparison is summarized in Table [ ASPC achieves the best performance on MuJoCo and Maze2d, and exhibits competitive results on Adroit and AntMaze. Most notably, ASPC attains SOTA performance on average across all four domains, which not only outperforms other adaptive policy constraint methods but also surpasses approaches that rely meticulous per-dataset hyperparameter tuning, highlighting its remarkable adaptability. Figure FigureZ|shows shows that the performance profile curves (left) place ASPC above all baselines for almost every threshold, and the min-max- scaled radar chart (right) gives ASPC the largest, most balanced polygon, visually confirming its strong and stable performance across tasks without per-dataset tuning.
medium-rcelly-42
modium-v2
medium-cxpent-42
expert-v2
halfcheetah
hopper
walker2d
20-
10
20-
Alpha
2-
0.0
0.5
1.0
0.0
0.5 Train Steps IM
1.0
0.0
1.0
Figure 3: Learning curves of a on halfcheetah, hopper, and walker2d across datasets of different quality. Higher-quality datasets yield smaller a (favoring BC), while lower-quality ones yield larger a (favoring RI... a is initialized to 2.5.
5.2 ADAPTABILITY OF THE SACLE FACTOR
Dataset Adaptability Figure shows the evolution of on HalfCheetah, Hopper, and Walker2d for four dataset quality levels, listed from highest to lowest expert, medium-expert medium, and medium-replay Across all three tasks, higher-quality datasets lead to smaller a, which places more weight on BC, whereas lower-quality datasets lead to larger a, shifting the emphasis toward RL. The consistent ordering confirms that ASPC automatically adjusts the policy- ponicy-constraint scale to dataset quality without any per-dataset hyperparameter tuning.
Task Adaptability Figure4ploos the trajectories on six heterogeneous tasks. Tasks such as door, pen, hammer, and relocate possess narrow expert data distributions; here a settles near 10-1, giving greater weight to BC. Conversely, antmaze and maze2d, whose datasets contain highly sub-optimal trajectories, drive above 10, shifting emphasis to RL. This task-aware scaling requires manual tuning and highlights ASPC's cross-task adaptability.
Training Adaptability Combining the curves from Figures 4 and 3 we observe common learning dynamic: & first drops (or rises only slightly) during the early training phase, indicating greater reliance BC when the policy is still immature. As learning progresses and the critic stabilises, a gradually increases, handing more control to RL. This smooth, stage-wise adjustment underpins ASPC's stable convergence across tasks and datasets.
7

---

Published as a conference paper at ICLR 2026
unmmae umaze x2
hammer expert-v]
pen expert reloeate-xxpenev1
doce-cxpet-vl
mazeld ummzzevv
10
10"
10 0.0
0.5 Steps
1.0
Figure 4: Learning curves on six different tasks. The algorithm automatically adjusts a based on the task characteristics. The y-axis is shown in logarithmic scale for better visualization.
5.3 NECESSITY OF DYNAMIC SCALE FACTOR ADJUSTMENT
As shown in Table [ the hyperparameters meticulously selected via grid search ultimately underper- form compared to the ASPC algorithm, which dynamically adjusts hyperparameters during training. This observation raises the question: grid search simply failing to find the best setting, is the dynamic adjustment in ASPC the true source its advantage? To answer this, we conduct three controlled tests. Naive o.. TD3+BC is with fixed scale factor 2.5. Converged a. TD3+BC run with a fixed to the final value reached by ASPC on the same dataset. Linear a. TD3+BC starts from 2.5 and linearly interpolates to the above converged value the training horizon. To ensure fairness, all TD3+BC variants utilize the same robust critic architecture as ASPC, comprising three hidden layers, each followed by a LayerNorm.
Table 2:
Results under different a settings. Values in parentheses indicate the percent difference from Naive. Blue denotes improvement, and red denotes degradation.
Domain
Naive
Converged
Linear
Dynamic (ASPC)
Mujoco
70.3
79.3 (+12.8%
77.0 (19.5%)
82.1 (116.8%)
Maze2d
61.9
133.2 (1115.2%)
103.3 (166.9%)
147.2 (1137.8%)
AntMaze
28.7
64.1 (112333%
56.3 (196.2%)
74.5 (1 159.2%)
Adroit
49.9
49.1 (1116%%)
47.6 (14.6%)
55.7 (111.6%)
Total Avg
57.0
71.8 (125.9%)
66.7 (117.0%)
77.9 (136.6%)
Table 2 summarises the mean normalised scores the four D4RL domains. Percentages blue report the relative gain over the naive baseline that fixes 2.5. Converged a and Linear both outperform the naive setting, which confirms that the value to which ASPC eventually converges is much more appropriate scale for the policy constraint. ASPC (Dynamic a)) still exceeds the Converged variant by wide margin, and the Linear schedule closes only part of the gap. These results show that simply finding good fixed is not enough. Adapting the scale throughout training is essential for the best performance. ASPC provides this dynamic adjustment automatically and therefore achieves the highest overall score.
5.4 RUNTIME ANALYSIS
ASPC employs second-order gradient computations for updating a, which increases cost. However, its update interval (ka)) can be set substantially longer than that the actor (kz), thereby minimizing the additional computational overhead. To evaluate runtime efficiency, we compare the execution time of one million iterations of ASPC against that of other baseline algorithms. Figure(3] presents bar chart comparing the runtime of ASPC against TD3+BC, CQL, IQL, wPC and A2PR the halfcheetah- siticheetth-mmiium---t dataset. The results indicate that ASPC introduces only a minimal additional computational overhead beyond that of TD3+BC.
We further analyze the relationship between ka, runtime, and performance, as illustrated in Figure 5 The baseline setting for Ka is 10, we observe that reducing ka does not lead to significant performance degradation. This suggests that ASPC effectively captures the correct gradient optimization direction, maintaining robustness even when the gradient step size is large. When hr is set 30, the runtime
8

---

Published as a conference paper at ICLR 2026
Run Time (mij)
200
164
110
27,9
26.8
-25.5
100
02
WPC
ASPC
(8)
(b)
Figure 5: (a) Runtime comparison of different algorithms. (b) Runtime and average performance under different a-update intervals (ka). ASPC introduces only minimal overhead compared to TD3+BC, and increasing the update interval reduces runtime while maintaining high performance.
is nearly identical to that of TD3+BC while maintaining strong performance. This highlights the efficiency of the ASPC algorithm.
5.5 ABLATION STUDIES
Robust Critic(RC) When using the original TD3+BC critic network (with two hidden layers and no LayerNorm), during the process of adjusting Q-values exhibit significant instability, frequently leading to overestimation, causing catastrophic failure of the algorithm. Since wPC is also designed based on the original TD3+BC framework, we include it in our experiments related to RC (with three hidden layers and LayerNorm). Figure 6a presents the experimental results. The results indicate that when RC is not utilized, both wPC and ASPC achieve limited performance improvement and even exhibit performance degradation on certain tasks.
TD3+BC
Differncee
WPC (wia RC)
Difference
WPC WIRC
-00-
ASPC w/RC
-10
E1
-00-
C2
MuloCo
Mase2d
AntMaree
Adriit
Total
MuloCo
Maze2d
AntMaze
Adnoit
Total Avg
(a)
(b)
Figure 6: (a) Percent difference relative to the baseline TD3+BC (w/o RC (critic with three hidden layers, each incorporating LayerNorm).. (b) Percent difference of outer loss variants equation relative to the full ASPC configuration.
Loss Function Figure reveals clear, domain-dependent effects when the regularization terms are added to the base loss Z1. Adding neither term (L, only) gives the poorest performance. Introducing only C2 lifts performance in MuJoCo and Adroit to the level of full ASPC, while leaving AntMaze almost unchanged. Conversely, adding only L3 significantly boosts AntMaze but has little effect on MuloCo or Adroit. For Maze2I,, neither single term suffices. Only the full loss L + L2 + L3 attains the best result. These results be explained by Proposition 4.2 which shows that La and L3 implicitly constrain one another. Consequently, adding C2 in MuJoCo and Adroit implicitly bounds AL BC well, the single-step performance guarantee of Theorem 4.4 already satisfied. Conversely, in AntMaze a direct L3 penalty implicitly limits (aQ)², again meeting the theorem's 's lower bound. For Maze2D, however, neither implicit relation is strong enough; both La and L3 must be enforced explicitly for the condition in Theorem 4.4 to hold.
5.6 EXTENDING ASPC TO OTHER OFFLINE RL METHOPS
Many offline RL algorithms follow the form of equation] [ To evaluate the generality ASPC, we integrate its adaptive policy constraint into three representative baselines, including IQL, CQL, and Diffusion-QU Wang et al. (202).. Each method contains a hyperparameter analogous to a that controls the balance between value learning and conservatism. We replace this manually tuned coefficient with a learnable parameter and update it using the same bi-level second-order procedure ASPC. The detailed objectives for each algorithm provided in Appendix

---

Published as a conference paper at ICLR 2026
Table Performance on Gym-MuloCo datasets. +ASPC denotes the bascline combined with ASPC, and the percent change indicates its relative improvement over the baseline.
Gym-MuJoCo
IQL
+ASPC
CQL.
+ASPC
Diffusion-QI.
+ASPC
halfcheetah-meedune
50.0
48.4
(13.2%)
46.8
56.3 (120.3%)
51.5
(91.00))))
halfcheetah- wwwrrah-edim-m-ooe expert
92.7
94.4 (11.8%
94.2
(10.6%)
96.8
96.7 (J0.1%)
srrcheetah-mechum-rrpp
42.1
44.4 (15.5%)
45.3
(11.69))
47.8
58.2 (121.8%)
hopper-medium
65.2
61.4 (15.8%)
61.3
71.6 (116.8)
90.5
101,0 (111.6%)
hopper-mediim-cxpert
85.5
100.2 (117.2%)
90.1
106.9 (118.6%)
111.1
111.1 (10.0%)
hupper-mediiun-rplaa
89.6
88.3 (L1.4%)
77.5
(13.1%))
101.3
100.4 (10.9%)
walker2d m-ddiiu
80.7
83.9 (14.0%)
82.6
83.8 (11.5%)
87.0
80.3 (J7.7%%
expperr
112.1
112.1 (10.0%)
109.1
109.7 (10.6%)
110.1
110.5 (10.4%)
walker2d-medium-eee
75.4
77.5 (12.8%)
74.5
81.7 (19.7%)
95.5
95.2 (J0.3%)
Average
77.0
79.0 (12.5%)
75.7
81.6 (17.8%)
88.0
90.3 (12.6%)
As shown 3 incorporating ASPC consistently improves the performance of all three baselines, which demonstrates the broad applicability of our approach. IQL yields the smallest improvement, and a possible reason is that performs implicit Q learning, so increasing a does not effectively shift the policy toward the RL objective. This implicit structure offers stability but limits the best achievable performance. CQL benefits more from ASPC because updating a directly adjusts the level of conservatism. Diffusion-QU QL already achieves very strong results, and ASPC further improves its performance, which highlights the robustness of ASPC even when applied to a strong baseline.
5.7 ADDITIONAL EXPERIMENTS ON OGBENCH
We further evaluate the generality and robustness of ASPC on OGBench Park et (025a)) . new benchmark for offline goal-conditioned RL. Results across ten datasets in Table show that ASPC clearly surpasses all existing baselines, indicating strong applicability beyond D4RT. Since Park al. (2025b) also follows equation [ integrate ASPC by making its scale factor learnable and applying the same level optimization procedure, with details in Appendix D This modification consistently improves FQL, further supporting the broad generality of ASPC across standard and goal-conditioned offline RL.
Table 4: Performance on OGBench. Each entry shows mean + std. FQL+ includes the relative performance change over FQL. Bold numbers indicate the best performance for each task.
OGBench
TD34BC
IQL
ReBRAC
ASPC
FQL
FQL+ASPC
antmaze- large-nn satgggenavvaaa---iniiene singletaskktask1-o
20 1 44
48 9
91 E 10
93 t
80 : 8
84 (75.0%)
antmaze-larger sats---vggaat-siinllasssseeeeeee
20 1 31
42 1
88 1 4
87 17
57 1 10
63 (110.5%)
antmaze-Jaie wawge--aviggtee-e--eee-e- wigggutt--innglcaassttss----0
1
7
51
4
1
(U5.4%)
antmaze- antmaze-large-eae large-navigate
31 37
51 9
81
80
70 (112.55)
35 + 38
54 2
90 =
88 +
83 =
80 (L3.6%)
antmaze- antmaze-giant-oo giant-navinateeeeeee
0+1 1
0 + 0
27 + 22
22 +: 20
4
2 (L50.009)
antmaze-giant gigmm-mmiiigaleesnnnlllnassasss
15 24
1
16 1: 17
74 19
9
26 (1188.9%)
antmaze-giane wwwggggiannnnnngeeeeee mnniiglle-sinnleuuskk-ss-t-
0::
34 22
18
(10.0%)
antmaze- caauzze-giant-ne angggipeeeessigllllsssss------
11
010
5 12
65 18
14
33 (35.7%)
antmaze- saaeee-gaant--aatnn belaaa--vviiattssiinleeeeee maviigate-inggletsssssss----
16
19
49
55 = 14
16
49 (1206.3%)
Average
20.6
28.7
53.5
68.4
43.6
49.5 (113.5%)
6 CONCLUSION
We presented ASPC, bi-level framework that adapts the RL-BC trade off by optimizing the scaling factor a through second-order updates. ASPC yields consistent improvements not only on TD3+BC but also when combined with other offline RL baselines, demonstrating strong generality. However, these simple integrations yield smaller gains than those seen with TD3+BC, indicating that different algorithms may require ASPC-style components tailored to their training dynamics. Future work includes developing such method sechoo--peeeiiic adaptive mechanisms under a unified principle and evaluating them larger benchmarks and real-world datasets.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Science and Technology Major Project of the Ministry of Science and Technology of China (20242D0608100) the National Natural Science Foundation of China (62506031, 62332017, U22A2022), the Postdoctoral Fellowship Program of CPSF under Grant Number GZC20251099.
ETHICS STATEMENT
This work focuses methodological advances in offline RL. All experiments conducted on standard simulated benchmarks, which do not involve human subjects, personally identifiable in- formation, sensitive data. We strictly follow the licensing terms of all datasets and simulation platforms used in this study. Our method, Adaptive Scaling of Policy Constraints (ASPC), is designed to improve the stability and reliability of offline RL algorithms. While RL has the potential for deployment in safety-critical domains, such as robotics and autonomous systems, the experiments in this paper remain purely in simulation. Any domain-specific safety checks and human oversight to avoid unintended harm.
use of these methods should be preceded by
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the reproducibility of our work. The proposed method is described detail in Section 4 and the complete theoretical derivations are provided Appendix [ Experimental settings and hyperparameters are reported in Appendix B Moreover, we include the full implementation code in the Supplementary Material to facilitate replication of all results.
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-basee offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 0177436-74477 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-drivee offline reinforcement learning. arXiv preprint arXiv:2202.11566. 2022.
PhilipJ Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-1 2021.
Ahmad El Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani, Deep reinforcement learning framework for autonomous driv-ing. stat, 1050:8, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-leane meta-learning for fast adaptation of deep networks. International conference on machine learning. pp. 1136-1135. PMIR, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riceardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning, pp. 1568-1577. PMLR, 2018
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforeement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2062-2062. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2026
Zhang- Chang-WWiiBe Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Hamesssing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint vrXiv22306.130883 2023.
Michael Janner. Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, pp. 267-274, 2002.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning drive in day. In 2019 international conference on robotics and automation (ICRA), pp. 8248-8254. IEEE, 2019.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Hya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit
g-learning. In International Conference on Learning Representations, 2022. URL https
//oeenevieww hat.:aaw.eee..ormmmmmooaa6622210
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1199, 2020.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. arXiv preprint arXiv:2211.15144 2022.
Kueng-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadar- rama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35:27921-27933 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 2020.
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage- guided policy regularization for offline reinforcement learning. In International Conference on Machine Learning, pp. 31406-314 1424. PMLR, 2024.
Yang Liu and Marius Hofert. Implicit and explicit policy constraints for offline reinforcement learnin In Causal Learning and Reasoning, pp. 499-513. PMLR, 2024.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning, pp. PMLR, 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine OGBench: Benchmarking offline goal-conditioned RL. The Thirteenth International Conference on Learning Representa- tions, 2025a. htRpsts:::/ http:/::///ppppeeeee http///pppeeee...eeeeeee net.///wweeeeeeeee..e.eee..ee.ee www.mooouiiiiiooo www.mmiddi9iaa I
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning (ICML), 2025b.
Zhiyong Peng, Changlin Han, Yadong Liu, and Zongtan Zhou. Weighted policy constraints for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 37, pp. 9435-9444 2023.
Niranjani Prasad, Li Fang Cheng, Corey Chivers, Michael Draugelis, and Barbara E Engelhardt. A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. In 33rd Conference Uncertainty in Artificial Intelligence, UAI 2017, 2017.
Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning, pp. 28701-28717, PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Jie Ren*, Xidong Feng". Bo Liu", Xuehai Pan*, Yao Fu, Luo Mai, and Yaodong Yang. Torchopt: An efficient library for differentizable optimization. Journal of Machine Learning Research, 24(367): 1-14, 2023. /RL/tttp://mmmr.co
html
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024a.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Information Processing Systems, 36, 2024b.
Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2447-2456, 2018.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL /ttpps:////eeeeeeeeee net forum htt.://wwww......ooooooooooo.aa. 11-=A-VEDP1-1
Huaqing Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang. Deterministic policy gradient: Convergence analysis. In Uncertainty Artificial Intelligence, pp. 2159-2169. PMLR, 2022.
Qisen Yang. Shenzhi Wang, Matthieu Gaetan Lin, Shiji Song, and Gao Huang. Boosting offline reinforcement learning with action preference query. In International Conference Machine Learning, pp. 39509-39523, PMLR, 2023.
Qisen Yang, Shenzhi Wang, Qihang Zhang, Gao Huang, and Shiji Song. Hundreds guide millions: Adaptive offline reinforcement learning with expert guidance. IEEE transactions on neural networks and learning systems, 35(1 I): 16288-16300, 2024.
Zhaolin Yuan, ZiXuan Zhang, Xiaorui Li, Yunduan Cui, Ming Li, and Xiaojuan Ban. Controlling partially observed industrial system based on offline reinforcement learning a case study of paste thickener. IEEE Transactions on Industrial Informatics, 2024.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4680-4688. 2022.
Junjie Zhang, Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Jun Yang, Le Wan, and Xiu Li. Uncertainty- driven trajectory truncation for data augmentation in offline reinforcement learning. In ECAI 2023, pp. 3018-3025. IOS Press, 2023.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL PROOFS
A.1 PROOF OF PROPOSITION 4.2
Throughout the argument, we adopt the following shorthand. We index the policies
My ne
Tt+1 = Tg-
We write
LBC B(s,a)~D((T((() - all²],
L = (( all²),
A LBC == | - I.PO,, c e= -VV...C... BB
x == Eewoll=++1 (8) [T(())19)
Lemma A.1 (Reverse triangle inequality). For all A, one has |A B 2 ||A| |B||
Lemma A.2 GGauchy-Scharre For square-integrable real random variables X,Y, [[XY]| -
(B[X²) 1/2
(E(Y3)) 1/2
Proof. The proof proceeds in three steps.
Step 1: A lower bound on ALBc. Expand the definition of AL and simplify:
AL
| (s))2(s) (T(6)] ( nT(+)]]]]
E (T)+ - (T++1()) - - (me(s) - - (T((s) To(S))
|. (T)))()) - TB(s) + (s( ) (F)+(()) ((()))
E. Be[1ITt1(6 (s) - [ +2 (T((S) To())) (T++1(s) (s()))
x 2E. (T) [g(s) (T))(()) (
[2]| 2 R. (TT(()) (s())) (T))(8)) +.()))
x 2 Ba (TT(()) - =)())) (T41(s)) ((()))
2 Eyl (s) (3) . V..lTTT4 (3))
x =x- 22.0.
(14)
Since AL 2 by definition, combining with equation 14 yields
ALBC max{ x: 2cvar, 0}.
(15)
Step 2: An upper bound on (2Q)². Jensen's inequality and the assumption 4.1 yield
(AQ)² = (B.(Q((,T++1(s) Q( (.()))
[QQ(ss T44(()) ((((s)) - Q(s, {s()))
(16)
Eg (s+) (s)l1
12 x
Step 3: Mutual bound, From equation 16
we have
Tmin (AQ)2/T2)
(17)
14

---

Published as a conference paper at ICLR 2026
Since ALge 2 max(x 2cyz,, 0} from equation 15
relate this expression to 2Q as follows.
The function h(z) = x 2c/ is non non-positive on [0. 4c"] and strictly increasing on [ 0c). When [AQ] I 2cLq, we have min 4c², and (( is pon-iioit thus ALac 2 0 2 h(*min).
When |AQ] 2cLQ, we have Tmin 4c² and h(x) is increasing for all 2 mmin, which gives ALBC 2 h((min). Combining the two regimes yields the bound
ALnc 2 max 0, (AQ)²
AQQ LQ
2c
(18)
L2
Similarly, using equation 15 obtain the following upper bounds for a::
(c V + ALBC)
(19)
Combining equation with equation 19 gives upper bound on (AQ)²:
(AQ)² < L3 (c+ v + ALBC)
(20)
equation
18 and equation 20 together yield the desired mutual bounds.
n
A.2 PROOF OF PROPOSITION
This section analyses conditions under which the one-step performance difference ( ( admits a tractable lower bound when training on a fixed offline dataset D collected under behavior policy n (so D = d
Lemma A.3 (Periomaaneediiffrennce lemma). For any policies कi and TT.
1 Barde, [Ea (=, V (s)].
((T1) - (
(21)
The proof of Lemma A.3 can be found in Kakade & Langtord (2002).
Lemma A.4. Under Assumption 4.1 the total variation distance between the visitation distributions of any policy πt and the behavior policy अh satisfies
1|d dr 111 11li=1a |d=z(s) (s) ds s SCLL Lp max (T(() - (
(22)
sES
where C > 0 is a constant.
The proof of Lemma A.4 can be found in the appendix of Xiong et al. (2022).
Lemma A.5 (Sup-norm version of equation 19p. Define
Fex sup
Te+ (8) - (+)11
2 sup (s) {s)1
BL %
sup (s) (T) (s)
- [
Then
Joo S (cx V 12 + A
(23)
Proof. For each s, let ALBC(S) = [(T(1(()) 2
- (s) T (s) Then
[| (8) - me (x)
= [TTT+(()) - -s)(s) (m(()) =>()))]
S (1TT+1(8) x + [|) {))
(VITTHH(6) (²) [s) =
(24)
(VITTe(S) - (2 (TBC(8) + (I [ss()))
< CoO + V + +ALBC) 2
15

---

Published as a conference paper at ICLR 2026
Taking the supremum over gives the stated result:
Too = sup [T+ {T(s)]1 2 (co + + + ALEO).
(25)
The proof of Lemma A.5 is finished.
Proof. In our deterministic setting, the conditional action distribution T(|ss) for any state s is a Dirac measure concentrated at single action. Specifically, for T2 in Lemma A.3 have:
Be 1Q (s, a)] Q (s, m2()),
(26)
Applying Lemma A.3 with π T++1 and 22 Ti gives:
((T+1)) (TTe Esnder (s, (s)) ("
(27)
Write the performance-differeee identity equation 27 as
- Qs. 1 7 E B [Q" (s,
(m++1) (T()
+ (3)
Q* ds
2 1
1-7 E (s,
(s)) (s (s))
AQ
((dm (8) - (s, - Q" m. (x))) ds
Te+
1
AQ |1dtej1 dayll |ll supl@ (s. Te+ (8)) Q (8, (x)
m (8)) (s, { AQ - -CLp max ||++| T hull.suppppc suplQ" (s,
2
1
AQQ CLPLQ max |++1 Tall max || 
1
{AQ- -0---CCPp LLpo (max |/+T1 Tell + max || - TTl)) max | - mell
2
1 {AQ 1----CLpii CLpLQ + coo) Vzoc]
{AQ CL.pT (c50 + /2 + AI.BO)2 + 4
+ ALBC + c20)
(AQ CLLLQ (3c2 + +30/ & + ALEC + ALBO)) BC }
(28)
Thus, the one one-step step performance satisfies the lower bound
J(#+11) - ((T.) 2 1- (AQ nr3r2 + +3e/ 62 + ALBC + ALBC)
80
(29)
= LLLQ.
The proof of Proposition 4.3 is finished.
16

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF THEOREM 4.4
We now show how our outer-loss components ensure the performance lower bound equation 29 is maintained.
L1 equation 0 updates a based on the relative gradients of Q-value and the BC loss. Under the initialization assumption VaE[Q] VeLBC, so Li updates a to favor Q-improvement.
In our algorithm, the two regularizers L2 and La play complementary roles in guaranteeing safe single-step improvements. Specifically, L2 in equation penalizes the squared change in the Q- function, AQ", to prevent overly large and unreliable Q-updates. Due to the bootstrapping error inherent in RL, the single-step Q-value changes can be noisy, and therefore we apply an exponential moving average (EMA) for stabilization. In order to preserve the one-step performance lower bound equation 29 L3 equation [ must impose a matching penalty on the bias term identified in that bound. By choosing L3 so that its curvature mirrors that of L2, we ensure the single-step performance guarantee remains non-negative.
Proof. We perform a second-order Taylor expansion of & + ALEC around A AT BC 0, assum- ing AL BC /c2. < 1, discarding higher-order and constant terms. Substituting into the square and
retaining only terms up O yields:
C3 = x² 3c2
3cx
c + ALBC + AL
ACBC)
x²
(303
3c
3Cx
Co0
+
AL
20w
8c
+ +0(AL + AL 2
x² (r2
ALBC
BC
O(AT. BC ) ))
(30)
x² 36 00 + +30 2 AL +O
x6 c4 30 2 AL O(ALBC)
30 2
ALBC BC
wee AL BC
w 30k
In practice, we scale L3 by the value of Le to match its regularization strength and simply set w to
1:
L3 = (AQ)² 2 AL
(31)
By setting appropriate w, the algorithm can guarantee that:
J(T+1) ((Te) 2
(32)
The proof of Theorem 4.4 is finished.
A.4 PROOF OF THEBREM 4.5
Proof. We split the total performance gap into two components:
J(xT) J(TT) = [J(mT) [(mo)) - (J(1)) J(TO) [J(22) - [TT) -...- [J(TT) ((T-
T-1
= J(x)) - J(TO) - Σ (
(33)
We first observe that the behavior-cloning loss
LBC ( =a12
(34)
decreases rapidly during early training. Hence there exists a warm-up time to such that
I BC Ea
Ba-D|ITA (8) 3(s)] S /ea.
(35)
17

---

Published as a conference paper at ICLR 2026
and we set
To == Tta 22 B.
then
J(x') - J((o))
1 1-y E B [r(s]] Bondao [r())]
1
/ (dx-(s) [ -doo (s) r(s) ds
1
|dh- (s) -dro (s) Rmax ds
Renax
dzz dzoll1
1 1-y
(36)
Rmax
1-7 CLp max || (s) -To(s>]
C Lp Rmax
max [|TT(() (s) - (()) + B
- -(()))
Cpp Rmax
(e.3 + Vεa).
We define
Do C Rmax
(Eg + EE
(37)
1-7
Next, each one-step update produces the gain equation 32
8: ( (Tt,) 0.
(38)
Summing these gains yields the unified bound
T-1
J(=)) - J(TT) S Ao Σ
5z
(39)
t=0
With a fixed regularization weight a, the sequence {&,} tends to decay rapidly toward zero or even become negative. Therefore, static leaves large residual gap in equation 39 Our meta-update dynamically adjusts a so that each ó; stays bounded below by a positive constant 8ani > 0 over long horizon. Thus
J( - J(TT) - An - T 8min-
(40)
The proof of Theorem 4.5 is finished.
B EXPERIMENTAL DETAILS
B. HARDWARE AND SOFTWARE
We use the following hadrward:
1) Intel(R) Xeon(R) Platinum 8352V CPU @ GHz
2) NVIDIA GeForce RTX 4090 GPU
We the following software versions:
1) Python 3.8.10
2) D4RL
3) MuloCo 3.2.3
18

---

Published as a conference paper at ICLR 2026
4) Gym 0.23.1
5) mujoco-py 2.1.2.14
6) PyTorch 2.2.2 CUDA 12.1
7) TorchOpt 0.7.3 Ren* et al. (2023)
Ren
B.2 HYPERPARAMETERS
The network structures and hyperparameter configurations of each algorithm corresponding to Table
P are follows.
Table 5: ASPC hyperparameters.
Hyperparameder
Value
Optimizer
Adam Kingma
2014:
Critic learning rate
32-4
Actor learning rate
3e-4
256
TD3-BC hyperparameters
Discount factor
0.99
Target update rate
5e-3
Policy aoise Policy noise clipping
0.2
(-0.5, 0.5)
Policy update frequency
2
256
Critic hidden layers
3
Critic activation function
RcLU
Architecture
Critic LayerNorm
Tuue
Actor hidden dim
256
Actor hidden layers
2
Actor activation function
ReLU
Initial
2.5
a learning rate
2o-3
ASPC hyperparameters
learning rate decay
Exponential
update interval
10
EMA smoothing factor
0.995
Table 6: TD3+BC hyperarrerrr...eerrr.s..
Hyperparameter
Value
Optimizer
Adam Kingma
2014
Critic leaming
3e-4
Actor learning rate
3e-4
Mini-batch size
256
TD3-BC hyperparameters
Discount factor
0.99
Target update rate
5c-3
Policy noise Policy noise clipping Policy update frequency
(-0.5, 0.5)
2
2.5
Critic hidden dim
256
Critic hidden layers
3
activation function
ReLUU
Architecture
Critic LayerNorm
True
Actor hidden dim
256
Actor hidden layers
2
Actor activation function
ReLU
C LEARNING CURVES
C.I SCALE FACTOR CURVES
Figure 7 plots the a learning curves for all 39 datasets. The curves show that our algorithm (i) drives toward distinct optima across tasks and (ii) merely modulates its step size and pace when the dataset
19

---

Published as a conference paper at ICLR 2026
Table 7: wPC hyperparameters.
Hyperparameter
Value
Optimizer
Adam Kingma
2014
Critic learning
30-4
Actor leaming rate
3e-4
Value learning rate
3e-4
Mini-batch size
256
WPC
Discount factor
0.99
hyperparameters
Target update rate
5o-3
Policy noise
0.1
Policy noise clipping Policy update frequency
0.5)
2
2.5
Critic hidden dim Critic hidden layers
256
Critic activation function
ReLUU
Critic LayerNorm
Thue
Architecture
hidden
256
Actor hidden layers
2
Actor activation function
RelJU
Value hidden dim
256
Value hidden layers
2
Value activation function
ReLU
Table A2PR hyperparameters.
pyprr-parameters
Value
Optimizer
Adam Kingma
2014
Critic learning rate
3C-4
Actor learning
3c-4
Mini-batch size factor
256
TD3+BC
hyperpurametees
Target update rate
5e-3
Policy noise
Policy noise clipping
(-0.5, 0.5)
Policy update frequency
2.5
Critic hidden
256
Q-Critic hidden layers
Critic Activation function
Rel.U
V-Critic hidden dim
256
Architecture
V. Critic hidden layers
V-Critic Activation function
ReIUU
Actor hidden dim
256
hidden layers
Actor Activation function
ReLU
Normalized state
A2PR hyperparameters
W/ M/2
Table 9: IQL hyperparameters.
Hy Hyperparameter
Value
Optimizer
Adam Kingma
2014
Critic leaming rate
36.4
Actor learning rate
3e-4
Value learning
3o-4
hyperparameters Mini-batch size
256
Discount factor
0.99
Target update
5e-3
Learning rate decay
Cosine
Critic hidden
256
Critic hidden layers
2
Critic activation function
ReL.U
Architecture
Actor hidden
256
Actor hidden layers
2
Actor activation function
ReLU
hidden
256
Value hidden layers
2
Value activation function
ReLU
20

---

Published as a conference paper at ICLR 2026
Table 10: IQL's best hyperparameters used in D4RL benchmark.
Task Name
IQL
Deterministic policy
halfchectah--raddom
3.0
0.95
False
nalfcheetah-mmeluu
3.0
0.95
False
halfehectah-exxper
6.0
0.9
False
halfcheetah-mmeium-
3.0
0.7
False
aalfchootaa-mmdium-rpppp
3.0
0.95
False
halfcheetah- full-replllll--ll
1.0
0.7
False
hopper-random
1.0
0.95
False
hopper-medium
3.0
0.7
True
hopper-expert
3.0
0.5
False
hopper-medine
6.0
0.7
False
papper-meduum-repeee
6.0
0.7
True
hopper-full-reley
0.9
False
walker2d-random
0.5
0.9
False
walker2d-medium
6.0
0.5
False
walker2d-expert
6.0
0.9
False
walker2d-medium--oo expert
1.0
0.5
False
walker2dd-mediun-rrplly
0.5
0.7
False
walker2d-full-repayy
1.0
0.7
False
maze2d-umaze
3.0
0.7
False
maze2d-medium
3.0
0.7
False
maze2d-large
3.0
0.7
False
antmazc-umaze
10.0
0.7
False
antmaze-une uuaee-uaa----eee diverse
10.0
0.95
False
6.0
0.9
False
antmaze-mediung medium-diverse
6.0
0.9
False
antmaze-large-play
10.0
0.9
False
antmaze- natmaaz-llagee-divrss
6.0
0.9
False
pen-human
1.0
0.95
False
pen-cloned
10.0
0.9
False
pen-expert
10.0
0.8
False
door-human
0.5
0.9
False
door-cloned
6.0
0.7
False
door-expert
0.5
0.7
False
hammer-humen human
3.0
0.9
False
hammer-cloned
6.0
0.7
False
hammer-expert
0.5
0.95
False
relocate-human
1.0
0.95
False
relocate-cloned
6.0
0.9
False
relocate-cxpert
10.0
0.9
False
Table 11: ReBRAC hyperparameters.
Hyperparameter
Value
Optimizer
Adam Kingma 2014
1024 on Gluoo.
Mini-batch size
256 others
ReBRAC
1e-3 on Mymm-uloCo,
hyperparameters
Learning rate
le-4 on AntMaze 0.999 AntMaze,
Discount factor yy
0.99 others
Target updute rate T
5c-3
Hidden dim networks)
256
Hidden layers netwerks)
3
Architecture
Activation function
ReLU
Critic LayerNorm
True
21

---

Published as a conference paper at ICLR 2026
Table 12: ReBRAC's best hyperparameters used in D4RL benchmark.
Task Name
G (actor)
Bg (critic)
halfcheetah--rnddmm
0.001
halfcheetah--medium
0.001
0.01
halfeheetab-expert
0.01
0.01
bulfichcttah-mmdii---xxxp
0.01
0.01
0.001
halfchhetaa--full-rella
0.001
0.1
hopper-random
0.001
0.01
hopper-medium
0.01
0.001
hopper-cxpert
0.1
0.001
hopper-mediium-exper
0.1
0.01
hopper-medium-rrppe replay
0.05
0.5
hopper-full-replay
0.01
0.01
walker2d-random
0.01
0.0
walker2d-medium
0.05
0.1
walker2d-expert
0.01
0.5
walker2d--cciuum--xx
0.01
0.01
walker2d
0.05
0.01
walker2d-full-ee full-rppaay
0.01
0.01
mazo2d-umaze
0.003
0.001
maze2d-medium
0.003
0.001
maze2d-large
0.003
0.001
antmaze-umaze
0.003
0.002
antmaze-umazze-diverse
0.003
0.001
antmazz--medumm-pue
0.001
0.0005
antmaze-meedium-diver
0.001
0.0
antmazo-large-play
0.002
0.001
antmaaz-larreeddivess
0.002
0.002
pen-human
0.1
0.5
pen-cloned
0.05
0.5
pen-expert
0.01
0.01
door-human
0.1
0.1
door-cloned
0.01
door-expert
0.05
0.01
hammer-human
0.01
0.5
hammer-cloned
0.1
0.5
hammer-expert
0.01
0.01
relocate-human
0.1
0.01
relocate-cloned
0.1
0.01
relocate-expert
0.05
0.01
22

---

Published as a conference paper at ICLR 2026
quality changes within the same task. This dual behaviour highlights the method's adaptability to both task differences and data-quality variations.
halfccocctat
hopper
walker2d
reper-ed
20.0-
Allepphye2
20.0
10
17.5-
17.5
malluss??
15.0-
madomes
15.0
12.5
12.5
10.0
experti22
10.0-
7.5
7.5-
falloplay
2xdiat-eraati12
5.0
4-
50-
medtor-mppp--
axllansd
medians]
midkes-1222
Tiii Supp/
Tran Skge/
Steps
maxc22
antinaee
kegevi
deeadel
aedkan-1
20.0
opentv
8-
17.5
2.0-
7-
15.0
4154
AUSa
12.5
Aldia
3
10.0
legedrenev2
1.0-
lape-siap-22
4-
0.-
3--
2.5
1.0
Truas Stape/
Trun Supee
Trass Meass
door
hammse
relocate
deeaderi ixport- basma-1
choed-13
dleedvvi
2.5
espen-cl lassavy
2.0
hasaa-1
2.0-
1.5-
1.0
1.0-
0.5
0.0
0.5
0.0
0.0
0.5
Trils Sagu/
Trae Stpe
Triis Siey/1M
Figure 7: Learning curves of a for nine tasks across 39 datasets.
C.2 PERFORMANCE CURVES
Figure g shows the learning curves of all four algorithms on the 39 D4RL datasets. ASPC rises much more rapidly than the baselines, typically within the first 0.2-0.3 M environment steps, and surpasses them long before the others stabilize. Its final normalized scores are almost always the highest (or very close the highest) across all task families, maintaining a clear margin where the competing methods usually plateau. Moreover, the shaded regions (mean + 1 s.d. over four seeds) remain consistently narrow for ASPC, and its curves show no late-stage collapses, pointing to lower variance and steadier adaptation across widely varying task dynamics and data quality. Overall, the figure suggests that ASPC combines greater sample efficiency, stronger ultimate performance, and more reliable behavior than the other approaches.
23

---

Published as a conference paper at ICLR 2026
balfehretaa--aandomsss
expertyd
hatfcheeral expert
replay-22
Nomalieal
90
30-
ballcheeta--ill-eppi----
hopper mapoom22
hoppermmdiim-22
hopponsttpon--22
hopper mwdumm-eppoo--
Nomaltoal
begpo-modiim-mrplly--
hoppper-full heappo--full-eepppp--- 2
callerrhdmmmmmmmm22
wcallarr
waalber 2
Nornel and Score
walkeesdd medium expertiV2
walker226 mediium ceplay
walkerDd roplay
maze umwazeeaauu
maae medium
Nomallzd Store Stere
mazeed
diverse
medium
medium diverse v2
Namalizd Scate
in3-
25
antmaae
divense humaz. vi
cloned
expert
Nammaiiaa Spore
6--
done-clonaler
dane-capantee
Normaleed
Train Staps IM
Nammnce--csperton
sclocate--ummmn-
cclsaste-chonniov
celesnte-cppett--1
Noctalled Scone
TD3-BC
WPC
A2PR
Train Steps/ / 1M
Tram Stepe IM
Train Steps IM
Train Steps IM
Figure 8: Learning curves comparing the performance of ASPC against other baselines.
24

---

Published as a conference paper at ICLR 2026
D INTEGRATING ASPC WITH OTHER OFFLINE RL ALGORITHMS
D.1 INTEGRATION WITH IQL
In IQL., the policy is trained by advantage- weighted behavior cloning. Let adv(s, a) denote the IQL. advantage estimate and / > 0 the temperature parameter. We integrate ASPC by treating / as the adaptive policy-constraint coefficient.
Inner objective.
The IQL actor minimizes
IQL (0; B) = (a [exp( adv(s, a)) (BC s))}
(41)
where (Tola s)) log (a s). A single gradient step yields the updated policy (a1,))
Outer objective. Following ASPC, construct outer loss the updated policy using normalized Q-improvement term and the corresponding BC loss:
E,[Q(s, [s))]
IQL (3)
[E[[Q( ma (+))]] + ([s..a) exp(3 adv(s, a)) (BC(Ta s))}
(42)
The second term measures the change in mean Q-value induced by the inner update:
22 IQL (3)
E. ((,, - [E[(((s (o)))))))))
(43)
The outer objective for adapting is
C (3) = (3) 120
(44)
D.2 INTEGRATION CQL
CQL constrains Q-values by penalizing larger Q-values out-of-distributio (OOD) actions. Let
> denote the conservatism coefficient. Following ASPC, we treat a as the adaptive policy- constraint parameter.
Inner objective.
Given batch a, r, the CQL critic update solves
CQL (W:a)
(Q(s.))) + a (Pa Qw(s
(45)
Bellinan regression
CQL penalty
where
(s) - single gradient step produces the updated critic Qū(a))
= y (nin(Qp-(s',
Outer objective. ASPC evaluates the updated critic with normalized Q-improvement term and the corresponding CQL penalty, forming
E, 1.|0 p n (s))]
(C)L(a)= = E. Q2 p (s))]]
+ (Bw-a(no));;(s.d) -B ((
(46)
The Q-value change induced by the inner update is
C cC
(a)
(47)
The outer objective for adapting becomes
CCQL (a) = C (CQL(()) - (a)
(48)
D.3 INTEGRATION WITH DIFFUSION-QL.
Diffusion-O1 QL trains diffusion policy by combining behavior-cloning loss with normalized Q-term. Let 77 > be the coefficient controlling the trade-off between policy improvement and imitation. Following ASPC, treat 7) the adaptive constraint parameter.
25

---

Published as a conference paper at ICLR 2026
Inner objective.
Given state-actiog pairs (s, a), the diffusion policy Th is trained under the objective
E, (ss, mo(s)) (-
Luner (0;n)
B(eea)) (BC-DDLLCCCoooo (To(()))))
E. 10 (o(s)) [[,,
(49)
Diffusion behavior cloning
normalized Q-improvement
A single gradient step produces the updated diffusion policy "ē(n)"
Outer objective. ASPC evaluates the updated diffusion policy through normalized Q-value term and the corresponding BC term:
E. (s Tg())
DQL
(Bs (TT()))
(50)
E. [Q,s Ta (s)]]]
The Q-improvement induced by the inner update is captured by
C2 (2)
(n)
B. (0,10(1, xa(s))] [[((s
(51)
The outer objective becomes
DOOL (17) -D C2
DDQL (n).
(52)
D.4 INTEGRATION WITH FQL.
FQL employs two policies: (i) teacher flow policy trained purely by flow-matching, and (ii) student one-step flow policy trained via distillation and Q-improvement. Only the student policy interacts with the Q-function, making it the component that requires adaptive scaling. We integrate ASPC by treating the student's trade-off coefficient a as the adaptive constraint parameter.
Teacher objective (BC Flow). The teacher flow policy is trained via standard flow-matching:
Lreacher = E [Ilfeo]]]llf,, [o Tz. t) - (a - =0]]
(53)
where v; (1 t) + to and fe denotes the flow velocity network. This loss is independent of
Inner objective (Student Flow). The student one-step policy vy predicts an action a single step and matches the teacher via distillation loss, while also incorporating normalized Q-improvement term. The inner objective is
E.1Q(s (s)]] [.[[Q(s To (+))
FQL Lnner (x; -Es,e
(8, []]] + (s,,) e) (I
(54)
distillation (BC) term
Dormalized Qrimprovement
single gradient update produces the updated student policy Tā(a))
Outer objective. ASPC evaluates the updated student policy by combining its normalized Q-value and distillation loss, and the Q-improvement incurred by the inner update:
E. (e( + E.,, [|(T((s,)) E) Treacher (s, =)1
c (QL((a) (a) -a E. [2.()((
(55)
c C(L((a) = (E.(Q(s, (s)) -
- (s))))
(56)
The outer objective becomes
CROL (a) -. (.FOT((a) L (a).
(57)
26

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EMPIRICAL ANALYSES
E. PERFORMANCE ON ANTMAZE AND ADROIT
As shown in Table I, ASPC does not achieve state-of-the-ari performance on AntMaze and Adroit. Both benchmarks characterized by extremely sparse rewards, with AntMaze in particular using binary 0-1 success signal Fu et al. (2020). In such settings, even online RL methods struggle to learn effectively, and behavior cloning plays dominant role in determining policy quality.
Although ASPC can adapt a toward more BC-dominated regime, the current BC term imitates all actions in the dataset, including suboptimal unsuccessful trajectories. This limits the attainable performance on sparse-reward tasks. To address this issue, we experimented with augmenting the BC term using serranttge-weighhee behavior cloning, where high-advantage samples receive larger weights. The modified loss improves the selectivity of imitation by emphasizing demonstrably good behaviors. Experimental results, shown in Figure ? indicate consistent performance gains on both AntMaze and Adroit when advantage- weighttg is applied. This suggests that selectively imitating high-quality behaviors crucial for sparse-reward offline RL tasks.
ASPC+AW
-8.15
+12.5
-8.3%
-0.9%5
-0,2%
Normalized
-21 6N
1B01%
72.8
-33.8%
H43 3%
autmata wwwhat play
mntomnee medium-- divene
entrace Larp play
GnenGa harge Jamm
AuiMa Avizage
human
per shand
petsapent
dure cxpert
Irearnat opprr
rolesatoon pport
Aduilt Avcrage
hummnr
Figure 9: Normalized scores on AntMaze and Adroit tasks. Each pair of bars corresponds to single dataset (plus the domain- wise average), comparing ASPC (orange) and ASPC+AW (green), where ASPC+AW applies saantage-weighhed behavior cloning. The percentages annotated above the green bars indicate the relative performance change of ASPC+AW with respect to ASPC on each task.
E.2 ABLATION ON THE FORMULATION OF THE L3 TERM
To better understand the role of each component in the La term, consider five variants, The first variant keeps only the third component:
(
[|()())) - all2 [|T(()) al121
= sup (s.a)ED
The second variant multiplies the third component by the squared BC deviation:
1
sup [}))()) - -al33 Lg (a.a)
The third variant replaces the BC-deviation factor with the detached L2 term:
(L2 detach) Lg
The fourth variant the complete formulation used in our method:
3a = (L2 detach)
[s) all (1)
sup (s.a)ED
27

---

Published as a conference paper at ICLR 2026
Table 13: Ablation on different formulations of the L3 term. Values in parentheses denote relative
change (%) w.r.t. the full formulation (variant 4). Positive changes are shown in blue, negative in red.
Formulation
Gym-MuJoCo
Maze2d
AntMaze
Adroit
Total Avg
(l) 76.7 (-6.6%)
97.2 (-34.0%)
31.7 (-57.4%)
$5.2 (-0.9%)
64.7 (-16.9%)
(2) 76.8 (-6.5%)
107.2 (-27.2%)
31.9 (-57.2%%
54.9 (-1.4%)
65.5 (-15.9%)
(3)
81.1 (-1.2%)
151.8 (+3.1%)
73.3 (-1.6%)
56.1 (+0.7%)
77.7 (-0.2%)
(4) 82.1
147.2
74.5
55.7
77.9
(5) (-0.1%))
149.2 (+1.4%)
74.6 (+0.1%)
55.5 (-0.4%)
77.9 (+0.0%)
The fifth variant replaces both supremum operators in I.g") by dataset expectations:
1 (I-2 detach) (E(aa))DlITo (s) all2) (()))) [ll - all || all
Table 13 summarizes the results. Variants (3)-(5), which include the detached L2 term, provide clear gains on Maze2d and AntMaze, showing that this component essential for these domains. By contrast, Adroit displays only minor differences across all variants, suggesting that Q-value gradients dominate BC-related gradients there, making the precise form of L3 less influential. Finally, variant (5) achieves performance nearly identical to the full formulation, implying that strict worst-case bounds using the sup operator are not essential in practice.
E.3
CASE STUDY OF ASPC DYNAMICS
Figure 10 shows the training dynamics on halfchertah-mmediu--- ASPC consistently increases both the estimated Q-value and the BC loss, while simultancously improving the normalized score. It is essential note that the increase in BC loss under ASPC does not indicate instability or degradation. Since ASPC deliberately allows the policy to deviate from the behavior policy when such deviations yield sufficient Q-value improvement, the BC loss can increase while performance improves. This matches our theory: whenever the Q-value gain compensates for the increased deviation, the update remains beneficial. Thus, an increasing BC loss indicates that ASPC is escaping the behavior cloning regime and moving toward higher-value actions. In contrast, TD3+BC rapidly plateaus in all three curves, indicating that its fixed trade-off between RL and BC limits its ability to continue improving.
Normalized Soore
Noroa ized Score
ized
846376
42.30
TD3-BC
ASPC
550
83
C4C123
25.30
6.5
10
Train Stepe IM
Trais Steps
Trais Steps /
Figure 10: Case study on halfcheetah-mediim--V v2. ASPC maintains increasing Q-values and BC loss throughout training, accompanied by continuous improvement in normalized score. In contrast, TD3+BC quickly saturates all three metries. This behavior is consistent with the theoretical single- step performance improvement condition, illustrating that ASPC sustains stable policy enhancement over the course of training.
F HYPERPARAMETER SENSITIVITY ANALYSES
SENSITIVITY TO THE INITIAL VALUE OF a
Figure II illustrates the influence of the initial value of on ASPC. Across Gym-MuJoCo, AntMaze, and Adroit, the intermediate setting on = 2.5 provides the strongest overall performance, while very small initialization (op 0.1) tends to bias the early update dynamics too strongly toward
28

---

Published as a conference paper at ICLR 2026
CymoMuleCo Aug
Ang
713
134
.412
153
Nomalizd Soore
VHM
0.1
2.5
10
0.1
2,5
10
0.1
10
Figure 11: Sensitivity of ASPC to the initial value of a. We compare three initializations (ax
0.1, 10) and report the domain-wise normalized averages.
BC, limiting the contribution of the RL term. Conversely, an excessively large initialization (e.g., ag 10) can overemphasize the RL component the beginning, which weakens the intended stabilizing effect of the BC objective and leads to performance drops, particularly on Adroit. These observations indicate that balanced initialization important for achieving stable optimization.
F.2 SENSITIVITY TO THE LEARNING RATE OF a
We study the effect of the learning rate used for updating The results show that different domains prefer different learning rate magnitudes. Too small values slow down the adjustment of the RL-BC trade-off, while too large values make the meta-update unstable and degrade performance.
Gym-Mul:Co.dg
MazeldAva
Adroi.va
162
Nomaltze Sore
60.5
19119
41.7
26
26
26-4
26-2
26-3
26-4
26-2
26-3
26-4
26-2
20.3
26-4
Figure 12: Sensitivity analysis on the learning rate across all domains. Each panel reports the domain-level normalized score under three learning rate settings (2 10-2, 2.2 10-3, 3, 2 x 10 10-4),
G THE USE OF LLM
Large Language Models (LLMs) were used to aid and polish the writing of this paper. In particular, they were applied to rephrase sentences for improved readability and refine grammar and wording to meet academic style requirements.
29


Reviews:
[{"review_id": "sKyX5LcljK", "reviewer": "Reviewer_ZXU7", "summary": "The paper proposes ASPC, a bi‑level, second‑order method that learns the policy‑constraint scale $\\alpha$ in offline RL. The inner loop is TD3+BC; the outer loop updates $\\alpha$ with a composite loss that encourages Q‑value improvement while penalising large Q and BC‑loss shifts. On 39 D4RL datasets with a single hyperparameter setting, ASPC attains the best overall average, with particularly strong gains on Maze2D and consistent performance on MuJoCo; AntMaze/Adroit results are competitive but not always the highest. Ablations show (i) dynamic $\\alpha$ matters more than picking a single good fixed value, and (ii) both L2 and L3 regularisers contribute, in a domain‑dependent way. The approach is practical and targets a pressing source of brittleness in offline RL.", "questions": "- Can you show ASPC plugged into a value‑based method (IQL/CQL) without major surgery, even on a subset of D4RL? A small table would help establish generality.\n- How sensitive is performance to the exact form and scaling of L3? Can you share one plot per domain? \n- Which parts of the monotonic‑step proof most clearly break in deep‑RL practice? Is there an empirical check to show the intended effect holds? \n- The sketch in App. A.3 mentions an initialisation where the Q‑gradient dominates BC. How sensitive is ASPC to the initial \n$\\alpha$ and to early‑stage critic noise? \n- L3 uses sup‑norms. Do rare, high‑error samples dominate $\\alpha$ updates?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Addresses a central practical problem with a clean bi‑level solution\n- Broad, careful experiments across 39 datasets; single‑config results are compelling\n- Good diagnostics: $\\alpha$ learns to down‑weight RL on expert data and up‑weight it on noisy data\n- Useful, although idealised, theory; ablations that clarify when each loss term helps; modest runtime overhead", "weaknesses": "- Theoretical guarantees depend on assumptions and a particular L3 construction; they guide design more than they certify behaviour in deep RL\n- Performance is not uniformly best on AntMaze/Adroit; understanding and closing this gap would strengthen the story\n- Method depends on a stronger critic; without it, ASPC can fail \n- Generality beyond TD3+BC is argued but not demonstrated empirically", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6FSoMJMBxe", "reviewer": "Reviewer_QQnS", "summary": "This paper proposes Adaptive Scaling of Policy Constraints (ASPC), a second-order differentiable framework for offline reinforcement learning that automatically adjusts policy constraint scales to avoid per-dataset hyperparameter tuning, and it outperforms baselines on different D4RL tasks.", "questions": "1. Will the choices of different outer update intervals differ from tasks?\n2. The three different outer update objectives, L1, L2, and L3, show different influence on the performance degradation in different tasks shown in Figure 6. For example, L3 is critical for AntMaze while seems not valuable for Adroit. Can you provide more insights on this observation? Can we choose some of the objectives for specific task optimization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. ASPC automatically adjusts policy constraint scales via a second-order differentiable framework, eliminating the need for  hyperparameter weight tuning.\n2. ASPC incurs only minimal computational overhead compared to previous methods while maintaining strong performance across diverse tasks and datasets.", "weaknesses": "1. Although no hyperparameter in the objective is needed, the user may still need to choose the update intervals between inner and outer updates.\n2. The average performance improvement of ASPC mainly comes from the Maze2d tasks, while its performance still underperforms in AntMaze and Adroit compared to other baselines. \n3. More elaborations on the three outer updates loss should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VPz3a7esmo", "reviewer": "Reviewer_RtEJ", "summary": "This paper proposes an adaptive, dynamically adjusted hyperparameter setting scheme using a meta-learning approach. The paper also derives a lower bound on policy improvement and constructs corresponding loss functions to tune hyperparameters accordingly, ensuring stable performance gains. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "see above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Introducing a meta-learning framework to hyperparameter tuning in offline RL is novel. To identify a suitable alpha, the authors obtain new parameters via an inner loss and evaluate their performance using an outer loss, enabling adaptive adjustment of this hyperparameter.\n\n2. The experimental evaluation is extensive and convincingly supports the method’s effectiveness. The paper reports results on 39 tasks, showing performance advantages. Moreover, the trajectory of alpha during training aligns with intuition.", "weaknesses": "1. According to the algorithm's description and Equations (6) and (7), the outer losses L1 and L2 evaluate the new, updated policy π_θ˜ using the Q-function from before the policy's \"inner update.\" In Actor-Critic frameworks, the accuracy of a Q-function is tightly coupled with the policy it evaluates. Using a \"stale\" Q-function relative to the new policy introduces bias, making the optimization target of the meta-objective itself imprecise. This weakens the reliability of the theoretical foundation for adaptively tuning the hyperparameter α via meta-learning.\n2. In Appendix A.1, the derivation in Equation (15) (lines 681-682) does not strictly follow the rules of inequality manipulation. The authors' application of the Cauchy-Schwarz inequality after the reverse triangle inequality (|a+b| ≥ ||a|-|b||) is not mathematically sound, as the substitution step within the absolute value operation is invalid. This error invalidates the derived lower bound for ∆L_BC and thus undermines the foundation of the proof for the mutual constraint relationship between ∆L_BC and (∆Q)².\n3. The paper's central theoretical contribution—the single-step performance guarantee (Theorem 4.4)—claims that the algorithm ensures monotonic policy improvement. The theoretical precondition for this guarantee is that the Q-value gain ∆Q must be no less than a complex penalty term Φ (as per Proposition 4.3). However, the algorithm's actual implementation optimizes an outer loss (L_outer) that applies soft penalties via L2 and L3 to ∆Q and an approximation of Φ, rather than enforcing a hard constraint. Minimizing a weighted sum of these penalties does not mathematically guarantee that the hard condition ∆Q ≥ Φ will be met at every update step. Consequently, while the algorithm may tend towards stable updates, the claim that it guarantees monotonic performance improvement is an overstatement that is not rigorously supported by the provided theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zoId5AGDIX", "reviewer": "Reviewer_AwGN", "summary": "This paper presents ASPC, a novel adaptive tuning strategy for offline reinforcement learning via a bi-level parametric optimization framework. ASPC mainly focuses on policy-constraining offline RL methods that mitigate the distributional mismatch in offline RL by constraining a learned policy to stay close to the behavior policy. Specifically, ASPC adopts a meta learning approach for the adaptive scaling; the policy with the standard RL-BC-weighted objective is trained in the inner loop, and the regularization parameter $\\alpha$ is updated by penalizing large drifts in both objectives in the outer loop. Theoretical proofs support the design of combined objectives for adapting $\\alpha$, while extensive experiments demonstrate ASPC's outstanding performance across diverse tasks and datasets in the D4RL benchmark.", "questions": "- Why is it hard to extensively fine-tune the scale of the regularization constraint (L48)? Could you provide more intuitive examples or implications?\n- How the learning rate of $\\alpha$ affects the performance across experiments? In Figure 6-(a), both wPC and ASPC show large differences in AntMaze and Adroit by adding an extra layer and normalization. I wonder how the learning rate, which determines the magnitude of a step taken by the gradient descent, affects the performance of ASPC in those tasks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written to support the novelty of ASPC.\n- The authors provide an in-depth view of policy constraints baselines in offline RL, which helps in understanding the contribution of ASPC.\n- Extensive experiments support the superiority of ASPC across heterogeneous environments and datasets in the D4RL benchmark, including MuJoCo, Maze2d, AntMaze, and Adroit.\n- Meticulously designed ablation studies resolve potential questions regarding ASPC.\n- Comparing with SOTA baselines in adaptive regularization (e.g., A2PR, wPC) strengthens the unique contribution of this paper.", "weaknesses": "- The contribution of ASPC is limited to only policy-constraining offline RL baselines. While the baselines used in this paper (e.g., TD3+BC, IQL, and CQL) still maintain strong performance across popular offline RL benchmarks, other branches in offline RL have suggested numerous algorithms that outperform such baselines. For instance, offline model-based RL [1,2] or offline RL with generative models [3,4] prove remarkable performance over denoted baselines in the D4RL benchmark. If ASPC can be extended to other methods that employ the RL-BC-weighted objective, the novelty of this paper would be further reinforced.\n\n[1] Sun Y, Zhang J, Jia C, Lin H, Ye J, Yu Y. Model-Bellman inconsistency for model-based offline reinforcement learning. ICML 2023.\n\n[2] Rigter M, Lacerda B, Hawes N. Rambo-rl: Robust adversarial model-based offline reinforcement learning. NeurIPS 2022.\n\n[3] Hansen-Estruch P, Kostrikov I, Janner M, Kuba JG, Levine S. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv 2023.\n\n[4] Wang Z, Hunt JJ, Zhou M. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QS0ML4SXNv", "reviewer": "Reviewer_KwF2", "summary": "This paper aims to balance the typical RL-BC trade-off in offline RL methods. Specifically, common offline RL objective can be abstracted as  \\$\\alpha L_{rl} + L_{bc}\\$, forming a balance between $L_{rl}$ and $L_{bc}$, controlled by $\\alpha$. This paper proposed a bi-level objective, which formulates $\\alpha$ as learnable parameters to adaptively control the update rate of RL and BC objective, and thus avoiding any of them to dominate the other one, thus balancing these two terms. The idea and motivation of this paper is straightforward and simple. The evaluations on D4RL benchmarks demonstrate the effectiveness of the auto-tuned $\\alpha$, based on its TD3+BC variants.", "questions": "Please see weakness for details.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea is simple and straightforward.\n2. This paper well-written and is easy to follow.\n3. The proposed method is easy to be implemented.", "weaknesses": "1 `Novelty and Significance`\nAutomatically tuning the conservatism strength in offline RL is not a new research direction. Several existing methods have already addressed this problem in ways that are both simple and effective. Therefore, I would not consider the novelty or significance of this paper to be its main strength.\n\n2 `Versatility of the Proposed Method`\nThe proposed approach is developed specifically on top of TD3+BC, which is now a relatively dated baseline. In fact, most offline RL methods can be abstracted as an RL + BC objective, including more recent diffusion- or flow-based approaches. However, the improvements in this paper are restricted to TD3+BC, and the additional loss function also appears tied to this specific algorithm, limiting its generalizability. Considering the already limited novelty, it would strengthen the contribution to derive a more versatile framework that can generalize to a broader range of offline RL approaches, rather than being confined to one.\n\n3 `Dated Benchmark`\nThe D4RL benchmark has long been saturated, making it difficult to draw meaningful distinctions between methods. To strengthen the empirical soundness of the paper, it would be beneficial to include evaluations on more challenging benchmarks, such as OGBench.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)