ACTIVE: Offline Reinforcement Learning via Adaptive Imitation and In-sample $V$-Ensemble
Authors: Tianyuan Chen
Ronglong Cai
Faguo Wu
Xiao Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Reinforcement Learning
In-sample Learning
Ensemble
Uncertainty Quantification
Dual Gradient Descent
Primary area: reinforcement learning

Abstract: Offline reinforcement learning (RL) aims to learn from static datasets and thus faces the challenge of value estimation errors for out-of-distribution actions. The in-sample learning scheme addresses this issue by performing implicit TD backups that does not query the values of unseen actions. However, pre-existing in-sample value learning and policy extraction methods suffer from over-regularization, limiting their performance on suboptimal or compositional datasets. In this paper, we analyze key factors in in-sample learning that might potentially hinder the use of a milder constraint. We propose Actor-Critic with Temperature adjustment and In-sample Value Ensemble (ACTIVE), a novel in-sample offline RL algorithm that leverages an ensemble of $V$-functions for critic training and adaptively adjusts the constraint level using dual gradient descent. We theoretically show that the $V$-ensemble suppresses the accumulation of initial value errors, thereby mitigating overestimation. Our experiments on the D4RL benchmarks demonstrate that ACTIVE alleviates overfitting of value functions and outperforms existing in-sample methods in terms of learning stability and policy optimality.

Published as a conference paper at ICLR 2025
ACTIVE: OFFLINE REINFORCEMENT LEARNING VIA ADAPTIVE IMITATION AND IN-SAMPLE V -ENSEMBLE
Tianyuan Chen 1,3.6 Ronglong Cai Faguo Wu W...3.466 Xiao Zhang2.3.45.66
2,3,4,5,6
School of Artificial Intelligence, Beihang University
2 School of Mathematical Sciences, Beihang University
3 Key Laboratory of Mathematics, Informaties and Behavioral Semantics, MoE, Beihang University
*Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University Hangzhou International Innovation Institute of Beihang University 7Zhonguuaneun Laboratory
ABSTRACT
Offline reinforcement learning (RL) aims to learn from static datasets and thus faces the challenge of value estimation errors for out-of-distribution actions. The in-sample learning scheme addresses this issue by performing implicit TD backups that does not query the values of unseen actions. However, pre-existing in-sample value learning and policy extraction methods suffer from over-regulatizatio, limit- ing their performance on suboptimal or compositional datasets. In this paper, we analyze key factors in in-sample learning that might potentially hinder the use of a milder constraint. We propose Actor-Ciio Critic with Temperature adjustment and In-sample Value Ensemble (ACTIVE), novel in sample offline RL algorithm that leverages an ensemble of V-functions for critic training and adaptively adjusts the constraint level using dual gradient descent. We theoretically show that the V-ensemble suppresses the accumulation of initial value errors, thereby mitigating overestimation. Our experiments on the D4RL benchmarks demonstrate that AC- TIVE alleviates overfitting of value functions and outperforms existing in-sample methods in terms of learning stability and policy optimality.
INTRODUCTION
Reinforcement learning (RL) has achieved significant success in various sequential decision-making tasks (Mnih et al. 2015 Silver et al. (0166.. However, notable drawback of online RL is the requirement for a huge number of continual environmental interactions, which might be impractical since real-world interactions can be costly or dangerous. Offline RL aims to learn from static datasets without further interactions (Levine al. 2020), which in principle makes it possible to exploit large previously collected datasets. In practice, the distributional shift between the collecting policy (also known as the behavior policy) and the learned policy poses significant challenges for offline RL algorithms. Improving the policy beyond the the dataset level requires querying the values of actions sampled from the policy, which are likely be out-of-diisrriutioo.. These -ottt--iisttitttinn actions can produce highly overestimated Q-values (Kumar et al. 2019 Fujimoto et al.) 2019).. which can be further propagated with bootstrapping, resulting in catastrophic inaccuracies value estimation.
To address the overestimation problem, prior offline RL methods typically add pessimism to the learning objective either by constraining the policy be close the behavior policy (Wu et al., 2019)
Fujimoto et al. 2019 Kumar et al. 2019 |eng et al. 2019 Nair et al., 2020 Fujimoto & Gu 2021
or by regularizing
value function to make pessimistic predictions for distri-iittiitttion actions
(Kumar et al. 2020 Kostrikov et al. 2021 An et al. [222 Yang et al. (022). More recently, implicit TD backups have been proposed in IQL (Kostrikov et al. (202)) that completely avoid querying values of unseen actions by approximating the in-sample maximum of Q-values using expectile regression. However, when the dataset is skewed towards suboptimal policies or mixed with lower- qower-quality data, prior in-sample learning methods could suffer from performance drop due to oveulaggulaiizztion (iiao et al. 2023).
"Corresoonding Authors. Emails: {faguo, xiao. 2h J@buaa edu. en.

---

Published as a conference paper at ICLR 2025
In this paper, aim improving the overall performance of IQL-style in-sample offline RL methods (Kostrikov et al. 2022 Xu et al., 2023 Garg et al. 2023). Empirically, we find that in implicit TD backups, ((s) could easily overfit the initial error of Q(s,, a), causing catastrophic overestimation and performance degradation. This phenomenon prevents the use of a larger expectile T or a lower level of implicit regularization ar (Xu et al.] 2023). Based on this observation, we propose an in-sample value learning scheme based on a Veensemble that mitigates the overfitting of value functions, thus allowing for less regularization. The in-sample V Veenseblle captures in-sample epistemic uncertainty, which is then used as a penalty during bootstrapping and advantage estimation. To better balance generalization and regularization in suboptimal datasets, we propose a policy extraction loss inspired by automated entropy adjustment in SAC (Haarnoja et al., 2018) that adaptively adjusts the strength of imitation based on dual gradient descent.
Our primary contribution is Actor-Critt with Temperature adjustment and In-sample Value Ensemble (ACTIVE), an ensemble-based in in-sample offline RI. algorithm that combines the aforementioned modifications. We conduct theoretical analysis of the output dynamics of the Q-function and the error suppression effect of the Vensemble. To promote understanding of the in-sample epistemic uncertainty, empirically analyze the distribution of advantage functions obtained using a single network and a V-ensemble on D4RL datasets (Fu et al., 2020). highlighting the need for a more discriminative advantage estimate. Our experiments on the D4RL benchmark show that ACTIVE improves upon existing in-sample methods in terms of learning stability and final policy performance.
2 RELATED WORK
Model-free Offline RL. The majority of recently proposed model-free offline RL methods rely on
either policy constraints or value regularization. Policy constraints can be implemented through
distance constraints towards dataset actions (Kumar et al. 2019 Fujimoto & Gu, 2022 Ran et al.
2023), or through explicit modelling of the behavior policy (Wu et al. [2019; Fujimoto al.) [019
Siegel et al. 2020; Ghasemipour et al. 2021 Zhou et al. 2021 Wu et al. (2022). Policy constraints
can also be imposed implicitly using weighted behavior cloning (Wang et al.
2018 Peng al.
2019: Nair et al. 2020). Value regularization methods drive the critic to be pessimistic about out-of-
distribution atate-action pairs (Kumar et al., 2020 Kostrikov al.] [021]] An et al. 202 Yang et al.
2022). Recently, in-sample learning was proposed in IQL (Kostrikov et al. (022) that completely
avoids querying the values of any unseen actions. Generalizations of IQL, including EQL GGarg et al. 2023) and SQL (Xu et al. (023) have been proposed. More recently, IDQL (Hansen- -Ettuuch ct al. 2023) connects IQL actor-critic methods through an implicit actor: Our work based on the generalized IQI. framework. In contrast, our work reveals major difficulty in implicit TD learning and provides an ensemble-basee based solution that can be applied to any variant of IQI..
Ensembles and Uncertainty Estimation. A number of studies have successfully employed ensem- bles for uncertainty estimation offline RL, Ensembles can be applied to the critic in model-free offline RL (Agarwal et al. 2020; An et al. 202 Bai et al. 2022 Ghosh et al. 2022 Ghasemipour Gu 2022] Yang et al. (022)) or the dynamies models in model- based offline RL (Yu et al. 2020) Kidambi et al. 2021 Sun al. 2023). In most of these methods, ensembles provide estimates of epistemic uncertainty (Clements et al. 2019), which are then used to form a pessimistic value prediction for -idttr-iittribution samples. Addrionally, modifications like OOD sampling in PBRL (Bai et al. 2022) and gradient diversification in EDAC (An et af. (202)) have been proposed to specifically target out-of-distribution samples. In our work, we focus on in-sample learning, which does not query the values of any unseen actions during critic training. Unlike previous approaches, the ensemble in our method captures in-sample epistemic uncertainty, which is used suppress error accumulation and form a more discriminative advantage estimate for in-distribution samples.
reeraaggulai in Offline RL. While staying close to the offline dataset limits potential overestimation and stabilizes training. recent studies have argued that these constraints can be overly conservative in certain cases (Jin et al. 2021 Buckman et al. 2021| Xie et al. 2021). In practice, it has been reported that over-regularization occurs in variety of offline RL algorithms (Zhou) ct al. 2021 Kumar et al., 2022 Fu el al. 2022 Xiao et all 2023). To balance conservatism and generalization, MCQ (Lyu al. 2022 proposed to learn a mildly conservative Q-function by actively training on OOD actions. From the policy constraint perspective, PRDC (Ran et al., (023) proposed to constrain the policy toward the neirest-neighbor state-action pair. In contrast, aim to mitigate the over over- regulariaaiitinnn of in -a-mapll methods by combining SAC-style automated entropy adjustment with weighted behavior cloning.
2

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
3.1 OFFLINE REINFORCEMENT LEARNING
RL is formulated Markov Decision Process (MDP) defined tuple (S, A. Po. p, r, 7). with state space S, action space A, initial state distribution Po (s), transition dynamics p(s' (s, a), reward function r(s, a) and discount factor Y (0. 1). The goal of RL find a policy (Tass) S A - [0, that maximizes the expected return E, 1rt= y'((st, at)] in the MDP. In this work, we focus on offline RL, where the agent only has access to a fixed dataset
{(s, a, r, s')} collected using a different behavior policy potentially multiple policies. The behavior policy is denoted as p(a(s), which is action distribution conditioned on states observed in the dataset.
3.2 GENERALIZED IMPLICIT Q-LEARNING
Implicit Q-Learning. Instead of explicitly constraining the policy regularizing the Q-function, Implicit Q-learning (IQL) (Kostrikov et al. 2022) approximates the in-sample maximum of the Q-function using value network Vs(s) and expectile regression:
(v( ([s.a))
(o(s))), where Lī(u) - L(u 0)ju².
(I)
The Q-function can then be learned by bootstrapping off the value estimate of the next state, thus avoiding any queries to values of out- cutt-ofddistrbbution actions:
LQ (s. [B(....))= >Vo(s') Qo(s,
For policy extraction, IQL uses Advantage Weighted Regression (AWR) (Peng et al. 2019):
C (Ø) B(ss))_plexp(((0((sss (s))) log T (a|s).
Generalized Implicit Q-Learning. Recently, Hanssnn-Esstruchh al. 2023) showed that IQL. can be rederived as an actor-critic method. For arbitrary convex loss f, define the general IQL V-update as:
(*(s) arg min lfo.
(s)))..
V(s)
The following theorem states that f corresponds an implicit actor Timp (als:
Theorem 3.1 (aaneen-Estruch et al. (203 Denote ov(e] as f', then for every state and convex loss function such that J'(0) 0, the solution to the optimization problem defined in Equation (4) is also a solution to the optimization problem
arg min Ba ((((s,)) V(s)) V(s)
VV" where (ass x w(s, a))(a a(s [Q(c.a)-V-(a)
Different choices of loss function f give rise to different implicit actor distributions. Recent variants of IQL like EQL (Garg et al. 2023) and SQL (Xu et al., 2023) matches the critic objectives for fo (u) exp(u/a) u/o and Jo (u) = I(1 + u/2xx 2ox > O) (1+u/2o u/2a) u/a respectively, Xu et al.
(2023) pointed out that in the loss functions be seen as the regularization coefficient in the
behavior-regularized MDP problem:
DC
r(se, F( Maclse)
max E
Σ
rd
for some regularization function
In generalized IQL algorithms, the strength of regularization is usually controlled using single scalar hyperparameter. In IQL (Kostrikov et al. [2022]. computing a larger expectile 7 E (0, can, in theory, better approximate the in-sample maximum V max.'eA, )()))s()>0 Q(s', a'). This enables better filtering of suboptimal actions by using the advantage function Q(s. a) V(s) during policy extraction. Similarly, using a smaller a in Equation (5) (Xu et al. 2023) better approximates the unconstrained policy optimization problem.
3

---

Published as a conference paper at ICLR 2025
4 VALUE LEARNING WITH IN-SAMPLE V- -ENSEMBLE
4.1 OVERFITTING OF V-FUNCTION AND IN-SAMPLE ERROR ACCUMULATION
Intuitively, higher (or a lower a in SQL (Xu et al. 2023) should be used in mixed datasets order to better approximate the in-sample maximum Qo(s, a). thus filtering out suboptimal actions much possible. However, we show that the overfitting of V-function impedes the desired modification. Empirically, we find that the initial error of Qo(s, a) would be captured by an overfitted (s) and propagated among states through bootstrapping. The result is that in-sample offline RL algorithms remain susceptible to catastrophic overestimation, despite the complete avoidance of value queries for unseen actions. Also, the use of higher (or lower o)) makes (s((s) more sensitive to highly overestimated Qo(s, a) values, which exacerbated the overfitting of V-function.
To demonstrate the problem computationally. we train SQL agents on the D4RL (Fu et al. 2020) antmaze-umaze-d-22 dataset for 1M up-
antmaze-4umaze
antmaze-umaza
Normalzed Score (EMA)
100
-20
75
-40
dates. The result is shown in Figure [ The
Batth Auwrage
AntMaze datasets typically require larger T (or smaller a) for effective value propagation. In vanilla SQL without additional regulariza- tion the value function Vi (s). taking down to 0.1 results numerical instabilities. sta- bilize bootstrapping. the official implementa- tion of SQL applied Dropout (Srivastava et al. 2014) with p 0.5 and layer normalization (Ba et al. 2016) to (s). Without Dropout and layer normalization, the critic diverges with small a 0.1 in SQL, and yet critic divergence could still happen after these modifications.
50
-60
25
100
0.25
1.00
-120
Training
Figure Normalized score and batch average ( a) of SQL and IVE-SQL. "SQL (no mod)" stands for SQL without value function Dropout and layer normalization.
Remark 4.1. We remark that the problem of V- -ffunction overfitting is related to the advantage sparsity defined as sQ, D) = E [ (Q a) V(s) S 0)] (uu et al. 2023 J. In regions with high epistemic uncertainty (Clements al. 2019), Vss) could easily overft Q a), resulting in highly sparse advantage estimates. Note that for some variants like SQL (Xu et al. (023) where the weight is simply max[Q(s, a) - V(s), 0}, such state-action pair provides zero learning signal. In the limit, AWR equivalent to behavior cloning when the advantage distribution is a Dirac delta 8(0) with
mean zero, making it harder to learn selectively from the dataset.
4.2 VALUE LEARNING WITH IN-SAMPLE V-ENSEMBLE
To mitigate overfitting of the V-function, propose using ensemble of value functions (Vo, |m to perform in-sample updates. Given a generalized IQL loss function f. the V-functions independently optimized using the following loss function:
(1)) (( a) (s))).
The Q-function can then be learned by bootstrapping against an aggregate of the V-ensemble. To control V-function overfitting and stabilize bootstrapping, we can use the ensemble minimum:
()) B.....))~ a) + min Vo. (8') 2o(s,a))²
To extract the policy from learned Qo(s, a) and (Ve, (s)),
perform AWR Peng al. 2019
or its variants using the advantage estimate Á(s, a) = Q3(s, a) EV,,,, where E VV. Ve the e-th quantile of (V v. pm AWR with V-ensemble maximizes the following objective:
AWR (b)
- log (a)s)))
(8)
where e [0, and BaWR E [0, oo] is an inverse temperature.
4

---

Published as a conference paper at ICLR 2025
The effect of the V-ensemble on in-sample learning is twofold. First, we take the ensemble minimum during bootstrapping in Equation ( which serves as a penalty for high-uncertainty states. The bootstrapping target can be inter- preted as utilizing the lower-confidence bound
ballchewllhhmmeee
heppersetee922
1.8
Re Remn Dunall Extmution
0.8
122
0.0
2.0
(LCB) of V-predictions. Assume that (s) fol-
Abverlag Edimals
Advartago SUmmaz
lows Gaussian distribution with mean su(s) and standard deviation ass). The ensemble [V<(s) [= contains samples of (s)). Similar to (An et al. (02T), the expected ensemble min-
walker2d v2
Termatto
1.0
0.4
cors
imum can be approximated as (Royston 1982)
min Vi(s) 15ism
Adviiage Extimate
Advartage Braat
(9)
Figure 2: Kernel density estimation of advantage functions obtained by training on 4 datasets from the D4RL benchmark using single network SQL (gray) and V-ensemble (blue. red and orange). [V] denotes the c-th quantile of the ensemble.
2
u(s) Φ-1
(s)
m
As we will show in our empirical evaluations, this penalty effectively mitigates the numerical instabilities encountered when a large (or small a) is used. Second, the V V-ensemble also reshapes the advantage distribution used by AWR. Unlike algorithms with -ittt--diitriuution (OOD) evalua- tions (e.g. SAC-N (An et al., 202T)) in which uncertainty penalties are intuitive for OOD state-action pairs, the V-eseeble captures in-sample epistemic uncertainty which less well understood. Here we provide an empirical study of the impact of V-ensemble on the advantage distribution. Figure 2 illustrates the distribution of advantage estimates obtained using Vanilla SQL and V-ensemble of size m 5 4 D4RL Gym MuJoCo datasets, from which we can see that V-ensembles generally pro- duce advantage distributions that are less sparse but sometimes biased. We find that the bias-sparsity trade-off controlled by [.] could significantly affect training outcome, and we will provide more empirical analyses in Section 7
4.3 THEORETICAL ANALYSIS
To analyze the output dynamics of Q-functions with generalized IQL updates using the in-sample V-ensemlle, we draw upon the work on the Neural Tangent Kernel (NTK) (Jacot et al., 2018 Lee
al.] [2199] Ghasemipour & Gu 2022). We will begin by establishing some additional assumptions and notations. First of all, Theorem 3. indicates that there exists an implicit policy Timp determined by V*, such that the V-function can be equivalently learned by regressing towards Q(8, Timp using a MSE loss. This leads to the following assumption:
Assumption 4.2. Given the current Q-network, the corresponding Timp can be sampled for any state appearing in the dataset D {(s, a, 8')] (s) denotes sample of Timp(als).
To facilitate our analysis. we also frame the problem as a fixed-policy evaluation featuring a slightly different learning target (justified by Equation ( by making the following assumptions.
Assumption 4.3. The implicit policy Timp is kept fixed throughout the value learning process.
Assumption 4.4. The regression target for Qa(s,a) a) is ((ss,)) a) + ([enn[V][s)
(Vens(V((s))
Next, we present notations relevant to our NTK-based analysis. Let O, O', R and X denote data matrices containing 8, s', (s, r and (s, Ting (s)g(s)) where (s,,, s') appears in the offline dataset D. We consider parammaramterrized Qo and Vy- For any two state-action data matrices A and B, the initial NTK of Qe is given by 6,0) (A,
VoQo, (A) VoQe, (B) Tr At infinite width, the kernel converges to a deterministic one denoted by @(w), Analogously, we define the initial NTK (at infinite width) of Vi acting on state data matrices as (()). Let Cimp = 6 (2,, 600 (x,, 21) and Cv = é (0) (O', O) 6(0) (0, ()- For policy evaluation, define Q()) = Qa and V(0) = Vv with parameters 0, bhy sampled from the initial weight distribution. Ensemble indices specified if necessary. {Q()) and {V(0) denote the sequence of intermediate estimates during training, where
v()) is updated using Q(0), VI 2 1. Assume 7|lCyyCim <
5

---

Published as a conference paper at ICLR 2025
We now ready present our main theoretical result: Theorem 4.5. Ler S2 Cv (x) Cimp (1))11()))))))
(X)), B
--VE Bens (Vio(()) Cv V (O))
}
After t + iterations of update 6 and (7) (with 4.4. the Q-estimate for (s,a) ~ A' is given by:
(x) = +1++- ...... (7C R
(10)
+7 + +... (yCVCCmp) [] (Ω B) 0 (7'
See Appendix A for detailed proof. Note that X' in Ω is likely to select high Q(9) values due to the implicit optimization of Timp. And Theorem 4.5 indicates that the proposed Q-function update penalizes the "initial error" term 52 by B 2 O. thus reducing the chance of catastrophic overestimation. Note that Assumption 4.3 is a strong one which is not satisfied in the actual IQL algorithm. The output dynamics of IQL under varying implicit policies remains an area of further study.
POLICY EXTRACTION WITH ADAPTIVE CLONING TEMPERATURE
Improving value learning alone can be helpful for the optimization of the implicit policy Timp- but may not completely solve the overreegularization problem since AWR always imposes a KL-divergence constraint (Peng al., 2019). Inspired by the automated entropy adjustment technique used in SAC (Haarnoja et 2018), we formulate a constrained optimization problem where the average likelihood of the state- action pairs from the dataset is constrained, while the likelihood at different states can vary. Formally, consider the following constrained optimization problem:
min -E,-Daww (Q((s, a)]
s.t. [all))
To (a|s)] 2 HDD
(11)
where HD is given constant. The Lagrangian of Equation 11 is given by
L(o 3) = -Ea-Doww [Q a)] + B ((-E(s,a)> [log (a[s)] + HD))
(12)
Similar to (Haarnoja et al. 2018), we perform approximate dual gradient descent on the constrained optimization problem, which alternates between optimizing the Lagrangian with respect to the primal variable and optimizing the dual variable 33 by minimizing the following loss:
[( E (ssaa)~ [B (als) H())).
(13)
Minimizing loss 11) increases the value of 3 if log
To (a]s) HD and vice versa. One potential
problem of this approach is that HDD is overestimated, the behavior of (I2) will gradually approach behavior cloning. We can alleviate this issue by combining (12) with weighted behavior cloning
Lx = -Es-Da (Q a)] - ( [u(s, a) log [a]ss]]
(14)
where w(s,, a) be computed using learned Q(s, a) and V-ensembe. As example, for AWR: w(s a) = exp(Q2(s, - E [V).]ss).
(15)
By optimizing the loss in Equation (44 in the worst case where 3 blows up due to an overestimated HD, we can still get similar behavior to weighted behavior cloning. Remark 5.1. I. Estimating the optimal Hp is difficult, since the average log-likelihood over D of the best policy =b that can be learnt from D varies greatly with the quality of D. We describe the tuning procedure for HD in Appendix B
ALGORITHM SUMMARY
Algorithm ACTIVE
Hyperparameters: f= So. m. HD. LR X. λs, EMA 7). 3, D.
We now describe our complete algo- rithm which combines our modifica- tions on value learning and policy ex- traction. We refer to the resulting algorithm as Actor-Critic with Tem- perature adjustment and In-sample Value Ensemble (ACTIVE) and sum- marize the approach in Algorithm T
Initialize: é. 0, for each gradient step do
3/s v.: AV #h Cf (wbz) (Equation (6)
0 - AV (Equation
Ô - (1 n)00 r7
3 3 V (Equation [3
+ 6 AV à (Ø) (Equation
[4
end for

---

Published as a conference paper at ICLR 2025
Table 2: Average normalized score over the final 10 evaluations and 5 seeds. + captures the standard deviation over seeds. The results of CQL, TD3+BC, IQL and SQL taken from the authors. The "-I" and "-S" suffixes indicate the use of -tt-et and SQI-style V-function losses, respectively. The highest scores bolded. Additionally, scores higher than the corresponding reproduced baseline (IQL for ACTIVE-I, SQL for ACTIVE-S) are marked in blue.
IQL
SQL
Dataset
CQL
TD3+BC
IQL
SQL
ACTIVE-I
ACTIVE-S
(Repeudaced)
(Reproduced
halfcheetah-m
44.0 #0,8
40.3
20.2
48.3 +02
47.4 1:01
47.8 10.0
50.6 #0.1
52.3 1002
hopper-m
58.5 #211
59.3 +42
66.2 45.7
75.5 =3.4
64.4 =2.7
+2.3
80.1 =3.8
86.1 +334
walker2d-m
72.5 +0,8
83.7 1211
78.3 48.
84.2 #4.6
80.1 1:33
82.8 +0.4
84.8 0.77
87.2 1100
halicheetah m-r
45.5 1.0.5
44.6 1.0.5
44.2 111.2
44.8 1.0.7
43.6 10.2
43,9 1.0.
51.1 1.03
51.7 1.022
hopper-m-r
60.9
94.7
1334
85.6
1011 1.6
1028:1077
walker2d mir
77.2 1.5.5
81.8 1:5.5
73.8 17.1
77.2 1.388
70.3 155.6
69.2 1.6.3
85.4 3355
79.3 1:4.6
halfcheetah-m-e
90.7
86.7 4.5.3
1:16
10.
1:10
hopper-m-e
105.4 6.88
98.0 49.4
91.5
111.8 222
98.7 77.9
104.9 45A
89.2 1009
109.9 +200
walker2d-m-e
109.6 0.7
110.1
+0.5
100.6
110.0 0.08
110.0 #:044
111.2 +0.1
112.0 0.4
111.7 +0.3
antmaze-u
74,0
78.6
87.5
91.2
85.8 #3.2
93.0
91.6 1T.
90.88 =1.
antmaze-u-d
84.0
71.4
+1..
74.0 =233
60.6 114.
49.6
78.4 =.11
43.8 =22.1
antmaze-m-p
61.2
10.6
71.2
80.2
1-62
72.8
antmazs-m-d
53.7
3.0
10.0
75.1 #4.2
73,8 #377
62.2
75.2 22
+3.03
untmaze-1-p
15.8
0.2
39.6 155.8
50.2 4488
48.2 16.55
38.2 4:5.2
51.2 55888
51.2 +:5.7
antmaae-1 I
14.9
47.5 19.5
$2.3 1.2
47,2 133.5
41.6 144
48.2 15888
42.6 133,2
krtchen-C
43.8
1:1.2
61.
76.4 2.7
67.0 #34
21.33
68.2 23.6
66.5 #3.
kitchen-p
kitchen-m
51.0 +66
528
67.4 #5.4
47.9
46.8
52.1 #2.9
73.2 114
7 EXPERIMENTS
In this section, we present empirical evaluations of ACTIVE against baseline algorithms. We first compare our method with baseline model-free offline RL methods on the D4RL benchmark. We then analyze the effect of in-sample Veenseble (IVE) and adaptive cloning temperature (ACT) in ablation studies. Finally, we present an empirical comparison of the running time of ACTIVE and prior in- sample learning methods.
7.1
COMPARISONS ON OFFLINE RL BENCHMARKS
Comparisons and Baselines. First, we evaluate our method on Gym MuJoCo, AntMaze and Kitchen datasets of the D4RL benchmark (Fu et al. 2020). For a fair comparison, follow the setting of IQL (Kostrikov et al., 2022) in which Gym MuloCo and AntMaze datasets take the "-v2" version while Kitchen datasets take the "-v0" version. Note that AntMaze and Kitchen datasets con- tain fewer netrmoptimal trajectories than Gym MuloCo datasets, and learning ef- fective policies from them requires "stitching" together sub-trajectories. We compare our method with popular model-free offline RL methods, including CQL (Kumar et al. 2020). TD3+BC (Fujimoto Gu 202I), IQL (Kotrikovv et al.) 2022) and SQL {Xu et al. 2023). The results are presented in Table We see that method performs comparably the best-performing prior method and outperforms the corresponding in sampee baselines (IQL for ACTIVE-I, SQL for ACTIVE-S), especially suboptimal diverse datasets (e.g. the m/m-r datasets, antmaze-u-d, kitchen-m).
Table 1: Comparison against ensemble-basee algorithms with similar ensemble sizes (m)) The results of RORL and MSG are taken from the authors.
ACTIVEA
ACTIVESS
Dataset
RORL (m 10)
MSG (m 4)
(m S
(In T
animaze-u
96.7 1 1.9
98.6
91.6
1:
90.8
1.5
antmaze-uu
90.7
2.9
76.6
78.4
5.1
43,8
22.1
antmaze-m-p
76.3 + 2.5
83.0
1: 7.1
74.6
5.5
78.8
1.7
69.3
83.0
75.2
antmaze-l-p
16.3 + 11.1
46.8 + 14.7
51.2
5.8
51.2
antmazo-I-d
41.0 + 10.7
58.2
48.2
42.6
Average
65.1
744
69.9
62.3
Additionally, we compare ACTIVE with popular ensemble- based algorithms, including RORL Yang et al. 2022) and MSG (Ghasemipour & Gu (022) on AntMaze tasks, and SAC-N (An et al. on Kitchen tasks. While it is known that ensemble-based methods like SAC-N performs well on realistic and challenging (Fu et al. 2020).
2021
MuJoCo tasks, AntMaze and Kitchen tasks can be
The results on AntMaze with similar ensemble sizes are shown in Table Note that MSG does not rely solely on ensembles but instead requires CQL-like regularizer. In contrast, ACTIVE does not require additional explicit regularization. Due to the lack of official results on Kitchen tasks, we run SAC-N with ensemble size m = 20 and 40. In both cases we find that SAC-N rapidly diverges and failed to learn any meaningful policy, as shown in Table 8, Note that an ensemble of size m 40 is already much larger than the value ensemble required our method (typically 5 S m S 7).
7

---

Published as a conference paper at ICLR 2025
Effect of In-sample V-Ensemble (IVE). We demonstrate the effect of V-ensemble and its interaction with 5 (or a).. First we compare IVE-S and SQL on the simpler antmaze-u dataset. The result is shown in Figure Note that Dropout can also be used
antmazz-umaaz--dv2
60
60
Normalized
40
I0 fr- 0.3)
to capture epistemic uncertainty (Gal & Ghahramani] (016) but
T
in SQL it is only used to mitigate overfitting. In ACTIVE we
20
m m- 1-5.1-000 1-0.5
m r
take a different approach of taking the ensemble minimum in bootstrapping and policy extraction. It is evident that our method successfully prevents catastrophic errors and is more consistent in reaching higher scores. We present additional results for reducing catastrophic overestimation on the antmaze-u-d dataset shown
0.00
0.25
0.50
0.75
1.00
Taining Iterations
le6
antmazo-umazo-dw22
Normalzzed Score
75
in Figure IT in Appendix [C] For harder tasks, such as "diverse"
50
versions of the datascts, a larger T is desired IQL to extract better policies but performs poorly due to numerical instabilities. In ACTIVE, a fixed-size Vensemble m = allows for a larger expectile {0.9, 0.95, 0.99), while larger ensemble size E 2,,2,3, 6} stabilizes bootstrapping off large expectile estimates = 0.99, as shown in Figure 3 This suggests that on datasets that require more value propagation, performance gains can be obtained by using a a V-ensebll and gradually increase T.
25
m= m=2.c-a0
GL (r- 6.8
m: a 8.94
m- 0.96
m= 0.99
0.00
0.25 0.50
0.75 1.00
Training Iterations
Figure 3: Normalized score of IVE-IQL with different ensem- ble size m and expectile T.
In Section 4.2 we mentioned that IVE introduces a bias-sparsity trade-off controlled by e in [V].. While the advantage bias could be optimistic when c < 0.5. The value functions learned by IVE is still pessimistic due to the target + min; V. Here we demonstrate the impact of EV[]] on policy performance. As shown in Figure 4 E [V]] could effectively aid training by trading bias off for sparsity control. For most MuJoCo datasets we find that large 1.0 is beneficial for action filtering. In cases where the advantage more sparse (e.g. antmaze-u-d), a small could facilitate learning. To promote understanding of the in-sample epistemic uncertainty, we present additional visualizations shown in Figure 15
in Appendix [
halfcheetah-m-w2 (IVE-S)
antmaze-u-d-v2 (IVE-I)
kitchen-m-va (IVE-I)
Normalized Score
50
Nomalzze Score
Normalized Score
60
60
80
40
45
0 M
20
20
-2
1
40
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Iterations
1c6
Training Iterations
1e6
Training Iterations
1e6
Figure 4: Normalized score of IVE-I/S with advantage estimates computed using different ensemble aggregates. [V] denotes the e-th quantile of the V -ensemble.
Effect of Adaptive Cloning Temperature (ACT). As discussed in Section 5 over-regularization can happen when the dataset is skewed towards suboptimal policies, and the average likelihood of dataset actions given by the learned policy can overshoot. We present learning curves of SQL and ACT-S on suboptimal datasets in Figure 5 During training, ACT successfully adjusts batch average log π T[aaa s) to neighborhood of the target likelihood HD. We can see that in the early stages of training. ACT performs similarly to weighted behavior cloning. On the other hand, a slightly lower Hip allows for more generalization and results in better performance.
halfcheetah-m--rw2
halfcheetah-m-ree
hallcheetall-m-r-V
Batch Average logn(als)
1.00
Normaized Soore
0.75
10 0.50
80
-10
35
(26
8.01
0.25
ACT-S (71p
11.00
00
0.25
0.75
-15
0.00
1.00
0.30
0.00
0.50
1.00
Training lierations
1e5
Training Berations
105
Training Heralions
Figure Comparison between SQL and ACT-S with varied H.p on suboptimal dataset.
8

---

Published as a conference paper at ICLR 2025
7.2 ABLATION STUDIES
In this section, we present the results of an ablation study over the components of ACTIVE, including In-sample V-Ensemble (IVE) and Adaptive Cloning Temperature (ACT). Learning curves and per- formance profiles are shown in Figures ((a) and 6(b) We can see that the performance improvement on compositional datasets like AntMaze is mainly attributed IVE, while ACT mainly improves performance on suboptimal datasets in the MuJoCo suite. We present additional results on Kitchen
tasks in Figure 12 in Appendix C
HujoCo
Maue
AntMaze
1.00
1.10
Wommsteed Scone
Normeliret Score
0.75
6.73
40
I0.
0.50
0.50
20
(VE-4Q)
0.25
AXTME-
IQL
0.25
ACTVV--0:
0.
6.c0
ACTME-10 (0-Q)
0.00
250
300
75
50
100
Training Rerations
(a) MuloCo
(b) AntMaze
Figure 6: Average learning curves with stratified bootstrap confidence intervals (CIs) and performance
profiles generated using rliable (Agarwal et al. 2021) on (a) MuloCo and (b) AntMaze tasks.
7.3 RUN TIME
Table 3: Run time of ACTIVE and baselines on the halfcheetah- medium-v2 dataset.
In this section, we compare the run time of ACTIVE and baseline in-sample algorithms by running them on the same hardware and software configuration for 1M steps. The re- sult is displayed in Table 3 We can see that while ACTIVE introduces ensembles for uncertainty estimation, for small net- works the run time does not increase signifieantly thanks to parallelization.
Algorithm
Iter/s
Time (min)
IQL
744
22.27
SQL
727
22.79
ACTIVE-I
650
25.89
ACTIVE-S
627
26.48
8 DISCUSSION
Although we have shown in Section 7.1 that adaptive temperature adjustment does improve final performance on certain datasets, SAC-style policy extraction loss in Equation (14) with an in-sample critic does not achieve similar results as EDAC (An et al. 2021) on MuJoCo tasks. We argue that critics trained an in-samlle manner may only generalize
close neighborhood of dataset samples. And for effective generalization to take place in offline RL tasks, it may be necessary to perform out-of-sample updates. Given that un-sam critics can be empirically more accurate (Fu al.. 2022). how to combine in-sample learning and out-of-sample learning for balancing generalization and regularization may be worth investigating. These discussions are beyond the scope of this work, and we leave them for future works.
Recent generalizations IQL such as EQL (Garg et al. 2023) studied the Bellman error distribution in MDPs and proposed using Gumbel regression to learn the V-function. And in ACTIVE, the Bellman error distribution could be affected by the Veensemlle. We analyze the Bellman error distribution of single-net IQL and IVE in Figure 13 in Appendix [ in which we do not observe significant difference between the two set of distributions. Nevertheless, the Bellman error distribution in offline RL methods may be worth investigating and remains area of further study.
CONCLUSION
We have introduced ACTIVE, an offline RL algorithm designed to alleviate over-regulaiization in existing in-sample learning methods. ACTIVE provides an ensemble-based value learning scheme and a policy extraction loss based on dual gradient descent that can be applied to any variant of implicit TD backups. Experimental results show that our method effectively mitigates overfitting of V-function and prevents over-regularization suboptimal datasets. Compared to existing in sampee offline RL algorithms, ACTIVE exhibits better stability and final policy performance on a variety of offiine datasets.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the National Science and Technology Major Project under Grant 2022ZD01 16401, the National Natural Science Foundation of China under Grant 62141605, and the Fundamental Research Funds for the Central Universities, China. We thank all the anonymous reviewers their insightful comments.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An Optimistic Perspective on Offline Reinforcement Learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. Advances in Neural Information Processing Systems, 34:29304-29320, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- Based Offline Reinforcement Learning with Diversified Q-Ensemble. Advances Neural Information Processing Systems, 2177436-7447,022
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv preprint 2016.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic Bootstrapping For Uncertainty-Driven Offline Reinforcement Learning. In International Conference on Learning Representations, 2022.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http: /ttph/gggtuub/ut com/ hoom///oole////// jax
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv Preprint arXiv:1606.01544 2016.
Jacob Buckman, Carles Gelada, and Marc G. Bellemare. The Importance of Pessimism in Fixed- Dataset Policy Optimization. In International Conference on Learning Representations, 2021.
William R. Clements, Bastien Van Delft, Benoit-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating Risk and Uncertainty in Deep Reinforcement Learning. arXiv Preprint arXiv: 1905.0905. 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, arXiv Preprint arXiv:2004.07219 2020.
Yuwei Fu, Di Wu, and Benoit Boulet. A Closer Look at Offline RL Agents. Advances in Neural Information Processing Systems, 1888591-8604 2022.
Scott Fujimoto and Shixiang Shane Gu. A Minimalist Approach to Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 14:20032--21444 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. In International Conference on Machine Learning. pp. 1587-1596. PMLR, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In International Conference on Machine Learning, pp. 1050-1059, 2016.
Divyansh Garg, Jocy Hejna, Matthieu Geist, and Stefano Ermon. Extreme Q-Learning: MaxEnt RL without Entropy. In International Conference Learning Representations, 2023.
10

---

Published as a conference paper at ICLR 2025
Seyed Kamyar Seyed Ghasemipour and Shixiang Shane Gu. Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters. Advances in Neural Information Processing Systems, 35:18267-18281, 2022.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMaQ: Expected- Max Q-Learning Operator for Simple Yet Effective Offline and Online RL. International Conference Machine Learning, pp. 3682-3699 PMLR, 2021.
Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline RL Policies Should be Trained to be Adaptive. International Conference on Machine Learning, pp. 7513-7530, PMLR, 2022.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algorithms and Applications. arXiv Preprint 1812 050000 2018.
Philippe Hansen-Fsttuuch lya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies. arXiv Preprint arXiv:2304. 10573, 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernl:: Convergence and Generalization in Neural Networks. Advances in Neural Information Prucessing Systems, 31, 2018.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is Pessimism Provably Efficient for Offline RL. In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- Based Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 33: 21810-2188 21823, 2021.
lly Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline Reinforcement Learning with Fisher Divergence Critic Regularization. In International Conference on Machine Learning. pp. 5783- PMLR, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. Advances Neural Information Processing Systems, 32: 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A Workflow for Offline Model-Free Robotic Reinforcement Learning. In Conference Robot Learning, volume 164, pp. 417-428. PMLR, 2022.
Jaehoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve Linear Models Under Gradient Descent. Advances in Neural Information Processing Systems, 32, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives Open Problems. arXiv Preprint arXiv:2005.016433, 2020.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 35:1711-1724 2022.
11

---

Published as a conference paper at ICLR 2025
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Na- fure, (1r551887540)552-55553 February 2015. ISSN 0028-0836, 1476-4687. doi: 10. 10.1038/n Maaaauree4236...
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. arXiv Preprint arXiv:2006.09235: 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv Preprint arXiv: 2019.
Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy Regularization with Dataset Constraint for Offline Reinforcement Learning. In International Conference on Machine Learning, pp. 28701-28717 PMLR, 2023.
J P Royston. Expected Normal Order Statistics (Exact and Approximate). APPLIED STATISTICS, 31 (2):161-165, 1982.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep Doing What Worked: Behavior Modelling Priors for Offline Reinforcement Learning. In International Conference on Learning Representations, 2020.
David Silver, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 42977587)4484489, January doi: 10. 103800/natu 16961.
2016. ISSN 0028-0836, 1476-4687.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. The Journal of Machine Learning Research, 15(1):1929-199 1958, 2014.
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning. International Conference on Machine Learning, pp. 33177-33194, PMLR, 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, Filamoura-Algare Portugal. October 2012. IEEE. ISBN 978-1-4673-1736- 978-1-4673-173755 978-1-4673-1735-1 doi: 10. 10.1100/ 20112.8800,20...8886000
Laurens der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008
Qing Wang, Jiechao Xiong, and Lei Han. Exponentially Weighted Imitation Learning for Batched Historical Data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6291-6300, 2018
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported Policy Optimiza- tion for Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 35: 31278-3122 31911, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior Regularized Offline Reinforcement Learning. arXiv Preprint arXiv:1911.11361. 2019.
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The In-Sample Sofumax for Offline Reinforcement Learning. In International Conference on Learning Representations, 2023.
Tengyang Xie. Ching-An Cheng, and Nan Jiang. Bellman-cconistent Pessimism for Offline Rein- forcement Leaming. Advances Neural Information Processing Systems, 34:6683-6694, 2021.
12

---

Published as a conference paper at ICLR 2025
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang. Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization. In International Conference on Learning Representations, 2023.
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust
Offline Reinforcement Learning via Conservative Smoothing. Advances in Neural Information Processing Systems, 35:23851-23866 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization. Advances in Neural Information Processing Systems, 33: 14144222-4414, 2020.
Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: Latent Action Space for Offline Reinforcement Learning. In Conference Robot Learning, volume 155, pp. 1719-1735. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
A
PROOF OF THEOREM 4.5
Following the assumptions and notations introduced in Section 4.3 we present the proof of Theo-
rem 4.3 The proof builds on existing work on NTK and ensemble-based offline RL methods (Lee
al. 2019 Ghasemipour & Gu 2022).
Proof. Consider the linearization (W.r.t. parameters) of Q(0) at initialization:
Qin(s, = = (s, + VoQ (8. a) (9tn - -000)
(16)
Let Q Qlin and consider performing updates on the linearized network to get (t) (2019) shows that with MSE loss and infinite network width, subject to technical conditions on the learning rate, the predictions of the trained networks Q(") coincide with the predictions of the trained linearized networks Q i.e. V(s, as, Qu (s, a) (s, Since we have augmented V train on X' using MSE loss by Theorem 3.1 and Assumption 4.2 we can analogously define the linearized Vin and analyze the dynamics of lin instead:
J. Lee et al.
Vin(() = ((0) (8) + (s)) (Nhin - -bb)))
(17)
Learning V. Following Lee al. (019)) (Section 2.2, Equations 9-1 1, by performing continuous-time gradient descent the limit of t
Lee
that (I - goot) - I), derive:
()- Vin (0') = (0)) + (0' (0,0)) ( (1))))
(18)
= =v('( + Cv (Q (x2) - v()
Under the NTK setting, Vs, Bens
Eens[V [V(
= 0. Therefore we have:
[V (1()(() (O')]
(00) Cv (0th v10 [0))] Cv -C (2)
Eens|
Eens
V ens
(v((1) (O')]
Egns
(v()(()) Cv (() vO) (Cv 2 (19)
(V(0 Cv (0)
Lenns
To simplify the notation, = V ens Vin ('))
Bootstrapping Q against V. By Assumption 4.4, the data matrix containing the learning target for Q is (t)) = R LCB(V (O')), where
Ys LCB(CCBVV (t) ii (s')) Eens (1)(() (s') V Vens (V(()) (s')).
(20)
Following Lee et al. 2019 (Section 2.2, Equation 9, f - co), we can derive:
( (X') = y(t))
(((+1) (2²) = (2)) - (,, (x,,)) -000
(21)
(0())(()) Cimp (D - (10)
Analogous the proof of Theorem 3.1 in Ghasemipour & Gu 2022) we can recursively compute ()())
14

---

Published as a conference paper at ICLR 2025
210) = R LCB(V (O')
= -R+y
Eens lin (O')]
Vens Iim
(O')
= R +
Cv (2))
Cv (x)) Cimp" ()(t-1))(((11) B)
-R+
R + B) + (CpCCmp
1 + ... (..r Cimp) (r
R (R+ (B)] (rCV Cimp) ( (R+1ICC(V
+ (n:D)imp)))
R+ ++t... [+...+ Cmpp ) ( (9- B) 0 (
(22)
Plugging into Equation
21 and using the NTK property, we have:
(x++)) Q (
(++1) (X)
[ ..... (1CCCimp))) R O
(23)
+ [ +1..... ......... (nCvOmmp) (Ω B).
B IMPLEMENTATION DETAILS
General. We implement ACTIVE and reproduce IQL (Kostrikov et 2022) and SQL (Xu et al. 2023) based on the author-provided source code. We mainly tune the implicit regularization level (a or T) along with ensemble size m and target likelihood HDD while most remaining hyperparameters remained unchanged from the corresponding baseline (IQL for ACTIVE-I, SQL for ACTIVE-S). For learning curves presented the paper, we evaluate the agent for 10 episodes every 5000 steps. For benchmark results in Table we follow IQL. and SQL. by averaging over 10 evaluations every 5000 training steps on MuJoCo/Kitchen. and averaging over 100 evaluations every 0.1M training steps on AntMaze. The full hyperparameters setting is shown in Tables 4 to
Ensemble Aggregate for Bootstrapping, We use min; V4, (s) as ensemble aggregate for bootstrap- ping on most D4RL datasets. walker2d-medium- www-rrdd-emdduu---paaa----eeee we use - v Σ VoL, (s) instead as empirically performs better. The full setup can be found in Table 7
Importance Weight in ACTIVE.
To isolate the effect of weight distribution and weight scaling
(which is controlled by adaptive temperature 9), for every batch we normalize w(s, a) in Equation (14)
by dividing each element by the batch mean (mean + c, = 1 x 10-4 for ACTIVE-S).
Tuning Hp. In this work, we tune Hp for each different dataset. To choose HD in offline manner, we first run IVE-SQL/IQL to obtain the average log likelihood Hy evaluated on the dataset (estimated using the final minibatches). And then starting with H.D Ho decrease HDD until the turning point of 3 (i.e. when 3 starts to decrease) gets lower than 1/2 of the total training iterations.
Software. We use the following software versions:
D4RL 1.1 (Fu et al. 2020) (Apache-2.0 license)
Jax 0.4.9 (Bradbury et al. 2018) (Apache-2.0 license)
MuJoCo 2.1.0 (Todorov et al. 2012) (Apache-2.0 license)
Gym 0.23.1 (Brockman et al. 2016 (MIT license)
Hardware. We use the following hardware:
NVIDIA RTX 3090
Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
15

---

Published as a conference paper at ICLR 2025
Table 4: ACTIVE-I/S general hyperparameters.
Hyperparameter
Value
10-1
Actor learning rate
2 x 10-4 for AntMaze in ACTIVE-S
10 10-4
Critic learning rate
2 10 -4 for AntMaze in ACTIVE-S
10TH
Value learning rate
2 10 for AntMaze in ACTIVE-S 256
Batch size
Optimizer
Adam 3 layers ReLU activated MLPs with 256 units
Network (all)
Discount
0.99
Polyak
0.005
Layer normalization
Off
Value Dropout
Off
Actor Dropout
Off (p 0.1 for Kitchen)
Table ACTIVE hyperparameters used Kitchen domain.
ACTIVE-I
ACTIVE-S
Dataset
T
PAWR
m
H2
a
m
C
H
kitchen-complete-vv
0.97
0.3
5
0.0
Off
0.05
5
0.0
Off
kitchen- partial-v0
kitchen-partial-00
0.9
0.5
5
0.0
Off
0.3
5
0.0
Off
bichen-mixed-vl
0.95
0.3
0.0
Off
0.1
7
0.0
Off
Table 6: ACTIVE hyperparameters used in AntMaze domain.
ACTIVE-I
ACTIVE-S
Dataset
BAWR
m
Hp
HD
antmaze-umaze-yy
0.99
3.0
0.0
-5.25
0.02
5
0.0
-5.55
antmaze-umaze-diverse--
0.99
10.0
6
0.0
-4.0
0.5
7
0.0
4.0
antmaze-medium-play
0.99
10.0
0.0
-5.0
0.01
5
0.0
-5.0
0.99
10.0
0.0
-5.0
0.02
5
0.0
-5.0
antmaze-large-play-v2
0.99
10.0
0.0
-5.25
0.02
5
0.0
-5.25
antmaze-large leetazz--arge-diversee..
0.99
10.0
0.0
-5.3
0.01
0.0
-5.3
Table 7: ACTIVE hyperparameters used in MuJoCo domain.
ACTIVE-I
ACTIVE-S
Ens. agg.
Dataset
BawR
Hp
Hi
for Q-update
malichhetah--mcciun-22
0.9
3.0
1.0
1.0
0.03
5
1.0
-1.0
min
hopper-medium--2
0.9
3.0
1.0
0.0
0.1
1.0
-0.9
min
walker2d-medium-y
0.9
3.0
1.0
1.0
1.0
min
halfcheetah- mediaaa---miiuuuuu--------
0.9
3.0
1.0
-11.0
0.1
5
1.0
-1.00
min
hopper-mediumt wendddiiu---pll--eee replay-v2 v2
0.9
3.0
1.0
-5.0
0.1
5
1.0
-5.0
min
walker2d-medinn medium-replay--v.
0.9
3.0
1.0
-7.0
1.0
0.0
-3.0
halfcheetah- halfcheetah-mmediu---
0.7
3.0
0.0
1.75
1.0
1.75
min
0.7
3.0
0.0
0.0
1.0
5
0.0
0.0
min
malker2d-mediume -x
0.7
3.0
0.0
1.75
1.0
5
0.0
1.75
min
16

---

Published as a conference paper at ICLR 2025
C
MORE RESULTS
nalfcheetah-medlum-y
hopper-medium-v2
walker2d-mediumm22
100
Normalized Score
Normalized Score Score
75
Normalized
20
SQL
IVE-SQL ACTIVE-SO!
25
0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Iterations
1e6
Training Ittraaioons
Training Iterations
halfchettah-mediu---raaaayy
hopper-mediun-repllyy
walkerz2-mmddiu--rppaayy
Normalized Score
Normalized Score
Normalized Score
40
100
75
50
20
50
25
Training iterations
1e6
Training Ittrations
le6
Training Iterations
1e6
halfcchetth--mediu-exppee
waller/2dmeddummeexpttt
100
Normalized Score
Normalized Score
Normalized Score
100
75
100
50
50
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training lterations
1e6
Training Ittrationn
1e6
Training Iterationn
1e6
Figure 7: Learning curves of ACTIVE-S, IVE-S and SQL on MuJoCo datasets.
antmaze-umaze-v2
antmaze-umaaze-divereew
antmaze-medium-playy-2
Nomalized Score
Normalized Score
80
75
60
60
Nomalizzd
50
40
40
IQU
25
IVE-ILL
ACTIVEIQU IQL
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Iterations
1e6
Training Iterations
1e6
Training Iterations
1e6
antrmaze-edduumddiveeeww
antmaze-large-playyv2
antmaaeelarge-ddivrreev
Normalized Score
80
Normallzer Sormllizeedeo Score
Normalized Score
20
20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Iterations
Training Iterations
Training Iterations
Figure Learning curves of ACTIVE-I, IVE-I and IQL on AntMaze datasets.
17

---

Published as a conference paper at ICLR 2025
kitchen-complete-v0
kitchen-partial-v0
kitchen-mixed-vo
80
80
80
Normalized Score
Normalized Score
60
60
40
Normalized
20
20
IVE SQL
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Iterations
Training Iterations
Training Iterations
Figure 9: Learning curves IVE-S and SQL on Kitchen datasets.
kitchen-complete-v0
kitchen- partial-v0 kitchen-partial-v0
kitchen-mixed-vo
Normalized Score
80
Normalized Score
60
60
Normalized
40
20
IQL
20
IVEIQUL
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Iterations
Training Iterations
1e6
Training Iterations
1e6
Figure 10: Learning curves of IVE-I and IQL. on Kitchen datasets.
antmaze-umaze-dvv2
antmaze-umaze-d-22
10000
80
Batch Average Qls, al
Normalized Score
60
5000
40
0.00
0.75
1.00
0.25
0.50
0.75
Training llerations
1e6
Training Iterations
166
IOL (T D. 0.99)
IQL (T- 0.99, offect 20.0)
IVE-IOL (T- 0.99)
(t 0.9 95)
(t-
40.00
Figure 11: Normalized score and batch average Q(s, a) of IQL and IVE-IQL with varied r. Addi- tionally, we train IQL agents with r(s, a) + +VV(s') ) + offset as the Q-target. We see that a constant penalty is not enough to suppress error accumulation and amplification.
Kitchen
Kitchen
J 1.00
Normalized Score
60
Fraction of Scores
40
0.50
SQL
IQL
0.25
IVE-SQL
IVE-IQL
0
0.00
100
150
20
Training Iterations
Normalized Score
Figure 12: Average learning curves with stratified bootstrap confidence intervals (CIs) and perfor-
mance profiles generated by the rliable library (Agarwal al. 202I) on Kitchen tasks.
18

---

Published as a conference paper at ICLR 2025
halfchestah
hepperms
walker22
Kund 2o
0.15
(r-
-------
Kard Derato Eltmationn
Kame Sanee Datnow on
0.30
0.3
0.075
0.08
0.025
Av18
0.030
Oelimann Error
Scliman Ertr
Bellman E01r
meliman Eroo
Figure 13: Kernel density estimation of Bellman errors obtained by training on datasets from the D4RL benchmark using single network SQL (gray) and V-ensemble (blue).
Kerne Dersity Estimation
halfcheetahh-rame
Karel Darsiy Estimation
saffehestah-medxx
Vere Dondity Estmaaiio
halfcheetah exp-v2
0.3
0-k
1.0
---
0.5
0.1
0.0
0.0
0.0
10
10
5.0
2.5
0.0
2.5
5.0
5.0
2.5
0.0
5.0
Advaniage Estimate
Advantage Estimate
Advaniage Estimate
Kene Dereity Extimation
0.08
walker2d-ran-v2
Extimation
1.00
walker26-med-v2
Edtimation
1.5
walker2d-exp-12
0.06
1.0
0.04
0.50
0.5
0.02
0.00
0.00
0.0
-50
-25
25
50
-10
-5
5
10
-10
-5
0
Advartage
Estimate
Advantage Estimate
Advantage Estimate mate
Extimation
0.100
happer-ranv2
Kome Denutty Estimation
happer-mad
Densty Estimatioo
nopper-exp-v2
Ago
0.8
AQo
Ag
0.075
-
0-
0.0
0.050
0.5
0.025
0.000
0.0
0.0
-50
-25
50
-10
-5
-10
0
10
Advantage Estimate
Advantage
Extmate
Acvantage Estimate
Figure 14: Kernel density estimation of the advantage distribution on datasets from the D4RL benchmark obtained using single network SQL (gray) and Vensemble (blue and orange).
kttchen-mixed-v00
kitchen-mixed-vi
kitch kitchen- en-iixe--00
krtchen-mixed-vv
180
175
160
rewar
150
reward
125
100
-5
20
20
-20
-20
20
Figure 15: Uncertainty estimate mean; V; LC)(V)) and reward visualization using t-SNE (van der
Maaten & Hinton 2008) projections of observations in Kitchen trajectories.
Table 8: Comparison against SAC-N on Kitchen tasks.
Dataset
SAC-N (m 20)
SAC-N (m 40)
ACTIVE-S
kitchen-c
0.0 + 0.0
0.0 + 0.0
66.5 + 3,3
kitchen-p
0.0 + 0.0
0.0 0.0
73.5 + 3.0
kitchen-m
0.0 + 0.0
0.0 + 0.0
73.2 + 1.4
19


Reviews:
[{"review_id": "fQ96vq2rIL", "reviewer": "Reviewer_J6er", "summary": "This paper aims to address the overestimation problem of offline RL methods that learn in-sample value functions to avoid querying out-of-distribution samples. The main idea is (i) to learn ensemble of V functions and use clipped V function for learning V functions and (ii) use automatic tuning of hyperparameter used for in-sample value learning such as quantile regression. Arguments for introducing the idea are strong. Several ablation/analysis are provided to support the claims in the paper. Experiments are conducted on D4RL benchmark.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength of this paper is that the paper does not have any big flaw and does not make any big mistake. Writing is clear, motivation is clear, intuition is clear, and the results look okay. The idea of learning ensemble of value functions has been very popular in offline RL context and also clipped Q learning is also widely-used, so the idea of using it for regularizing IQL-like method is solid.", "weaknesses": "- The main problem of this paper is that it's not clear how important the problem this paper aims to address, or the importance of such problem is not clearly evidenced & presented in the writing, mainly because D4RL is quite saturated benchmark at this point and it's not clear how the observation / results in this benchmark would transfer to other scenarios.\n- It's not clear the motivating overfitting problem can be generally observed in diverse domains or it is only happening on D4RL tasks. Motivating figures could be provided on more tasks / domains to support that overfitting is a common problem across various scenarios. More results on different tasks from different benchmarks such as NeoRL or visual offline RL benchmarks can be a plus.\n- Weak performance compared to MSG is not clearly justified to me; if that is from not using CQL-like regularizer, can ACTIVE be better if it incorporates similar regularizer?\n- It could be nice to make tables be more self-contained as far as possible. For instance, make it clear that what -I and -S means in each caption, or make it clear what $m$ means in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sJHnzatMn7", "reviewer": "Reviewer_wzeE", "summary": "The paper focuses on resolving the problem of overregularization induced by in-sample learning methods in an offline RL setting. It provides a motivating experiment to demonstrate that existing approaches like IQL could still suffer from overestimation in the late learning stages. The proposed method differs from existing IQL algorithms mainly in two ways: 1) it introduces an ensemble to learn the value function; and 2) it introduces an entropy constraint on the policy learning, which is further reformulated by Lagrangian and solved by dual gradient descent. Empirical results on commonly used offline RL datasets are presented to verify the effectiveness of the proposed algorithm.", "questions": "see above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper studies an important topic.\n2. The paper presents promising empirical results. \n3. The paper explains its main algorithm clearly.", "weaknesses": "The main weaknesses are as follows. \n\nDefinition of Over-Regularization: The paper repeatedly emphasizes that existing approaches suffer from over-regularization induced by in-sample constraints, while the proposed method can mitigate this. However, the term \"over-regularization\" is never clearly defined. Intuitively, it might imply that the learned action values are somewhat underestimated due to in-sample constraints. Yet, it is confusing that Section 4.1 claims that existing algorithms like SQL still overestimate. If overestimation is an issue, doesn’t this suggest that there is insufficient regularization? And, isn't it simpler to bring in methods to mitigate overestimation (there were already many methods proposed for this)? \n\nTheoretical Clarity: The theoretical result appears confusing. Theorem 4.5 states that the initial error can be penalized, but it is unclear how this penalization of initial error relates to preventing overestimation.\n\nChoice of Competitors for Empirical Testing: Figure 1 omits tests on the original IQL and CQL. Additionally, a fair comparison seems to be missing; the proposed approach should ideally be compared with in-sample softmax by Chenjun Xiao et al., as cited by the authors. This paper directly bootstraps from in-sample actions and also incorporates entropy regularization, and it was proved to converge to an optimal entropy-regularized policy on the empirical MDP defined by offline dataset.\n\nEffectiveness of the Proposed Method: Line 286 states, \"...but cannot completely solve the overregularization problem.\" However, it remains questionable whether even with the proposed dual gradient descent method, this problem can be fully resolved.\n\nNeed for Ablation Study: The paper should include an ablation study on the use of the ensemble value function and dual gradient descent to carefully verify their effect and necessity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mVP44zRhV7", "reviewer": "Reviewer_L27K", "summary": "This paper addresses the observed overfitting issue of V-functions in offline reinforcement learning. Specifically, the authors claim that a V-function may overfit to unstable overestimates in the target Q-values, which can, in turn, adversely impact the learning of both the actor and critic. The authors propose two primary techniques to mitigate this issue:\n\n**First Technique: IVE (V-function Ensemble and Pessimistic Estimation).** To mitigate overfitting and prevent exploding Q-values, the authors propose maintaining an ensemble of V-functions and using their minimal estimate as the target during policy improvement:\n\n$$L\\_V^f(\\psi\\_i) = \\mathbb{E}\\_{(s,a)\\sim D} [f(Q\\_{\\hat{\\theta}} - V\\_{\\psi\\_i}(s))]$$ \n\n$$L\\_Q(\\theta) = \\mathbb{E}\\_{(s,a,s')\\sim D} [ \\Big(r(s,a) + \\gamma \\min\\_i V\\_{\\psi_i}(s') - Q\\_\\theta(s,a)\\Big)^2]$$\n\n, where $f$ is a general distance function. Seems the author use $\\alpha$-divergence, as implied in Algorithm 1 (ACTIVE) on page 6. \n\nThe intuition behind this V-function ensemble is that for unstable or highly overestimated target Q-values, there is likely to be at least one V-function instance in the ensemble that does not fit this overestimated target well. This imperfect approximation helps prevent catastrophic overestimation.\n\n**Secondly, ACT (Adaptive Cloning Temperature)**\n\nThe authors propose an adjustment to the advantage function used in policy extraction, building on modifications to the vanilla Advantage-Weighted Regression used in SQL and IQL:\n\n$$\\hat{A}(s,a) := Q\\_{\\hat{\\theta}}(s,a) - \\mathbb{E}^c \\[V\\_{\\psi_i}\\](s)$$\n\nwith $\\mathbb{E}^c$ denotes the c-th quantile of the V-predictions. Besides, the author conduct policy extraction on both Q-function and A-function, which is different to IQL and SQL.", "questions": "I tend to rate this paper as “marginally below the acceptance threshold” for the following reasons:\n\n1. **V-function ensemble in offline RL**: Introducing a V-function ensemble for offline RL is somewhat interesting. Although V-function ensembles are new to me, they appear conceptually similar to existing Q-function ensembles, such as REM and other methods with pessimistic estimates (e.g., SAC-N, PBRL, EDAC, VPQ).\n2. **Insight on overestimation in sota Offline RL**: This paper makes an interesting observation that sota offline RL algorithms still encounter overestimation issues in some scenarios. While the authors refer to this as “overfitting,” it is, in effect, overfitting to unstable overestimated Q-values.\n3. **Clarity of contributions.** The contributions of this paper feel somewhat overshadowed by an abundance of technical details. The modifications proposed by this paper include:\n    1. *Equations (6) and (7)*: A V-function ensemble designed to approximate stable Q-targets while avoiding unstable overestimated values,\n    2. *Equation (6)*: Using a suitable  f-divergence for mapping the V-function to the Q-value of the best action in the dataset, and\n    3. *Equations (14) and (15)*: Extracting the policy based on the learned Q-function, V-function, and a quantile.\n    \n    Each of these components would benefit from comprehensive analysis, but in this paper, they are presented in a tightly condensed manner aimed at deriving a sota offline RL algorithm. While I respect the author’s effort, this approach makes it challenging to appreciate each individual contribution.\n\nIn summary, I find the core idea of this paper less compelling, with limited new insights or utility in the analysis. The primary contribution appears to be the development of a potentially strong offline RL algorithm, rather than a well-motivated solution for offline RL. Nevertheless, I rate this paper “marginally below the acceptance threshold” to avoid letting my personal preferences bias the review of this well-executed experimental study.\n\nQuestion for the author: I believe Equation (8) may be unnecessary, as it could potentially confuse readers, especially when read alongside Equations (14) and (15). Right?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-organized. Presenting work with two distinct techniques can be challenging, yet this paper does so effectively. The descriptions of each technique are clear, with a justification provided for the first technique, IVE.\n\n2. It is a noteworthy finding that sota offline RL algorithms, such as SQL and IQL, still encounter overestimation issues in certain scenario. The introduction of a V-function ensemble to mitigate this problem yields incremental improvements, as demonstrated in Table 2 of the experiments. I also observed that the reproduced IQL results are slightly better than those in the original paper, while the SQL results are slightly lower—both of which align with expectations based on my own experience reproducing these algorithms in my research.", "weaknesses": "**(Minor)** **Inconsistent naming of techniques**. The names of the proposed techniques are inconsistent, which may confuse readers. The second technique, ACT, is mentioned under various names, including *automating likelihood adjustment*, *adaptive cloning temperature*, and *adaptive imitation*. Maintaining a single, consistent name for each technique would improve clarity.\n\n**(Minor) Insufficient theoretical analysis for V-function ensemble.** The theoretical analysis supporting the V-function ensemble feels somewhat lacking. My understanding is that the theorem attempts to show that recursively expanding the pessimistic target could provide certain benefits:\n$$Q = [1 + \\gamma + ... + \\gamma^t]r + (\\Omega - B) \\gamma [1 + \\gamma + ... + \\gamma^t] + \\mathcal{O}$$\nwhere B is some form of $V_\\text{var}$, and $\\Omega$ is a form of initial error. I believe this is not entirely new to the RL community. Additionally, the analysis lacks a convergence guarantee to ensure that such a recursive expansion can be reliably conducted.  Besides, the assumption 4.3 that $\\pi_\\text{imp} \\propto \\dfrac{|f'(Q(s,a) - V^*(s))|}{|Q(s,a) - V^*(s)|}$ is fixed throughout the learning process, seems infeasible. Since we are continuously updating the Q-function during Q-learning, it would be challenging to hold this assumption constant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZfodZYMfYB", "reviewer": "Reviewer_oLG4", "summary": "This paper proposes an offline reinforcement learning algorithm ACTIVE that can mitigate the over-regularization problem of existing in-sample learning schemes. This paper first demonstrates through experiments that using a higher $\\tau$ in IQL (or using a lower $\\alpha$ in SQL) will lead to unstable estimates during the training process. And they demonstrate theoretically that V-ensemble can mitigate value overestimation by suppressing the accumulation of initial value errors, and experimentally demonstrates that ACTIVE can mitigate the overfitting of the value function and thus outperforms existing in-sample methods.\n\nI have reviewed this manuscript submitted to NeurIPS 2024, and the authors have modified the manuscript in response to my comments and added comparison experiments with some ensemble algorithms, and these modifications address most of my questions.", "questions": "1. Besides the run time, can you provide a comparison of the algorithm's parameters?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This method has reasonable motivation and theoretical basis.\n2. This method could help address the issue of in-sample value learning and policy extraction methods performing poorly on suboptimal or compositional datasets.\n3. The experiments demonstrate that the method achieves some empirical improvements.", "weaknesses": "1. The innovation of this paper is moderate. As far as I know, the use of ensemble tricks to stabilize value estimation has been widely used in the RL field. \n2. The algorithm introduces two new superparameters $c$ and $\\mathcal{H}$, which are somewhat sensitive according to Figure 4 and Figure 5. Especially $\\mathcal{H}$, it needs to be adjusted for specific datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)