Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
Authors: Gaon An
Seungyong Moon
Jang-Hyun Kim
Hyun Oh Song
Venue: NeurIPS 2021 Poster (2021)
Keywords: Offline Reinforcement Learning

Abstract: Offline reinforcement learning (offline RL), which aims to find an optimal policy from a previously collected static dataset, bears algorithmic difficulties due to function approximation errors from out-of-distribution (OOD) data points. To this end, offline RL algorithms adopt either a constraint or a penalty term that explicitly guides the policy to stay close to the given dataset. However, prior methods typically require accurate estimation of the behavior policy or sampling from OOD data points, which themselves can be a non-trivial problem. Moreover, these methods under-utilize the generalization ability of deep neural networks and often fall into suboptimal solutions too close to the given dataset. In this work, we propose an uncertainty-based offline RL method that takes into account the confidence of the Q-value prediction and does not require any estimation or sampling of the data distribution. We show that the clipped Q-learning, a technique widely used in online RL, can be leveraged to successfully penalize OOD data points with high prediction uncertainties. Surprisingly, we find that it is possible to substantially outperform existing offline RL methods on various tasks by simply increasing the number of Q-networks along with the clipped Q-learning. Based on this observation, we propose an ensemble-diversified actor-critic algorithm that reduces the number of required ensemble networks down to a tenth compared to the naive ensemble while achieving state-of-the-art performance on most of the D4RL benchmarks considered.

Incertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
Gaon 12 Seungyong Moon Jang-Hyun Kim Hyun Oh Song? 123
Seoul National University!
Neural Processing Research Center?
DeepMetrics
Wwhite0234,symoone haw.oalii.uaaauuuuuuuuuuuuu hyunoh) @mllab. snu. kr.
Abstract
Offline reinforcement learning (offline RL), which aims to find an optimal policy from a previously collected static dataset, bears algorithmic difficulties due to function approximation errors from mut-of-distribution (OOD) data points. To this end, offline RI. algorithms adopt either a constraint or penalty term that explicitly guides the policy to stay close to the given dataset. However, prior methods typically require accurate estimation the behavior policy or sampling from OOD data points, which themselves can be non-trivial problem. Moreover, these methods under-utilize the generalization ability of deep neural networks and often fall into suboptimal solutions too close the given dataset. In this work, we propose an uncertainty- the confidence of the Q-value prediction and does not require any estimation sampling of the data distribution. We show that the clipped Q-learning, a technique widely used in online RI, be leveraged to successfully penalize OOD data points with high prediction uncertainties. Surprisingly, we find that is possible to substantially outperform existing offline RL methods on various tasks by simply increasing the number of Q-networks along with the clipped Q-learning. Based this observation, we propose ensemble-diversified actor-critic algorithm that reduces the number of required ensemble networks down to tenth compared to the naive ensemble while achieving state-of-th the-art performance on most of the D4RL benchmarks considered.
offline RL method that takes into account
1 Introduction
Over the recent years, deep reinforcement learning (deep RL) has achieved considerable success in various domains such as robotics [20], recommendation systems [6], and strategy games [26]. However, a major drawback of RL algorithms is that they adopt an active learning procedure, where training steps require active interactions with the environment. This trial-and-error procedure can be prohibitive when scaling RL to real-world applications such as autonomous driving and healthcare, as exploratory actions can cause critical damage to the agent or the environment [19]. Offline RL, also known batch RL, aims to overcome this problem by learning policies using only previously collected data without further interactions with the environment [2. 11, 19].
Even though offline RL is promising direction to lead more data driven way of solving RL problems, recent works show offline RI. faces new algorithmic challenges [19]. Typically, the coverage of the dataset is not sufficient, vanilla RL algorithms suffer severely from extrapolation
First authors have equal contributions Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

error, overestimating the Q-values of most offline RL methods apply some constraints or penalty terms on top of the existing RL algorithms to enforce the learning process to be more conservative. For example, some prior works explicitly regularize the policy to be close to the behavior policy that was used collect the data [11, 15]. A more recent work instead penalizes the Q-values of OOD state-action pairs to enforce the Q-values to be more pessimistic [16].
(OOD) state-action pairs [15]. To this end,
While these methods achieve significant performance gains over vanilla RL methods, they either require an estimation of the behavior policy or explicit sampling from OOD data points, which themselves can be non-trivial to solve. Furthermore, these methods do not utilize the generalization ability of the Q-function networks and prohibit the agent from approaching any OOD state-actions without any consideration on whether they are good or bad. However, if we can identify OOD data points where we can predict their Q-values with high confidence, is more effective not to restrain the agent from choosing those data points.
From this intuition, we propose an uncertainty-baat based model-free offline RL method that effectively quantifies the uncertainty of the Q-value estimates by an ensemble of Q-function networks and does not require any estimation sampling of the data distribution. To achieve this, we first show that well-known technique from online RL, the clipped Q-learning [10], can be successfully leveraged as an uncertainty-bastt penalization term. Our experiments reveal that we can achieve state-of-th of-the-art performance on various offline RL tasks by solely using this technique with increased ensemble size. To further improve the practical usability of the method, we develop an ensemble diversifying objective that significantly reduces the number of required ensemble networks. We evaluate our proposed method on D4RL benchmarks [9] and verify that the proposed method outperforms the previous ofte--of-the-at by a large margin on various types of environments and datasets.
2 Preliminaries
We consider an environment formulated as a Markov Decision Process (MDP) defined by a tuple (8, r,,, do.
7), where S is the state space, A is the action space, T(s' s, a) is the transition probability distribution, r: S x A - R is the reward function, do is the initial state distribution, and 7 E (0, 1] is the discount factor. The goal of reinforcement learning is find an optimal policy Taa s) that maximizes the cumulative discounted reward Eg, ar 12 7 ('r(s), a a.).. where So ~ do()). a, ~ a(- st), and $ T-- s,, at).
One of the major approaches for obtaining such a policy is Q-learning [12, 20] which learns a state-action value function Qs, a) parameterized by neural network that represents the expected cumulative discounted reward when starting from state s and action a. Standard actor-critic approach
[14] learns this Q-function by minimizing the Bellman residual (Q,(s, a) 8 Qs a)) where 8eta (s, a) = (- [r(s.a) a) + y Fa Q+ (s', a') is the Bellman operator. In the context of offline RL, where transitions are sampled from a static dataset D, the objective for the Q-network becomes minimizing
(Q+) == (sa,")>> (Qo(s..a) (r(s, a) + Ea' Ba' 12 (s', a²))))) a')))) )
(I)
where Qa represents the target Q-network softly updated for algorithmic stability [20]. The policy, which is also parameterized by a neural network, is updated in alternating fashion maximize the expected Q-value: Jp(TT) ;= Ba--Da--To(1s)) [Q, a)]
However, as the policy is updated to maximize the Q-values, the actions a' sampled from the current policy in Equation (1) can be biased towards OOD actions with erroneously high Q-values. In the offline RL setting, such errors cannot be corrected by feedback from the environment as in online RL. To handle the error propagation from these OOD actions, most offline RL algorithms regularize either the policy [11, 15] or the Q-function [16] to be biased towards the given dataset. However, the policy regularization methods typically require
accurate estimation of the behavior policy. The previous -fate-of-the-ar method CQL [16] instead learns conservative Q-values without estimating the behavior policy by penalizing the Q-values of OOD actions by
min Jg(Qo) + a BewD (aw()a IQo (s) E(g,a)~D (Qo (s, a)).
2

---

where is an approximation of the policy that maximizes the current Q-function. While CQL does not need explicit behavior policy estimation, requires sampling from an appropriate action distribution sa(: s).
3 Uncertainty penalization with Q-ensemble
halfcheetah-medium
hopper-medium
Average Return
6000
Average Return
4000
DQL
SAC-2
1000
SAC-3
SAC-2
SAC-5
500
SAC-50
SAC-10
SAC-200
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Training Steps
166
Training Steps
166
Figure 1: Performance of SAC-N on halfcheetah- and hopper- datasets while varying N, compared to CQL. 'veraae Return' denotes the undiscounted return of each policies on evaluation. Results averaged over 4 seeds.
In this section, we turn our attention to a conventional technique from online RL, Clipped Double Q- Learning [10], which uses the minimum value of two parallel Q-networks as the Bellman target: y
r(s, a) + (Ba'<a(--s")) minnmm,,2 Qwy (s', a Although this technique was originally proposed in online RL mitigate the overestimation from general prediction errors, some offline RL algorithms
[11, 15, 28] also utilize this technique to enforce their Q-value estimates to be more pessimistic. However, the isolated effect of the clipped Q-learning offline RL was not fully analyzed in the previous works, as they use the technique only as an auxiliary term that adds up to their core methods.
To examine the ability of clipped Q- Q-learning to prevent the overestimation in offline RL on its own, modify SAC [12] by increasing the number of Qeensembles from to N:
min By
(Qo.(s.a)
(r(s, a) + + En'nmat-le)
min j-i.....
Qaş (s'.a') B log (a' s (a(ss)))
max EawD
min Qa, (s. a) 3 (a 8) j=I...,,
for L....,N. N. We denote this modified algorithm as SAC-N.
Figure 1 shows the preliminary experiments on D4RL salffheetah-mmddiiee and hopper-medium datasets [9] while varying N. Note that these datasets are constructed from suboptimal behavior policies. Surprisingly, as we gradually increase N, we can successfully find policies that outperform the previous state-of-the-aar method (CQL) by large margin. In fact, SAC-N outperforms CQL on various types of environments and data-collection policies.
will present in Section 5,
To understand why this simple technique works so well, we can first interpret the clipping procedure (choosing the minimum value from the ensemble) as penalizing state-action pairs with high-variance Q-value estimates, which encourages the policy favor actions that appeared in the dataset [11]. The dataset samples will naturally have lower variance compared to the OOD samples as the Bellman residual term in Equation (2) explicitly aligns the Qavau predictions for the dataset samples. More formally, we can regard this difference variance as accounting for epistemic uncertainty [8] which refers to the uncertainty stemming from limited data and knowledge.
Utilization of the clipped Q-value relates to methods that consider the confidence bound of Q-value estimates [24]. Online RI methods typically utilize the Q-ensemble to form an optimistic estimate of the Q-value, by adding the standard deviation the mean of the Q-ensembles [18]. This optimistic Q-value, also known as the upper-confidence bound (UCB), encourage the exploration

---

of unseen actions with high uncertainty. However, in offline RL, the dataset available during training fixed, and we have to focus on exploiting the given data. For this purpose, is natural to utilize the lower-confidence bound (LCBB of the Q-value estimates, for example by subtracting the standard deviation from the mean, which allows us to avoid risky state-actions.
The clipped Q-learning algorithm, which chooses the worst-case Q-value instead to compute the pessimistic estimate, can also be interpreted as utilizing the LCB of the Q-value predictions. Suppose Q(s, a) follows a Gaussian distribution with mean m a) and standard deviation a a). Also, let (s, م be realizations of Q(s, a). Then, we can approximate the expected minimum of the realizations following the work of Royston [23] as
N 8 (N== o )= N 7 (N-
min (5(s,a) = m(s. a) Φ-1
(3)
j= N
where is the CDF of the standard Gaussian distribution. This relation indicates that using the clipped Q-value is similar penalizing the ensemble mean of the Q-values with the standard deviation scaled by a coefficient dependent on N.
15.0
N=1
10
12.5
N-5
Clp Penelty
10.0
Penalty (O0D)
5.0
2
2.5
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1.0
2.0
3.0
1.0
2.0
3.0
Traning Stops
1wB
Traning Steps
16b
Traning Steps
1e6
(a) Clip penalty
(b) Standard deviation
(c) Clip penalty gap
Figure 2: (a) and (b) each plots the size of the clip penalty and the standard deviation of the Q- value estimates for in-distribution (behavior) and OOD (random) actions while training SAC-10 on halfcheetah-meeeee dataset. (c) plots the gap of the clip penalty between the in-distribution and OOD actions while varying N. Results averaged over 4 seeds.
We now move to the empirical analysis of the clipped Q-learning. Figure 2a compares the strength of the uncertainty penalty on in-distribution and OOD actions. Specifically, we compare actions sampled from two types of policies: (1) the behavior policy which was used to collect the dataset, and (2) the random policy which samples actions uniformly from the action space. For each policy, we measure the size of the penalty from the clipping as (x=D) Σi= Q (s, a)
minj==...... Qa, (s, a)]. Figure 2a shows that the clipping term penalizes the random state-action pairs much stronger than the in-distribution pairs throughout the training. For comparison,
also measure the standard deviation of the Q-values for each policy. The results in Figure 2b show that as we conjectured, the Q-value predictions for the OOD actions have a higher variance. We also find that the size of the penalty and the standard deviation are highly correlated, as we noted in Equation (3).
As observe that OOD actions have higher variance on Q-value estimates, the effect of increasing N becomes obvious: it strengthens the penalty applied to the OOD samples compared to the dataset samples. To verify this, measured the relative penalty applied to the OOD samples in Figure 2c and found that indeed the OOD samples are penalized relatively further as N increases.
4 Ensemble gradient diversification
Even though SAC-N outperforms existing methods on various tasks, it sometimes requires an excessively large number of ensembles to learn stably (e.g., N 550 for hopper-mediumg While investigating its reason, we found that the performance of SAC-N is negatively correlated with the degree to which the input gradients of Q-functions V VaQa, (s, a) are aligned, which decreases with N. Figure measures the minimum cosine similarity between the gradients of the Q-functions min;y ( Qv. (s, a), VaQo, (s. a)) to examine the alignment of the gradients while varying N on the D4RL hopper-medium dataset. The results imply that the performance of the learned policy degrades significantly when the Q-functions share similar local structure.

---

Qa (8.))
(5,
Valo, a)
Valo, as
W1W1
Q
V Vo2 (s.
VaQ (s.
Ay Wa
p0a (s.a)
A2
Va Valos (s,a) a)
2
(ar(Qu; (8, kw2)) small that a + kw2 is not sufficiently penalized.
Var(Qu, a kw) is large for every direction w so that all OOD actions are sufficiently penalized.
(a) Without ensemble gradient diversification
(b) With ensemble gradient diversification
Figure 3: Illustration of the ensemble gradient diversification. The vector X,w; represents the normalized eigenvector w; of Var Var(VnQ0, a)) multiplied by its eigenvalue A;
We now show that the alignment of the input gradients can induce insufficient penalization of near-distribution data points, which leads to requiring a large number of ensemble networks. Let VaQu, (s, v, a) be the gradient of the j-th Q-function with respect to the behavior action a and assume the gradient is normalized for simplicity. the gradients of the Q-functions are well-aligned as illustrated Figure 3a, then there exists unit vector w such that the Q-values for the OOD actions along the direction of w have low variance. To show this, we first assume the Q-value predictions for the in-distribution state-action pairs coincide, i.e., Qo, (s, a) = Q(s, a) for j = L...., .N. Note that this can be optimized by minimizing the Bellman similarity between the input gradients of error. Then, using the first-order Taylor approximation, the sample variance of the v-values at OOD action along varying the number of Q-functions. can be represented as
1.0-
4000
Minimum Cosine Similarity Average
Cosine Similarty
Avera Average Return
0-
-200
-..5-
1000
2
5
1.00
20
50
100
200 500
Number of Q-functions
Figure 4: Plot of the minimum cosine
Q-functions and the average return while
Var (Qo, (s. a a kw)) ~ Var (Qo, (s, (s,,
Var (Q(s, + k VaQo,
Var ((w, VaQo,
War (V Q (s. w,
where - denotes an inner-product, & E IR, and Var (v (s, a)) is the sample variance matrix for the input gradients VaQe, (s, a). One interesting property the variance matrix is that its total variance, which is equivalent to the sum of its eigenvalues, can be represented as function of the norm of the average gradients by Lemma
Lemma 1. The total variance of the matrix Var (VaQ (s, is equal l| where q I Ei N j=m VaQo, (s,a) a).
Let Amin be the smallest eigenvalue of Var (vaQo, (s, a)) and Wmin be the corresponding normalized eigenvector. Also, let Then, using Lemma 1, we can prove that the variance of the Q-values for an OOD action along Wmin upper-bounded by some constant multiple of e, which given by Proposition 1.
be the value such that min;y) (VaQo. (s, a), VaQe, o, (s, a))
c.
Proposition 1. Suppose Q6, (s, hood of a for all j [N]. Let Auin and Wini be the smallest eigenvalue and the correspond- ing normalized eigenvector of the matrix Var V VaQu, 2, (s, a)) and > 0 be the value such that
Q(s, a) and Qo, (s..) is locally linear in the neighbor-
5

---

miney (V.Qo, VaQu, (s. = = Then, the variance of the Q-values for an OOD (s. action the neighborhood along the direction of Wmin upper-bounder as follows:
N
Var (Qe, (s, kwmin)) S
K2C,
A
N
where |A| is the action space dimension.
We provide the proofs in Appendix A. Proposition I implies that if there exists such > 0 that is small, which means the gradients of Q-function are well-aligned, then the variance of the Q-values for an OOD action along specific direction will also be small. This in turn degrades the ability of the ensembles to penalize OOD actions, which ultimately leads to requiring a large number of ensemble networks.
To address this problem, propose regularizer that effectively increases the variance of the Q- values for near-distribution OOD actions. Note that the variance lower-bounded by some constant multiple of the smallest eigenvalue Amiu-
Var (Qo, (s, a + kw)) k2 TTwTVar (s. w
2 k2 w Var (VaQ (s,
a)) Wmin
k2 Amin-
Therefore, an obvious way to increase this variance is maximize the smallest eigenvalue of Var Qva (s, a)), which can be formulated as
maximize Eg,a~D [Amin (Var (vaQo, (s, a)))]
where ø denotes the collection of the parameters {Øj} There are several methods to compute the smallest eigenvalue, such as the power method or the QR algorithm [27]. However, these iterative methods require constructing huge computation graphs, which makes optimizing the eigenvalue using back- propagaoooatnon inefficient. Instead, we aim to maximize the sum of all eigenvalues, which equal to the total variance. By Lemma 1, it equivalent to minimizing the norm of the average gradients:
1
N Valo,
minimize EgDDnD
Σ
VaQo,
(4)
With simple modification, we can reformulate Equation (4) as diversifying the gradients of each Q-function network for in-distribution actions:
minimize JES (Es) B
E (s, Va (a. ISHSIE
N
ESorro,
Concretely, our final objective can be interpreted measuring the pairwise alignment of the gradients using cosine similarity, which we denote as the Ensemble Similarity (ES) metric FSS.. (s, a), and minimizing the ES values for every pair in the Q-ensemble with regard to the dataset state-actions. The illustration of the ensemble gradient diversification shown in Figure 3b. Note that we instead maximize the total variance to reduce the computational burden. Nevertheless, the modified objective closely related to maximizing the smallest eigenvalue. The detailed explanation can be found in Appendix A.2.
We name the resulting actor-critic algorithm as Ensemble- Diversified EnsembleeDiverrifeed Actor Critic (EDAC) and present the detailed procedure in Algorithm (differences with the original SAC algorithm marked in blue). Note that Algorithm reduces to SAC-N when 1)=0, and further reduces vanilla SAC when also N 2
5
Experiments
We evaluate our proposed methods against the previous offline RI. algorithms on the standard D4RI. benchmark [9]. Concretely, we perform our evaluation MuJoCo Gym (Section 5.1) and Adroit

---

Algorithm Ensemble-Diversfiee Actor Critic (EDAC)
1: Initialize policy parameters 0, Q-function parameters { target Q-function parameters
[dr and offline data replay buffer D
2: repeat
3: Sample a mini-batch = {(s, a, r, s')} from D
4: Compute target Q-values (shared by all Q-functions):
v(r,s')=1r = (
min Qož (s', - 3 log vo (a' la()s)) (J_l...,
a' ~ (- (-1s')
5:
Update each Q-function Qp. with gradient descent using
1 Σ
2 N
7
Σ
ES Es ,s, (8)
B
(8.0,,,s))BB
ISIFJNN
6:
Update policy with gradient ascent using
Qs, (s, āo(s)) - 3 log To (āo(s) s)
min I...,N
B
SEB
where (s) is sample from Tt(- ( |s) which is differentiable w.r.t. @ via the reparametrization
trick. Update target networks with oi < pol + (1 -p)
7:
(Section 5.2) domains. We consider the following baselines: SAC, the backbone algorithm of our method, CQL, the previous state-of-the-art on the D4RL benchmark, REM [2], an offline RL method which utilized Q-network ensemble on discrete control environments, and BC, the behavior cloning method. We evaluate each method under the normalized average return metric where the average return is scaled such that 0 and 100 each equals the performance of random policy and an online expert policy. In addition the performance evaluation, we compare the computational cost of each method (Section 5.3). For the implementation details of our algorithm and the baselines, please refer to Appendix and C. Also, we provide more experiments such as comparison with more baselines, CQL with N Q-networks, and hyperparameter sensitivity from Appendix E to H. The code available online?.
5.1 Evaluation on D4RL MuJoCo Gym tasks
We first evaluate each method on D4RL MuJoCo Gym tasks which consist of three environments, halfcheetah, hopper, and walker2d, each with six datasets from different data-collecting policies. In detail, the considered policies online expert, medium: a suboptimal policy with approximately 1/3 the performance of the expert, medium-expert:: a mixture of medium and expert policies, medium-replay: the replay buffer of policy trained up to the performance of the medium agent, and fill-replay: the final replay buffer of the expert policy. Each dataset consists of 1M transitions except for medium-expert and medium-replay.
random: a uniform random policy, expert: fully trained
The experiment results Table show EDAC and SAC-N both outperform or are competitive with the previous state-of-ther thethh-rrt on all of the tasks considered. Notably, the performance gap is especially high for random, medium, and medium-replay datasets, where the performances of the previous works relatively low. Both the proposed methods achieve average normalized scores over 80, reducing the gap with the online expert by 40% compared to CQL. While the performance of EDAC marginally better than the performance of SAC-N, EDAC achieves this result with much smaller Q-ensemble size. As noted in Figure 5, on hopper tasks, SAC-N requires 200 to 500 Q-networks, while EDAC requires less than 50.
Figure compares the distance between the actions chosen by each method and the dataset actions. Concretely, measure ((s.a))D, [|| ||]] for EDAC, SAC-N, CQL, SAC-2, and random
Phttps://gitae cww.soo.ccuuuoo//eeee maii-mmlabbbe/eee /DAC

---

policy on s-medium datasets. We find that our proposed methods choose from more diverse range of actions compared CQL. This shows the advantage of the uncertainty-biut penalization which considers the prediction confidence other than penalizing all OOD actions.
Table I: Normalized average returns on D4RL Gym tasks, averaged over random seeds. CQL (Paper) denotes the results reported in the original paper.
CQL
CQL
SAC-N (Ours)
EDAC
Task Name
BC
SAC
REM
(Paper)
(Reproduced)
(Ours)
halfchoetab-randdm
2.2:0..0
29.7
-0,81111
35.4
31.3
28.0:10.9
28.4=10
hallcheetal--medum
43.2:40.6
55,2427.8
-0.841.3
44.4
46.940.4
67.5+11.2
68.9=0.6
halfcheetah-exxpa expeet
91.8+:1.5
48+188
4.1+5..7
104.8
97.3++1.1
105.2-12.6
106.8
halfcheetah-ment seweretaa--meiuu-eppppee
44.0:11.6
28,4419.44
0.7::3.7
62.4
95.0:114
107.1:1..0
106.3=1.9
halfchectah- meadod--edimmiiiee replay
37.64-2.1
08
6.6+111.0
46.2
45.3++.3
63.9-+0.8
61.3=119
halfchectab- full-replay
62.9:0.8
86.81:10
27.8:13544
76.9=0.9
84.5
84.6=0.9
hopper-random
3.740.66
9.111.5
3,442.2
5.340.6
31.3-10.0
25.3=100.4
hopper-medium
54.14:3.8
08+00
0.7:00.0
86.6
61.946.4
100.3::0.3
101.6 0.6
hepper-expert
107.7419.7
0.740.0
0,8:0000
109.9
106.549.1
110.3:10.3
110
hopper-mediume medium expert
53.9+t4.7
0.740.0
0.8+0.0
111.0
96.9+15.1
110.1++0.3
110.7=0.1
seepeer-medium-tee
16.6144.8
7.410.5
27.5
48.6
86.327.3
101.::0.5
101.00.55
hopper-full- reappll
19.9412.9
41.1=17.9
19.74:2..6
101.9#0.6
102.9:10.3
105.4-0.7
walker2d-random wa valker2d-medimm
1,3400.1
094008
6,948.3
7.0
5.411.7
21.7:10.0
16.6=7.0
70.9:11.00
0.33+0.2
0,2:00.7
74.5
79.5+
87.9:10.2
92.5=0.8
walker2d- expert
108.7:+0.2
0.740..
L.++233
121.6
109.3+0.1
107.4++2..4
115.1=1.9
walker2d-med expert
90.11:1..2
1.43399
-0.10000
109.1=0.2
116.7:1044
1...7=0.9
walker2d-mediune 2dmmedimm--pupayy
20.3:199.8
-0.4:0.33
12.546.2
32.6
10.8
78.7:10.77
87.1=2.3
w walker2d-full--pee
68.8:117.7
27.9147.3
-0,2400.3
94.221
94.6:10.5
99.8=0.7
Average
49.9
16.2
6.2
73.7
84.5
85.2
halfcheetah
hopper
walker2d
10
SAC-N EDAC
500
1000
3
100
20
50
20
10
medium
expert
medium
expert
medium
expert
Figure 5: Minimum number of Q-ensembles (N) required to achieve the performance reported in Table 1, M-E denotes medium-expert. We omit the results of medium-repla and full-replay as SAC-N already works well with small number of ensembles (less than or equal to 5). For more details of the experiment, please refer to Appendix C.
hallcheetah-medium
hopper-mediumg
wa walker2d-medium
0.5
SAC
0.5
0.5
COL
0.4
Deneity
Density
0.3
0.3
0.3
0.2
0.2
0.2
0.0
0.0
0.0
Distance
Distance
Distance
Figure 6: Histograms the distances between the actions from each methods (EDAC, SAC-N, CQL, SAC-2, and random policy) and the actions from the dataset. For details of the experiment, please refer to Appendix C.

---

5.2 Evaluation on D4RL Adroit tasks
We also experiment on the more complex D4RL Adroit tasks that require controlling 24-DoF robotic hand to perform tasks such as aligning pen, hammering nail, opening a door, or relocating a ball. We use two types of datasets for each environment: human, containing 25 trajectories of human demonstrations, and cloned, 50-50 mixture between the demonstration data and the behavioral cloned policy on the demonstrations. Note that for the Adroit tasks, we could not reproduce the CQL. results from the paper completely. For the detailed procedure of reproducing the results of CQL, please refer Appendix D.
Table 2: Normalized average returns on D4RL Adroit tasks, averaged over 4 random seeds.
CQL
CQL
SAC-N (Ours)
EDAC
Task Name
BC
SAC
REM
(Paper)
(Reproduced)
(Ours)
pen-human
25,818.8
4.3
5,4
55.8
35.2:1.66
9.51111
52. 92.6
hammer-human
3.1+3.2
0.2+0000
0.3+0000
2.1
0.6++0.5
0.3++0.0
0.8+00.4
door-human
2.8+0.7
-0.3+0.0
-0.3-0.0
9.1
1.2.188
-0.310.0
10.7:16.8
relocate-human
0.0+00.0
-0.3+0.0
-0.3+0.0
0.35
0.0++0.0
-0.1+
0.1+0.11
pen-cloned
38.3+11.9
-0.813.2
-1.0=0.1
40.3
27.2=11.3
6..1
68.2+t7.3
hammer-cloned
0.7:-0.3
0.140.11
-0.3-0.0
S.7
1.4:2..1
0.2:10.2
0.3.10.0
door-cloned
0.0+0..0
-0.32011
-0.30000
3.5
2.422.44
-0.3+1.00
9.638..3
relocate-cloned
0.1+0.00
-0.1+0.1
-0.2+0.2
-0.1
0.0:00.0
0.0+00.0
0.0+
The evaluation results summarized in Table 2. For pen-* tasks, where the considered algorithms achieve meaningful performance, EDAC outperforms or matches with the previous state-of-the-an Especially, for pen-cloned, both EDAC and SAC-N achieve 75% higher score compared CQL. Unlike the results from the Gym tasks, we find that SAC-N falls behind in some datasets, for example, pen-human, which could in part due to the size of the dataset being exceptionally small (5000 transitions). However, our method with ensemble diversification successfully overcomes this difficulty.
5.3 Computational cost comparison
We compared the computational cost of our methods with vanilla SAC and CQL hopper-medium--2, where methods require the largest number of Q-networks. For each method, measure the runtime per training epoch (1000 gradient steps) along with GPU memory consump- tion. We run our experiments on single machine with one RTX 3090 GPU and provide the results in Table 3.
Table 3: Computational costs of each method.
Runtime (s/epoch)
GPU Mem.
(GB)
SAC
21.4
1.3
COL
38.2
1.4
As the result shows, method EDAC runs faster than CQL with comparable memory consumption. Note that CQL is about twice slower than vanilla SAC due to the additional computations for Q-value regularization (e.g., dual update and approximate logsumexp via sampling). Meanwhile, the inference the Q-network ensemble in SAC-N and EDAC is embarrassingly parallelizable, minimizing the runtime increase with the number of Q-networks. Also, we emphasize that our gradient diversification term in Equation (4) has linear computational complexity, as we can reformulate the term using the sum of the gradients.
SAC-500
44.1
5.1
EDAC
30.8
1.8
Related Works
Model-free offliie RL popular approach for offline RL regularize the learned policy to be close to the behavior policy where the offline dataset was collected. BCQ [11] uses a generative model to produce actions with high similarity to the dataset and trains a restricted policy to choose the best action from the neighborhood of the generated actions. Another line of work, such as BEAR [15) or BRAC [28], stabilizes policy learning by penalizing the divergence from the dataset measured by KL divergence or MMD. While these policy-constrain methods demonstrate high performance

---

datasets from expert behavior policies, they fail to find optimal policies from datasets with suboptimal policies due to the strict policy constraints [9]. Also, these methods require an accurate estimation of the behavior policy, which might be difficult in complex settings with multiple behavior sources or high-dimensional environments. To address these issues, CQL. [16] directly regularizes Q-functions by introducing a term that minimizes the Q-values for -iit----iitiitttionn actions and maximizes the Q-values for in-distribution actions. Without such explicit regularizations, REM [2] proposes to use a random convex combination of Q-network ensembles on environments with discrete action spaces [4].
Estimation bias in Q-learning While Q-learning one of the most popular algorithms in reinforce- ment learning, it suffers from overestimation bias due to the maximum operation max E.A Q a') used during Q-function updates [10, 25]. This overestimation bias, together with the bootstrapping. can lead to catastrophic build-up of errors during the Q-lcarning process. To resolve this issue. TD3 [10] introduces a clipped version of Double Q-learning [25] that takes the minimum value of two critics. Subsequently, Maxmin Q-learning [17] theoretically shows that the overestimation bias can be controlled by the number of ensembles in the clipped Q-learning. The overestimation problem in Q-learning can be exacerbated in the offline setting since the extrapolation error cannot be corrected with further interactions with the environment, and existing offline RL algorithms handle the bias by introducing constrained policy optimization [11, 15] or conservative Q-learning frameworks [16].
Uncertainty measures in RL Uncertainty estimates have been widely used in RL for various purposes including exploration, Q-learning, and planning. Bootstrapped DQN [21] leverages an ensemble of Q-functions to quantify the uncertainty of the Q-value, and utilizes it for efficient exploration. Following this work, the UCB exploration algorithm [5] constructs an upper confidence bound [3] of the Q-values using the empirical mean and standard deviation of Q-ensembles, which is used to promote efficient exploration by applying the principle of optimism in the face of uncertainty [7]. Osband et al. [22] proposes a randomly initialized Q-ensemble that reflects the concept of prior functions in Bayesian inference and Abbas et al. [1] introduces an uncertainty incorporated planning with imperfect models. The notion of uncertainty has also been considered in offline RL, mostly in the framework of model-based offline RL. Especially, MOPO [29] and MOReL [13] measure the uncertainty of the model's prediction to formulate an ppenallitty-paalizeed policy optimization problem in the offline RI, setting. These methods introduce an ensemble of dynamics models for the quantification of the uncertainty, whereas our work adopts an ensemble of Q-functions for uncertainty-awan Q-learning.
7 Conclusion
We have shown that clipped Q-learning can be efficiently leveraged to construct uncertainty- based offline RL method that outperforms previous methods on various datasets. Based on this observation, we proposed Ensemble-Diversiff required number of ensemble networks for quantifying and penalizing the epistemic uncertainty. Our method does not require any explicit estimation of the data collecting policy or sampling from the dut-o--distiibution data and respects the epistemic uncertainty of each data point during penalization. EDAC, while requiring up 90% less number of ensemble networks compared to the vanilla Q-ensemble, exhibits state-of-the-ar performance on various datasets.
Actor-Critic (EDAC) that effectively reduces the
Acknowledgements
This work was supported in part by Samsung Advanced Institute of Technology, Samsung Electronics Co., Ltd., Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-008 0-08882, (SW STAR LAB) Development of deployable learning intelligence via self-sustainable and trustworthy machine learning and No. 2019- 0-01371, Development of brain-inspired AI with human-like intelligence), and Research Resettlement Fund for the new faculty of Seoul National University. This material is based upon work supported by the Air Force Office of Scientific Research under award number FA2386-225 20-0--2---2 1--1--100 14043. Hyun Oh Song is the corresponding author.
10

---

References
[1] Zaheer Abbas, Samuel Sokota, Erin Talvitie, and Martha White. Selective dyna-style planning under limited model capacity. In ICML, 2020.
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In 2020.
[3] Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exploiatioo tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science, 410(19): 1876-1902, 2009.
[4] Marc Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[5] Richard Y Chen, Szymon Sidor, Pieter Abbeel. and John Schulman. Ucb exploration via g-ensembles. arXiv preprint arXiv: 1706.01 2017.
[6] Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, and Le Song. Generative adversarial user model for reinforcement learning based recommendation system. In ICML, 2019.
[7] Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor-critic. In NeurIPS, 2019.
[8] William Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning, arXiv preprint arXiv:9905.09638, 2019.
[9] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXtv:2004.07219, 2020.
[10] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018.
[11] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019,
[12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.
[13] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In NeurIPS, 2020.
[14] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. NeurlPS, 2000.
[15] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NewrIPS, 2019.
[16] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurlPS, 2020.
[17] Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning In ICLR, 2020.
[18] Kimin Lee, Michael Laskin, Aravind Srinivas. and Picter Abbeel. Sunrise: simple unified framework for ensemble learning in deep reinforcement learning. In ICML, 2021.
[19] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016443 2020.
[20] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 518(7540):59-553, 2015.

---

[21] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In NeurIPS, 2016.
[22] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep rein- forcement learning. In NeurIPS, 2018.
[23] JP Royston. Expected normal order statistics(exact and approximate). Applied Statistics, 31(2): 161-5, 1982.
[24] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. arXiv preprint arXiv:1206.294 2944,, 2012.
[25] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learming. In AAAI, 2016.
[26] Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wo- jeiech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2, 2019.
[27] David S Watkins. Understanding the qr algorithm. SIAM review, 24(4):427-44)) 1982.
[28] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:1911.11380 2019.
[29] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In NeurIPS, 2020.
12


Reviews:
[{"review_id": "tYCAnXWQWK3", "reviewer": "Reviewer_PrWt", "summary": "In this work the authors propose a new algorithm for the problem of offline RL. One of the main\ndifficulties in offline RL is estimating the Q-values of Out-Of-Distribution action state pairs. \nTo overcome this, they authors propose a new actor critic method, where an ensemble of Q values for the critic\nis trained  using clipped Q-learning. They show that this can be used to penalize OOD data \nwith high prediction uncertainties. The resulting algorithm achieves state-of-the-art performance in \na range of D4RL benchmarks. Furthermore, in order to reduce the size of the ensemble required to\nachieve good performance they propose a modification on the critic loss. They show that the resulting\nalgorithm called Ensemble-Diversified Actor-Critic can match the performance of the naive approach while\nusing only one tenth of the number of ensemble Q values. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a simple but novel approach to the problem of offline RL, while reporting suprisingly\ngood results. The use of ensembles in offline RL is a interesting approach and opens up new research avenues.\n\nSome comments on the paper:\n\nIn figure 1, increasing the number of ensembles hurts performance in halfcheetah-medium but helps in hopper-medium.\nThe authors shoudld provide an explanation for this result.\n\nline 84: Mentions the sampling of the action distribution as computationally expensive which can be a limiting factor.\nHowever, maintaining multiple Q values and calling inference on them can incurr an even higher cost, so this cannot be\nused as an argument in favor of their method.\n\nline: 163: Define Proposition 1 before using it.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "t4X1GapmzWX", "reviewer": "Reviewer_jwqx", "summary": "This paper considers ensemble actor-critic structure for offline reinforcement learning.\nIn particular, they focus on SAC-$N$, a version of SAC where the $Q$-function is updated pessimistically w.r.t. a $Q$-function ensemble. \nThe authors notice that a large value of $N$, SAC-$N$ seems to improve its performance. \n\nThis effect is imputed to the fact that the pessimistic estimate is due to the uncertainty expressed by the ensemble (e.g., the higher variance in the ensemble, the more pessimistic is the estimate). Having a large ensemble can result in a higher variance of the estimate, and therefore on a stronger penalization of out of distribution (OOD) state-action pairs.\n\nThe authors, therefore, argue that favoring diversity in the ensemble should reach the same effect, without the need of having a large ensemble. To this end, they introduce a regularizer that induces a higher variance of the action-gradients.\n\nThe algorithm is validated on different tasks.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The literature is rich in papers proposing methods to select \"in-distribution\" state-action pairs while penalizing ODD state-actions.\nThe novelties of this paper, as far as I can see, rely on\n\n1) a pessimistic $Q$-estimate using a $Q$-ensemble;\n2) the introduction of a new regularizer to induce diversity in the $Q$-ensamble.\n\nWhile the pessimistic approach in the face of uncertainty still makes sense (but yet explored in the community [1, 2]), there is not a clear logic flow that drives the selection of the regularizer chosen by the authors. One could have selected to favor the variance between the $Q$-estimate for example. The explanation given by the authors is, in my opinion, quite fuzzy. \n\nIn short, the contribution of the paper is limited. And the logical argumentation of the algorithm is also weak.\n\nThe empirical section is somehow better but still lacks comparison with state-of-the-art algorithms. The algorithm is validated on different, state-of-the-art, benchmarking tasks. The proposed algorithm seems to achieve better results than behavioral cloning, SAC (which is not originally thought for offline policy optimization), REM, CQL.\nFor such empirical work, I would recommend comparing also with other related algorithms, like BRAC, BCQ, but also MOPO and MOReL. \n\n\n__Note:__, in __Algorithm 1__ the update rule of the policy (6:) is incorrect. Please, fix it. \n\n\n[1] Lee, Seunghyun, et al. \"Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble.\" arXiv preprint arXiv:2107.00591 (2021).\n[2] Jin, Ying, Zhuoran Yang, and Zhaoran Wang. \"Is Pessimism Provably Efficient for Offline RL?.\" International Conference on Machine Learning. PMLR, 2021.\n\nUPDATE\n--------------------------\n\nI appreciate the effort of the authors in including new baselines and in explaining well their perspective. \nI still think that the novelty introduced is limited. However, I must also admit that with the new results, the empirical section will look stronger and I give credit to the author by raising my score accordingly.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations of their proposed approach. \nThey did not discuss the societal impact, as that does not apply to this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "pIaEBkttDJE", "reviewer": "Reviewer_DiJ9", "summary": "The paper presents a simple but rather effective idea (probably known by many but never formally tried): having an ensemble of N Q-networks and then using their *minimum* in the Bellman backup. This is roughly analogous to penalizing the objective function of fitted-Q iteration by the variance of the ensemble, which causes a somehow lower-confidence bound on the learning, which in turn causes conservative values for out-of-distribution actions at each seen state. This framework is then used as part of a soft actor-critic algorithm, called SAC-N. This significantly outperforms SOTA (but much slower in convergence rate). However, this setup can require an extremely large ensemble, which can be a limitation in practice. The authors next observe that the performance of SAC-N is highly correlated with the diversity of the Q-functions’ gradient w.r.t. actions (increased by the size of ensemble, N). To maximize diversity, they proposed to minimize the total cosine similarity matric, which gives rise to maximizing the variance for smallest eigen value of the variance-gradient matrix. As it gets into a computationally intractable problem, they instead maximize the sum of all eigenvalues, which is equal to the total variance. This final matric is called Ensemble Similarity and the resulting algorithm is named Ensemble-Diversified Actor Critic (EDAC). Finally, several experimental results are presented, largely outperforming previous methods. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents several interesting ideas and nice results. I quite enjoyed reading the manuscript. Here are some comments to help improving the paper:\n\n-- An observation from Figure 2 is that the *deviation from minimum* increases for OOD actions as training advances. This implies that the average value is diverging for those actions; i.e., the overestimation of their values grow with more training. It would be interesting to perhaps suppress this as well, for example by a weak regularization proportional to the size of Q itself. In a related note, this problem could also be due to value overflow. The following paper has a discussion on value overflow in deep RL (which is different from value overestimation):\n\n> Fatemi, M., Sharma, S., van Seijen, H. & Kahu, S. E. (2019), Dead-ends and Secure Exploration in Reinforcement Learning, In International Conference on Machine Learning (pp. 1873-1881).\n\n-- The narrative of section 3 (specially the last part) is rather descriptive than formal. This is fine as long as it gets noted that most of the results are indeed conjectures based upon the experimental results. While I personally enjoyed reading this section too, the claims should be tailored in a correct way. For example, L134: *… we confirm that …* -> is too strong; nothing has been confirmed, but just a nice trend has been observed (which seems to be correct, yet not formally proved). \n\n-- Figure 3: there is a sharp break point for average return at N=100. Is there any explanation for this dramatic change of slope? In particular, both before and after this point, the trend looks surprisingly linear. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is mostly technical, so no direct societal impact is expected at this stage. However, when a method like the one proposed in this submission is deployed in practice, it can cause significant impact in high-risk domains like healthcare (is specially so because offline RL largely targets healthcare). Perhaps it could be helpful to have a discussion on how to use the variance as a measure of uncertainty to warn against situations where the learned value/policy is not trustworthy. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "CHs_gHImdc", "reviewer": "Reviewer_MGH5", "summary": "This paper proposes two novel approaches to address the overestimation of OOD action value in offline RL, increasing the number of critics in the online RL algorithm (SAC-N), and forcing each ensemble to estimate diverse values with gradient similarity minimization term (EDAC). Both of them outperform the baseline methods (BC, SAC, REM, and CQL) in D4RL benchmarks (MuJoCo and Adroit tasks).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Summary\nThe authors propose a simple solution to handle the uncertainty in offline RL, increasing the number of Q ensembles and diversifying each estimate. \nHowever, these enormous amounts of Q-functions and computation of gradient similarity seem to require a huge amount of computational costs, which might be an intractable solution. Also, some ablations might be missed. Considering these aspects, I lean towards rejection.\n\n\n## Pros\n- This paper is well written and easy to follow.\n- The idea of diversifying Q ensemble estimates seems a simple yet effective approach.\n- The experimental results seem good in D4RL MuJoCo and Adroit tasks over popular benchmark methods (e.g. CQL).\n\n\n## Cons\n- The proposed methods require a large number of Q-networks (up to 500 in SAC-N, 50 in EDAC), which seems sometimes an intractable approach. Also, I guess the gradient similarity term (Algorithm 1, L5) requires a lot of computational costs.\n- In checklist 3 (d), the authors say the amount of computation is described in Appendix B, but I can't found it. Could you describe the computational burden (or point out the line number where you mentioned it)? \n- Popular offline methods like BCQ, BEAR, BRAC, and CQL, also employ a clipped double Q-learning. The ablation of (Offline RL)-N, or offline RL w/ diversification might be missed, which might lead to better performance than SAC-N or EDAC. \n- Figure 6 shows the average action distance to the datasets. I'm curious about the histogram of standard SAC (SAC-2). I guess that it might look the same as SAC-N or EDAC ones. If so, the performance gain of EDAC or EDAC might come from a different perspective.\n- Eq. 5 shows an example of that variance of critics are bounded under some conditions and degrade the ability to penalize OOD actions, which motivate the diversification of the gradient of Q function. I think that Eq. 5 also holds in EDAC. Could the authors provide a tighter bound or intuitive interpretation?\n- As for the related work, the authors can include EMaQ [Ghasemipour et al. 2021], which suggests that Q ensembles is an effective approach in offline RL, and UWAC [Wu et al. 2021], which proposes uncertainty-based regularization in offline RL. It may better to address the clarification of the connection to such works. If the code is available, the authors could take it into consideration to compare them empiricaly.\n-  In appendix B, Table 1 shows that EDAC sometimes chooses $\\eta$=0, which seems to reduce to SAC-N. I wonder this implies that the gradient correction in EDAC might be unstable and highly dependent on the parameter choices.\n- (Minor) L77-79 argues that the methods that require an estimated behavior policy such as BCQ, BEAR, and BRAC just work in \"simple environments\". However, the benchmark environments used in those papers are the same as this paper, MuJoCo locomotion tasks, and they show decent success. I'm not sure this expression is a proper one.\n- (Minor) L82-84 argues that CQL loss requires an appropriate action distribution and sampling, and it causes computational inefficiency. However, I wonder the proposed methods (SAC-N, and EDAC) also requires a lot of sampling procedure during training (Algorithm 1 L4) since the number of Q networks is large. I think it also causes computational inefficiency.\n- (Minor) Notation of $\\mu$ seems to be used in a different context (in L84, and Eq. 3).\n\n### Reference\nGhasemipour, S. K. S., Schuurmans, D., & Gu, S. S. (2021). Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning.\n\nWu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., & Goh, H. (2021). Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning. In International Conference on Machine Learning.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "## Limitation\nThe proposed methods require a large number of Q-networks (up to 500 in SAC-N, even 50 in EDAC), which seems sometimes an infeasible approach since previous methods typically use only two.  I think such computational inefficiency is an obvious limitation and should be described (although there is a trade-off between the performance and costs).\n\n## Societal Impact\nN/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors note that using an ensemble with N-networks in SAC and using the minimum for the Bellman backup performs well on Offline RL. This is a simple approach and interesting observation. As using a large N is computationally expensive, they develop an approach to achieve similar results with many fewer networks.

After the author response, all reviewers and I found the empirical section to clearly demonstrate the strength of their approach. Although there are some concerns on novelty, I think the strong empirical results and simplicity of their approach makes this paper a worthwhile contribution. The clarifications made during the response phase were important, so I encourage the authors to revise the paper with these in mind.
Decision: Accept (Poster)