SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning
Authors: Dohyeok Lee
Seungyub Han
Taehyun Cho
Jungwoo Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Reinforcement Learning
Ensemble Q-learning
TL;DR: We propose tractable Q-ensemble independence regularization for ensemble Q-learning based on the random matrix theory.

Abstract: Alleviating overestimation bias is a critical challenge for deep reinforcement learning to achieve successful performance on more complex tasks or offline datasets containing out-of-distribution data. 
In order to overcome overestimation bias, ensemble methods for Q-learning have been investigated to exploit the diversity of multiple Q-functions. 
Since network initialization has been the predominant approach to promote diversity in Q-functions, heuristically designed diversity injection methods have been studied in the literature. 
However, previous studies have not attempted to approach guaranteed independence over an ensemble from a theoretical perspective. 
By introducing a novel regularization loss for Q-ensemble independence based on random matrix theory, we propose spiked Wishart Q-ensemble independence regularization (SPQR) for reinforcement learning. 
Specifically, we modify the intractable hypothesis testing criterion for the Q-ensemble independence into a tractable KL divergence between the spectral distribution of the Q-ensemble and the target Wigner's semicircle distribution. 
We implement SPQR in several online and offline ensemble Q-learning algorithms. 
In the experiments, SPQR outperforms the baseline algorithms in both online and offline RL benchmarks.

SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning
Dohyeok Lee', Seungyub Han!.2 Tachyun Choo, Jungwoo Lee!
Seoul National University, "Hodoo AI Labs
ttaaluaaaanntaaauann valium,junglee, esnuu ac ac.kr kr
Abstract
Alleviating overestimation bias is critical challenge for deep reinforcement learn- ing to achieve successful performance on more complex tasks or offline datasets containing out-o--distributiin data. In order to overcome overestimation bias, ensemble methods for Q-learning have been investigated to exploit the diversity of multiple Q-functions. Since network initialization has been the predominant approach to promote diversity in Q-functions, heuristically designed diversity injection methods have been studied in the literature. However, previous stud- ies have not attempted to approach guaranteed independence over an ensemble from theoretical perspective. By introducing a novel regularization loss for Q-ensemble independence based on random matrix theory, we propose spiked Wishart Q-ensemble independence regularization (SPQR) for reinforcement learn- ing. Specifically, we modify the intractable hypothesis testing criterion for the Q-ensemble independence into a tractable KL divergence between the spectral distribution of the Q-ensemble and the target Wigner's semicircle distribution. We implement SPQR several online and offline ensemble Q-learning algorithms. In the experiments, SPQR outperforms the baseline algorithms in both online and offline RL benchmarks.
Introduction
Reinforcement learning (RL), especially deep reinforcement learning (DRL) that contains high capacity function approximators such as deep neural networks, has shown considerable success in complex sequential decision making problems including robotics [. video games [28], and strategy games [12]. Despite these promising results, modern DRL algorithms consistently suffer from caamlle-infffccenne and overestimation bias.
Vanilla off-po Q-learning faces the overestimation issue both online and offline reinforcement learning. The overestimation problem is specifically worse for out-of-distribution (OOD) state-action pairs. In order to address the overestimation problem, many existing methods utilize ensemble Q-learning. Ensemble Q-learning introduces multiple Q-networks to compute Bellman target value
to benefit from the diversity of their vavuue.. Double Q-learning [15] proposes clipped leaaning as the first method to adopt an ensemble DRL. Along with double Q-learning, Ensemble DQN
3, Maxmin Q-learning [22), randomized ensembled double Q-learning (REDQ) [6) and SUNRISE
24 are proposed as efficient ensemble Q-learning algorithms on online RL tasks. Clipped SAC
2 is proposed for offline RI. tasks to alleviate the overestimation bias of Q-values from OOD data.
Despite the objective of ensemble Q-learning, which enables RI agents access diverse Q-values for each state-action, existing ensemble Qelarniigg algorithms have only focused on the computing Bellman target over an ensemble. MED-RL [3I] and EDAC [2] propose a diversifying loss for online and offline reinforcement learning, such as the Gini coefficient and the variance of the Q-value on
CCorespooning author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

OOD data. However, these methods have a limitation in that the proposed losses do not guarantee independence from theoretical perspective, but diversify Q-functions from heuristics and empirical studies although they have an i.i.d assumption. Most ensemble Q-learning algorithms assume that bias follows uniform and independent distribution, which may not be true when using deep neural network approximation and a shared Bellman target. Hence we need to reinvestigate the notion of Q-ensemble independence more rigorously through a random matrix theory. The random matrix theory studies the statistical behavior of random matrix and has wide range of applications in diverse fields such as physics, engineering, economics, and machine learning as shown in [5] 25 29]
In this paper, we propose spiked Wishart Q-ensebble independence regularization (SPQR), a tractable Qensemble independence regularization method based on the random matrix theory and spiked random model perspective. SPQR is regularization method that theoretically guarantees to improve the independence of a Q-ensemble during training. SPQR penalizes the KI. divergence between the eigenvalue distribution of the Q-ensemble and an ideal independent ensemble. If a Q-ensemble becomes closer to the ideal independent ensemble, satisfies the independence assumption of ensemble Q-learning. To show how effective SPQR is, we combine SPQR with various algorithms and online/offline RL tasks, SPQR outperforms baseline algorithms in MuJoCo, D4RL Gym, Franka Kitchen, and Antmaze. To summarize, our contributions are as follows:
1. We propose a tractable loss function for Q-ensemble independence without any prior as- sumption of Q-function distribution by adopting random matrix theory and spiked random model for the first time in DRL algorithms.
2. It is empirically shown that the universality and performance gain of SPQR for various algorithms and tasks. SPQR is applied to SAC-Ens, REDQ, SAC-Min, CQL, and EDAC and evaluated in various complex and sparse tasks online and offline RI., MuloCo, D4RI. Gym, Franka Kitchen, and Antmaze. To ensure our reproducibility and fair comparisons, we prodive our source code
2 Related Works
Ensemble Reinforcement Learning. Ensemble DQN [3 is the first method to adopt ensemble Q-learning for DRL. Ensemble DQN aims to reduce overestimation bias by computing a Bellman target as an average Q-value in the ensemble. Maxmin Q-learning [22] computes a Bellman target as a minimum Q-value over the Qeensemble and shows significant performance enhancement based on the Atari benchmark. REDQ 6 constructs sample efficient ensemble Q-learning algorithm based on Maxmin Q-learning and SAC with a probabilistic ensemble by selecting subset over Qeensebbee. Since REDQ combines the update-to-data (UTD) ratio, which is orthogonal method with ensemble subset minimization, REDQ achieves state-of-the-art performance the online MuJoCo environment. For offline reinforcement learning tasks, clipped Q-learning (2), denoted as SAC-Min is proposed, which computes the Bellman target as in Maxmin Q-learning. Diversification methods are proposed to improve the performance of ensemble Q Q-learning. MED-RL [31] optimizes the Gini coefficient to diversify an ensemble, and EDAC 2 optimizes the variance of the Q-value on OOD data to diversify an ensemble from the OOD data. SPQR proposes Q-ensemble independence loss from theoretical perspective, based on a spiked random model and random matrix theory to make ensemble Q- networks as independent as possible. In addition, conservative Q-learning (CQL) is proposed by 21,, which penalizes the Q- Q-values of OOD actions and performs well on numerous offline datasets.
Spiked Random Model. Signal recovery for a generative model using spiked random model is analyzed by [8] SPQR analyzes Qeensebble independence for ensemble Q-learning in a spiked random model perspective. A universal data-driven optimal likelihood hypothesis test with linear spectral statistics (LSS) for a spiked random model as a central limit theorem for test statistics is given by 7 18). SPQR proposes tractable universal data-driven test T(X) to regularize the Q-ensemble independence.
'Code implementation is available

---

Figure 1: Left 1-4: Histogram visualization of Q-value distribution over networks in the ensemble. Each plot represents Q-value histogram for one state-action data. The X-axis represents the Q-value of each Q-network and the Y-axis represents the number of Q-networks in the histogram bin. The horizontal line represents uniform distribution and the blue solid line represents kernel density estimation for given histogram. Rightmost: Heatmap visualization of the Pearson correlation coefficient matrix between each Q-network in the ensemble. Detailed values and explanations are given in Appendix B
3 Backgrounds
In this section, describe ensemble Q-learing algorithms and visualize empirical study of their limitations. In order to overcome its limitations, we introduce the random matrix theory and spiked random model to formulate the notion of Q-ensemble independence from theoretical perspective.
We consider Markov decision process (MDP), defined by tuple (S,A, P. r,, 7), where S is finite state space, A is finite action space, P: S x A x S + [0, 1] is the state-action transition probability, r: S x A + the reward function, and y E [0, 1) is discount factor. We define a state-action value function, called Q-function, Qo: S x A + R, represents an expectation of the discounted sum of rewards for the trajectory started from 8, a, given by E. F,,, 7 ((8,,,))). 4z)]-
3.1 Ensemble Q-learning
- We first define bias ei_ - a). where (' a) is the optimal Q-value given for (s, a)
(ssss)
and Qi(s, a) is the Q-value of the i-th networks in ensemble. Ensemble Q-learning assumes that the bias follows a zero mean, uniform, and independent distribution, i.e., e fni and although REDQ does not assume uniform distribution. Given N O-functions the ensemble, there are three typical methods to compute a Bellman target: Ensemble DQN, Maxmin Q-learning, and REDQ.
is independent with respect
Ensemble DQN. Ensemble DQN computes the Bellman target as ensemble average Q-value. The bias and variance of the bias monotonically decrease as the ensemble size N increases, and they become as N goes to infinity. We implement the SAC-style of Ensemble DQN, denoted as SAC-Ens.
Maxmin Q-learning. The Bellman target of Maxmin is to select the minimum Q-value within an ensemble, which can control the level of overestimation with finite number of Q-functions. A modified version called SAC-Min is introduced specifically for offline RL tasks.
REDQ. REDQ computes the Bellman target as the minimum Q-value in subset of the ensemble to become a sample-efficient algorithm.
Detailed explanations for Bellman targets and the evaluation function of each algorithm, denoted as Enstar and Enseval. are described in Appendix C
To confirm the validity of uniform and independent assumptions of ensemble Q-learning, we plot Q-value distribution in Figure I and test whether each Q-distribution is uniform using the x" test. We train 50 Q-networks 3M timesteps by the SAC-Min algorithm in the D4RL hopper-random dataset and evaluate it by the hopper-full-replat full-replay dataset. We also perform the x test for independence in Table I Out of 1000 batch data, only 35.8% can accept that the distribution uniform and 30.4% can accept independence. Thus, e ~ Unif((7) and i.i.d. assumption is empirically inaccurate and it necessary to introduce the notion of Q-ensemble independence. More details on the x² test given in Appendix E
3

---

4=0 0.35
y=1e-05
0.2- y=1e-05
04-
33
03-
0.90
02-
-12
-0.
47
-22
11
45 2gerr/akk
aigenvalue
Figure 2: Eigenvalue plot for the spiked Wishart model with Wigner's semicircle law. For effective visualization the spiked model, we use a complex hermitian random matrix, GUE. Blue dots represent each data in complex plane. The red line represents Wigner's semicircle distribution. The blue line in the histogram represents kernel density estimation. Left 1-2: Perturbation power is 0. Higenvalue distribution follows Wigner's semicircle law. Right 3-4: Perturbation power d = 10-5 Figenvalue distribution almost follows Wigner's semicirele law except for the largest eigenvalue, interpreted spike.
As (30 also points out, the early collapse of Q-values, which means that the Q-values in the ensemble become almost identical, occurs during the training. We empirically visualize strong correlation among Q-networks in Figure [I] Although the main advantage of ensemble Q-learning is based on diverse Q-functions, network initialization is the only process maintain diversity over an ensemble since the Bellman target is identical across all networks without diversification and independence regularization methods.
3.2 Random Matrix Theory
Random matrix theory studies the probabilistic properties large matrices with random entries. Dyson [10] constructs some categories of the random matrix real symmetric matrix Gaussian orthogonal ensemble (GOE), complex hermitian matrix as Gaussian unitary ensemble (GUE), and quaternion self-dual matrix Gaussian symplectic ensemble (GSE), called Dyson's 's classification. In this paper, we restrict our interest to matrices GOE.
Definition 3.1 (Gaussian Orthogonal Ensemble, (26). Let random matrix W NNN its entities random variables Wi, P[W] = P(w1.... wya) to be a joint probability density function of w,, Define the set of W as GOE when these properties hold:
(I) w; are independent with respect to i.
(2) W is a real symmetric matrix.
(3) W is rotational invariance, i.e., P[W]dW- P[W"]aw' nal.
PWWjdW=PW"jdW'' W' UWU-1, is orthogo-
GOE is the most natural way to model random matrices in various applications, including machine learning. In simplified terms, we can consider GOE as Wigner matrix, where entries are independent, symmetric, and follow Gaussian distribution. One of the main areas in random matrix theory is how eigenvalues or singular values of large random matrix are distributed. More specifically, we only consider limiting properties of empirical spectral density (ESD), which defined as, random matrix W E TR NXNV and its eigenvalues Xi, -,...,N N.
1 Σ 8(A 8()
PW (A)
The limits the empirical spectral density the GOE, called the Wigner's 's semicircle law, are given by [34]. For GOE W E NxN with variance of entries o², Aiy..., An to be eigenvalue of
N
W vn W and empirical spectral distribution pw = Σ 8(2 Ag), then pw converges to Psc ==1
weakly almost surely, where
(Ac = 2TO2 V 402 - 221 27,20l
(2)

---

We call Pse as the Wigner's s semicircle distribution. Wigner's semicirele law does not rely on the probability density of entries.
3.3 Spiked Random Model
Random matrix theory models random data well, but in reality, most data have both informative signals and random components. A spiked random model is theory about constructing model of perturbation with a data matrix. The objective of this theory is to separate true signals from perturbated ones with random noise. True signals show spiked phenomena that make the largest eigenvalue detach from the bulk of eigenvalues. A spiked model was first introduced by [17] as an application for principal component analysis. We restrict our analysis to a spiked Wishart model, the most natural methodology to model a general perturbation. A spiked Wishart model is defined as follows.
Definition 3.2. Let Y E R.N Pps U, i is unknown vector, Y spiked Wishart model when
(1) = V &uv² W, which equivalent to sample covariance matrix & = IN + a wnu
(2) u are ii.d from zero mean priors Pu, v ~ Po. support bounded in radius by Ku, Kg
W Po is a matrix with i.i.d noise entries
(4) M/N + w
M, where we can ignore property 4. In Figure 2 we visualize how the spiked Wishart model works in both the data and spectral domain. For detailed explanation, see Appendix
We will assume the spiked Wishart model with N =
Qensemble Independence Regularization
In this section, we present a method Q- Qenssemble independence regularization that is computation- ally feasible through the utilization of spiked Wishart model. We bring the concept of ensemble Q-learning to the spiked Wishart model as Y P. representing learned Q-value by Q-learnin algo- rithm and W ~ Io representing the perfectly independent Q-ensemble. If W dominates Y, Y follows independent assumptions of ensemble Q-learning. If not, a collapse Q-value and high correlation occurs, which cannot be benefited by the ensemble method. By integrating with ensemble Q-learning and Q-ensemble independence testing, we propose a tractable Q-ensemble independence loss with gain 8, which is further described in the following section. To help understand the independence and diversity of Q-ensemble more clearly, we mentioned that 3 can be simply considered as loss gain (weight) for Q-ensemble independence regularization before providing a detailed explanation about SPQR and B. A higher 3 represents highly- likelllt---tt-iiiiee to--t-ddddeeeeent..
4.1 Diversity and Independence of Q-ensemble
To clarify our work, distinguish between the notions of diversity and independence. Numerous ensemble Q- learning methods improve their performance by minimiz- ing the overestimation bias and the variance of the Bell- man target estimate. However, such works assume that the biases e are zero-mean independent and identically dis- tributed over although early collapse, high correlation between Q-ensemble, occurs due to the shared Bellman target. On the other hand, diversification methods such as EDAC develop an ensemble diversification objective reduce the number of ensemble networks required. In particular, EDAC increases the variance of the ensemble to penalize OOD actions. In our work, we focus on the independence of a Q-ensemble to reduce the effect of cor- related behavior resulting from a shared Bellman target. Roughly speaking, diversification can be interpreted as flattening the distribution of a Q-ensemble, while inde- pendence regularization can be interpreted as promoting
2-1.
Figure 3: Mean of predicted Q-value of SPQR-SAC-Min with various , on hopper-random dataset, averaged over 4 seeds.
5

---

an i..dd sampling procedure for given distribution. We verify that our proposed regularization successfully exploits both the conservative behavior and the diversification of a Qensemble. The Q-ensemble independence regularization empirically shows large performance gain by controlling the conservatism, as shown Figure 3 similar to the performance improvement of the ensemble method caused by the conservatism. For further details, see Appendix E
4.2 Tractable Loss for Q-ensemble Independence
We construct a tractable testing criterion to determine whether a distribution is independent in Theorem 4.1] (Pp(P(()) Psc))) does not require any prior assumption about Po and P& since in spectral domain, Wigner's semicircle law is universal for all random matrices. Proof of Theorem 4.1 is shown in Appendix B
Theorem 4.1. Following definition 3.2 as N - oo with probability least 8, following test T(A) is optimal.
Hg : Po 26)=
T(A) = 1 (D)) (c)
H1 Y ~ P
We now describe practical Q-ensemble independence loss for ensemble Q-learning using the spiked Wishart model and Theorem 4.1 Let Y and for the spiked Wishart model denote Q-value < & or is minimized, is
matrix and an unknown signal. By the test T()), if D KL (A). theoretically guaranteed that Y is a random matrix, i.c. a Q-ensemble is independently distributed.
In conclusion, propose DKL(Pu Ps)())) for ensemble Q-learning, without any prior assumption such as uniform distribution.
Q-ensemble independence regularization loss
4.3 Algorithm
As described in Section 3.1 ensemble Q-learning has crucial but inaccurate assumptions about independence. To address the issue insufficient independence, we propose SPQR: spiked Wishart Q-ensemble independence regularization for reinforcement learning, which we describe in Algo- rithm [ Using the random matrix theory and a spiked random model lens, SPQR minimizes D (A),
form of regularization loss, denoted as SPQR loss, the baseline ensem- ble Q-learning algorithms. DKL (pw (A)( penalizes for going too far from an independent Oensemble, which means promoting Q-ensemble to follow Po- We visualize our concept in Figure To compute the SPQR loss using (A), we need to construct symmetric Q-matrix Y filled with Q-values. Then, normalize the Q-matrix with a zero-mean and unit variance compute the Wigner's semicircle distribution Psc(A) with a = 1. Finally, we compute eigenvalues with numerically stable algorithm by constructing the ESD (A) and calculate the KL divergence between Pv (A) and (A). The loss function LSPQR is computed as expectation of D (P& (A), (ec( over the batch data. detailed explanation of implementations and algorithms is shown in Appendix
Algorithm 1 SPQR
Algorithm 2 SPQR Loss
1: Input: 7, a, N, B, r.
1: Input: Q-network 4 =-(Qa (5', (s',a')
2: Initialize policy 6, Q-function Φi, target Q-function 6/ replay buffer D
2: Y E DDD, D=
|V++8N 11VI+SN
3: repeat
3: Ym + (Dpto: if
Ygp: otherwise
4: v(r,5) - (0)) log so(adlss)) mogg
4: Y + (Y -uY)/oy
5: for 1,..., ,.. do
5: {A: IN + Eigen 1 Y)
6: LSPOR
VN
6: Pead (A) = 1 E, 8(1 X1)
7: Ve. [B] (s)a(()a)) w(r, BLsPaR
1
N
Pwigner (X) -I-X
8:
end for
8: Esron + [2 E, E.Pesa(>)) log Prene(a)
25
9: Ve [
Σ EveR Ensimu((os) a log To(ā(s)
Purigarr (Ax)
9: Output: SPQR loss LSPQR
10:
until

---

4. R1.02..... Qwl
shuffle
fill symmetricc matrix with
Q
Q0
Consenale
Q4
Q5
Qn
[Order of arbitrary
Irguta
vI 8.V
D
SPOR
Beseline
Figure 4: Illustrative example of SPQR.
Figure 5: Illustration for building Q-matrix.
Experimental Results
To reveal the validity and universality of SPQR from algorithms to tasks, we implement SPQR in various ensemble Q-learning algorithms and various online/offline RL tasks. We implement SPQR in SAC-Ens, REDQ. SAC-Min, and CQL, and evaluate them in online MuJoCo Gym tasks, offline D4RL Gym. Franka Kitchen, and Antmaze tasks. We denote the SPOR-implemented version of each ensemble algorithm as SPQR-* such as SPQR-SAC-EIG the same hyperparameter and experimental protocol with a baseline, For fair evaluation, the SPQR code is written on top of each official baseline code, RED EDACH and CQL Implementation of SPQR can be done by only 50 lines and training time only increases by 5%. We empirically show that SPQR outperforms baseline ensemble Q-learning algorithms in both online and offline RL tasks. See the implementation details in Appendix D
All the following implementations use
Is SPQR really affecting independence? We perform independence hypothesis testing and vi- sualize the spectral distribution to investigate whether SPQR really affects the independence of the Q-ensemble. First, we analyze the performance gain of SPQR from spike reduction perspective. We train Q-ensemble using SAC-Min and SPOR-SAC-Min on the hopper-random dataset and test it on the hopper-full-replay dataset. Figure 6 and Table [ show the evaluation results for the spike-reducing phenomenon of SPQR, which deviates less from Wigner's semicirele law, Compared to SPQR- SAC--in with 3 0.1. SAC-Min has significantly more spikes, in particular, 2434 spikes compared 2182 spikes over 25600 data. For extreme regularization gain, = 100, the spike is drastically reduced to 698, 71% % less than the baseline algorithm. We also plot spike histogram for another diversification method, EDAC. EDAC increases spikes to 54 % more than SAC-Min, which can be interpreted SPQR being the orthogonal method with EDAC.
We also demonstrate the X2 test with null hypothesis Ho P(Q,, Q,) P(Q)) and significant level a = 0.025 for independence hypothesis testing on the same experimental setting above. Since SPQR promotes being "more" independent, we use the proportion of data that can accept an independent hypothesis
measure of more or less independence. In Table [ we report the proportion of trials over 1000 tests that accept the null hypothesis Ho, meaning that the Q-ensemble independent. These empirical studies can be interpreted as SPQR having Q-ensemble independence regularization power since the acceptance rate increases monotonically as increases. Orthogonality between SPQR and EDAC is also found from independence testing perspective. However, reducing spikes and being more independent does not guarantee optimal performance since there is an optimal conservatism for each task. We can find the proper conservatism by controlling the independence regularization power, 33. More details can be found in Appendix
httpaa//githab..ccmmmmuaaaaaaa/p

---

Table 1: Number of spikes, independence hypothesis testing acceptance ratio, and per- formance of SAC-Min, EDAC, and SPQR- 3 = 1.0, 10.0. 100.0. We repro- duce EDAC-1 1.0. The reported value of EDAC-1 = 1.0 is given in the Section H of 2
2014
Piw.sio-noomm--1 i 59
Algorithm
Spike
Performance
Accept ratio
SAC-Min
2434
31.3 + 0.0
30.4%
EDAC-1 1.0
3751
10.3
0.0%
Figure Histogram of spikes for Wigner's semicir-
SPQR-8 0.01
2360
33.6
59.0%
SPQR-2 - 0.1
2182
35.6 1.4
80.4%
cle (black line), SAC-Min (blue), EDAC (orange), and SPQR-2 1.0
1844
0.4
91.4%
SPQR-3 = 100.0 (green). For effective visualization, SPQR-6
10.0
1603
32.7 0.4
92.3%
we plot only the eigenvalues that violate Wigner's semi- SPQR-8 circle.
100.0
698
32.9
97.6%
Online Reinforcement Learning. We implement SPQR-SAC- and SPQR-REDO with UTD ratio 1 and 20, and compare them to SAC-Ens and REDQ show how SPQR improves performance. We evaluate each algorithm on the OpenAl Gym MuloCo tasks. Figure 7 shows the training curves for SAC-Ens, REDQ, SPQR-SAC-Ens, and SPQR-REDQ. Since the two main contributions of REDQ. UTD ratio, and subset minimization, are orthogonal methods to each other, we experiment with SPQR-REDQ for UTD ratio and 20. high UTD ratio can be interpreted frequent gradient updates, thereby resulting in better performance, For each algorithm, we plot the average return as solid curve and the standard deviation as shaded area. The implementation of SPQR yields notable improvements in the performance of SAC-Ens and REDQ across all tasks, as well as under both UTD ratio and 20 conditions. Specifically, SPQR-SAC Ens performs times better than SAC-Ens on the Hopper task. Also, PPQR-REDQ-UTTE and than REDQ-UTD=1 and REDQ-UTD= for the Walker2d and Ant tasks.
show 1.1 and 1.3 times better
Humanoid
Hopper
6000
SAC-Ens
3500
SPGR-SAC-Ens
5000
REDQ-U TD=1
APOR-REDO-UTD::
4000
REDQ-UTD-20
Avnraga retum
SPGR-REDO-UTD-20
2000
500
50020
100000
150003
200000
250020
20000
42000
50000
83000
100030
timesteps
timestaps
Walker2d
Ant
6000
5000
6000
4000
Avargge Whh
4000
3000
3000
2000
2000
1007
0-
50020
100000
150003
200000
250020
50000
100000
150000
202000
250000
timestops
timestaps
Figure 7: Average returns of SPQR- SPOOR-AAC-BLL SAC-Ens, SPOR-REDDQUTIDDI, and on the MuJoCo environment. Averaged over 4 random seeds.
SPOR-REDO-UTID-2 SDD.O

---

To investigate why SPQR performs better, we evaluate the bias of the Q-value computed by the Monte- Carte-CCrlooaaoo algorithm. The average the Q-bias decreases with SPQR, which can be interpreted as the relationship between the distribution of the ensemble becoming more independent. We also evaluate the ablation study for the UTD ratio. SPQR-SAC-Ens shows performance improvement and bias alleviation with low standard deviation as the UTD ratio increases although high UTD SAC causes large and unstable bias. In terms of diversity, we also evaluate the standard deviation with different 3 for SPQR. As increases, the standard deviation for explored state-action also increases all training timesteps. In summary, Q-ensemble independence is the key factor in improving the performance of SPQR. Further details and experimental results be found in Appendix E
Offline Reinforcement Learning. SAC-Min formally constructed and evaluated 2 We follow the reported value and evaluation protocol from the paper. We consider baseline behavior cloning (BC), CQL, EDAC, and SAC-Min. The evaluation results of APQR-SAC-Min compared with SAC-Min and other baselines shown in Table 2 For most tasks, SPQR outperforms all baselines well. Specifically, for x-random dataset, SPQR improves SAC-Min by about 16% when CQL and EDAC fail for +-random dataset. It implies that SPQR is robust for the quality of data. Since SPQR gives more conservatism, which shows identical behavior with increasing ensemble size N, SPQR-SAC-Min needs smaller number of ensemble networks to outperform SAC-Min. For the hopper-mediug SAC-Min uses 500 Q-networks and SPQR uses only 50 Q-networks, which is about 90% efficient in terms of computational cost. It implies that the independence regularization of the Q-ensemble becomes computationally efficient.
We also implement SPQR CQL(H), denoted SPOR-COL-Min. Referring to the implementation details in the paper and the original baseline code of CQL, the original CQL uses a twin Q-function trick, which is equivalent to SAC-Min with N 2. We modified CQL CQL-Min use N
Table 2:
Normalized average returns of SPQR-SAC-NE and SPQR-CQL-Min on D4RL Gym, Franka Kitchen, and Antmaze tasks, averaged over 4 random seeds. Except for SPQR-*, the evaluated values of baselines are reported refer to [2]11] and reproduced for =-full-replay tasks of CQL-Min. Reproduced values are indicated as italic font.
Gym
BC
CQL-Min
EDAC
SAC-Min
SPQR-SAC-Min
walker2d-random
1.3 + 0.1
7.0
16.6 6..6+77 7.0
0.0
24,6 1.1
walker2d-medimm
70.9 11.0
74.5
92.5 0.8
0.2
98.4
walker2d-exper
108.7=0.2 0.2
121.6
115.1
116.7
115.2 1.3
walker2d-medt -eder/ddmdddum-eeppee -xpeert
114.7
116.7
0.7
wulker2d-mediume me--mdduum-replay
2.8
32.6
87.1 1.3
7..7
87.8 2.5
walker2d- wareerdddff-flrllllee
68.8
98.96
9.7
94.610.5
101.1 1 0.4
maffcheetah-randon
2.0
35.4
2.
28.0
3 2.5
halfchectah-une -eedimm
0.6
44.4
6..9
6.2
74.8
halfchoctah--exper
91.8
104.8
106.8
1.5.2
112.8 0.3
halfchectah- medium-expeet expert
4..0
62.4
10
107.1
114.0
halfeheetah- medium-replay
3716
46.2
613
63.9
69.7
halfehectah-full-cee
6.89
82.07
8..
1.2
88.5 0.5
hopper-random
3.7
25.3 10.4
3.0
35.6
mupper-medium
54.1
100.3
hopper-expert
107.7
109.9
110.1 0.11
110.3 0.3
112.0 1 0.2
hopper-medium- expert
53.9
111.0
110.7 0.1
110. 0.
112.5 - 0,3
16.6 4.8
48.6
10L.0 + 0.5
101.8 = 0.5
104.9 - 0.7
hopper-full-reple. replay
19.9 1 12.9
104.85
105.4 1 0.7
102.9 1 0.3
109.1 1 0.4
Average
49.9
70.9
85.2
84.5
89.6
Franka Kitchen
BC
SAC-Min
BEAR
CQL-Min
SPQR-CQL-Min
kitchen-complete
33.8
15.0
0.0
43.8
47.9 =: 12.3
kitchen-partia partial
33.8
0.0
13.1
49.8
54,2 1: 7.2
kitchen-mixed
47.5
2.5
47.2
51.0
55.6 11.6:177 7.9
Average
38.4
5.8
20.1
48.2
52.6
Antmaze
BC
SAC-Min
BEAR
CQL-Min
SPQR-CQL-Min
65.0
0.0
73.0
74.0
93.3 4.7
umaze-diverse
55.0
0.0
61.0
84,0
80.0 0.0
medium-play
0.0
0.0
0.0
61.2
80.0 :: 8.2
Average
40.0
0.0
44.7
73.1
84.4

---

Q-networks to compute the Bellman target. Hence, the original CQL can be called CQL-Min with N We evaluate -PQR-CQL-Min on D4RL Franka Kitchen and Antmaze, one of the most complex and sparse tasks in D4RL.
In T evaluate SPQR-CQL-Min with other baseline methods. SPQR-CQL-Min outperforms all baseline algorithms, notably 2.6 times better than BEAR. For Franka Kitchen tasks, SPQR consistently improves CQL by 10% for all three tasks, which can be interpreted as robustness over the quality of data. Surprisingly, SPQR achieves the maximum score on different Antmaze tasks, which outperforms all existing algorithms. Further details and results are presented in Appendix D and E
Can other diversification methods benefit from the SPQR? Since SPQR regularizes Q-ensemble independence, which is the orthogonal method with other diversification methods, we implement SPQR in EDAC to benefit from the combination of two orthogonal methods. We evaluate SPQR- EDAC for the tasks where EDAC outperforms the baseline algorithm SAC-Min. SPQR and SPQR- EDAC show large performance gains over SAC-Min, as shown Table
Table 3: Normalized average returns of SPQR-SAC-Min and SPQR-EDAC on D4RL Gym tasks, averaged over 4 random seeds. Except for SPQR-* the evaluated values of baselines are reported.
Refer 2
Gym
EDAC
SAC-Min
SPQR-SAC-Min
SPQR-EDAC
calker2d-medinee
92.5 0.8
0.2
98.4 2.0
1.0
malker2d-mdduum-rplla
87.1
0.7
87.8 2.5
89.6 1.8
fikkerrdd-ull-replaa
0.7
9..6
101.1
1
halfcheetah--ee experet
106.8
10
112.8 0.3
110.3 0.2
halfcheetah-full full-replay
0.9
84.5
88.5 0.5
86.8 0.2
hopper-medium
101.6
1.0
100.2
103.8 + 0.1
medper-medium-expeer
110.7+0.1
+.3
112.5 0.3
11 + 0.0
hopper-full-ree reepply
105.4
102.9
109.1
108.2 + 0.5
6 Conclusions
We propose SPQR, a Q-ensemble independence regularization framework for ensemble Q-learning by applying random matrix theory and spiked random model. We theoretically propose a tractable loss for Q-ensemble independence. The proposed theoretical approach does not require any prior assumption on the Q-function distribution, such uniform distribution or i.i.d. In addition, we present the difference between diversity and independence and empirically show the orthogonality of the diversification method and SPQR. Furthermore, SPQR serves conservatism control framework for ensemble Q-lcarning by effectively regularizing independence with various empirical evidence. We also empirically show that SPQR is computationally efficient and outperforms many existing ensemble Q-learning algorithms on various online and offline RL benchmark tasks.
Acknowledgements
We appreciate Minyoung Kim for creating an illustration for our paper, well as Jachoon Choi and Yejin Kim for writing advice. We also thank the anonymous reviewers for their insightful feedback and reviews. This work is in part supported by National Research Foundation of Korea (NRF, 202IR 1A2220140502200)) communications Technology Planning & Evaluation (IIT,, 0011--00016((0%) (20%)) 0021-00-2008((0%%) grant funded by the Ministry of Science and ICT (MSIT), INMAC and BK21 FOUR Program.
Institute of Information &
10

---

References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Huee Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do i can, not say: Grounding language in robotic affordances, 2022.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In NewrIPS, pages 7436-7447, 2021.
[3] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning, pages 176-185. PMLR, 2017.
[4] Jess Banks, Cristopher Moore, Roman Vershynin, Nicolas Verzelen, and Jiaming Xu. Information-theorrtii bounds and phase transitions in clustering, sparse pca, and submatrix localization. IEEE Transactions on Information Theory, 64(7):4872-435 2018.
[5] Nicholas Baskerville, Diego Granziol, and Jonathan P Keating. Appearance of random matrix theory in deep learning. Physica A: Statistical Mechanics and its Applications, 590: 2022.
[6] Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Represen- lations, 2021.
[7] Hye Won Chung and Oon Lee. Weak detection of signal in the spiked wigner model. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1231-1241. PMLR, 09-15 Jun 2019.
[8] Jorio Cocola, Paul Hand, and Vlad Voroninski. Nonasymptotic guarantees for spiked matrix recovery with generative priors. Advances in Neural Information Processing Systems, 33: 15185-15197 2020.
[9] Momar Dieng. On the distribution of eigenvalues and their spacings: rapid introduction to random matrix theory.
[10] Freeman J Dyson. The threefold way. algebraic structure of symmetry groups and ensembles in quantum mechanics. Journal of Mathematical Physics, 3(6): 121-1 1962.
[11] Ahmed EI Alaoui El Abidi. Detection limits and fluctuation results in some spiked random matrix models and pooling of discrete data. University of California, Berkeley, 2018
[12] FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624): (066624):1067-111 1067-1074, 2022.
[13] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine, D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/99 2020.
[14] Mike Giles. An extended collection of matrix derivative results for forward and reverse mode automatic differentiation. 2008
[15] Hado Hasselt. Double gelaanning. Advances neural information processing systems, 23,
2010.

---

[16] Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix backpropagation for deep networks with structured layers. 2015 IEEE International Conference on Computer Vision (ICCV), pages 2965-2973, 2015.
doi: 10. 10.1109/CCCV2201533399
[17] lain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, 29(2):295-327, 2001.
[18] Ji Hyung Jung, Hye Won Chung, and Ji Oon Lee. Detection of signal in the spiked rectangular models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5158-5167. PMLR, 18-24/0 Jul 2021.
[19] Robert W Keener. Theoretical statistics: Topics for a core course. Springer, 2010.
[20] Hya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcementte learning with implicit /6160000000
q-learning. CoRR, abs/21 10.06169, 2021. URL
[21] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33: 1179-1191, 2020.
[22] Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2020.
[23] Lucien LeCam. Locally asymptotically normal families of distributions. Univ. California Publ. Statist., 3:37-98, 1960.
[24] Kimin Lee, Michael Laskin, Aravind Srinivas, and Picter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 6131-6141 PMLR, 18 24 Jul 2021.
[25] Zhenyu Liao and Romain Couillet. The dynamics of learning: A random matrix approach. In International Conference on Machine Learning, pages 3072-3081, PMLR, 2018.
[26] M.L. Mehta. Random matrices and the statistical theory of energy levels, 1967.
[27] Karl Pearson. X. on the criterion that a given system of deviations from the probable in the case of correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 177-55) 1900.
[28] Julian Schrittwieser, Ioannis Antonoglou. Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. CoRR, abs/191 108265, 2019.
[29] Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures. In International Conference on Machine Learning, pages 8573-8582. PMLR, 2020.
[30] Hassam Sheikh, Kizza Frisbee, and Mariano Phiclipp. DNS: Determinantal point process based neural network sampler for ensemble reinforcement learning. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19731-19746, PMLR, 17-23 Jul 2022.
[31] Hassam Sheikh, Mariano Phielipp, and Ladislau Boloni. Maximizing ensemble diversity in deep reinforcement learning. In International Conference on Learning Representations, 2022.
[32] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
12

---

[33] Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[34] Eugene P. Wigner. Characteristic vectors of bordered matrices with infinite dimensions. Annals of Mathematics, 62(3)544--64, 1955. ISSN 0003486X.
13

---

A Review for Detection Theory and Spiked Random Model
We call strong detection when we perfectly detect the presence of spike signal and weak detection when we detect the presence of spike signal with some error by hypothesis testing. A formal
definition for a strong detection can be constructed under the notion of contiguity by [23]. The
contiguity of the spiked Wishart model is proven by the second moment method, which means that we cannot distinguish without error whether data is pure noise or contains some signal.
Definition A.1 (Contiguity). Let P,, and Qn be two sequences of probability measures defined on the sequence of measurable spaces Fn)) Qn is contiguous with respect to the sequence P,, if (An) + 0 implies Qn (AA,) 0 as + for every sequence of measurable sets A,, Fu.. denoted as Qn < Pn. The sequences Pn and Qn are mutually contiguous if both Qn <I Pn and Pa <1 nn holds, denoted as Qn <>
Lemma A.2 (Le Cam's 's First Lemma). Let Pn and On be two sequences of probability measures de- fined on the sequence of measurable spuces (ln,Fn) Then the following statements are equivalent:
1. Qn 4 Pn
2. If dP
On, U along a subsequence, then P(U > 0) =1
3. If dQn/dP,, Pns V along a subsequence, then E[V] = For any statistics Tn Pn + RR IfTT I"> O, then Tr 2n,
Lemma A.3 (Second moment method). Let P., and Qn be two sequences of probability measures defined on the sequence of measurable spaces (St,, Fa), and Badon-Nikodyye derivative of Pa with respect Qn as L, dPn//Qn Iim sup Eq,, [L2] oc, then P,, <1 Qn
Lemma A.4. For spiked wishart model following definition 3.2 K < 1, the families of distributions Po and Ps, inineexed by M, N, are mutually contiguous in the limit N + co, M/N -
Hypothesis Testing for Weak Detection. To distinguish whether spiked Wishart model Y follows Fo or Fo. detection error is inevitable since contiguity holds. Even if the perfect test is not available, needs be better than random guess, which is achievable by constructing a hypothesis test. Let (S,F, p) be probability space, (X, B3 be a topological space with Borel o-algebra B, and θ to be set of parameters. For observed random variable X Ω X' with X ~ Pa. 0 E O, and fixed & * 0, Objective of hypothesis testing is to distinguish between two hypotheses, called null hypothesis Ho and alternative hypothesis H1
Ho 0 = 0, H1 0=
Constructing a measurable function T : + {0, 1] } to represent test, T(X)= = 0 means accept Ho and T(X)= = 1 means accept H1. The error of test T is the sum of Type-I error, false positive, and Type-II error, false negative, stated as,
err(T) Pa (T(X) (rr(T)=PP(((())
0) Po(T(X)= 1)
Optimal test 7* is defined as minimizing Type-ll error subject to the constraint that Type-I error is no greater than predetermined level a. 7 must have a form of likelihood ratio test by following Peyman-PParseen Lemma and propositions from 119]
Lemma A.5 (Neyman-Pearee Lemma). Let null hypothesis Ho with probability measure po and density fo. alternative hypothesis Hi with probability measure pt and density fú, density ratio r(x)= = fi(z)/ fz( and constant E IR such that po po(r(x) > k] S a for a given significance level There exist optimal test function T RDD + R. with mild continuity assumption, such that
(') = (f1 ff)))(()))))))))))))))))())))))) k) = ((r(x) > k)
To construct an optimal test by applying Neyman Peaassn Lemma, we define a likelihood ratio, Gradon-Nikodyy derivative between P.., and Po as L(Y; (b) by the following equations. For convience.
let Hamiltonian H N-N + R be as follows:
H(u, v) = = V
(5)
2N
dP&N (u) = dP.(w))
8 ... dP, dP,(ux))
(6)
14

---

where & represents a tensor product, L(Y; 2b) has form of dP
= dPo - o dPoN (u)dpaM(w)
The optimal test for spiked Wishart model is given as follows:
Lemma A.6. Let us assume that a spiked Wishart model is defined in definition 3.2 Let null hypothesis Ho Y Po, alternative hypothesis H1
Fg. likelihood ratio L between Ps and Po, and define tests T -RN + {0, 1}. By Lemma A.5 optimal test is,
Ho Y Po. if Lo S
(Y) = 1(lgg 7 -20
H1 : Y Pv.
if log L >> 0
Proof of A.6 is given in Section B
Corollary A.7. For optimal test T Error of
errMMN (T*) Po(log xb) > + Pw log L(Y; 5 (rv(Pr,PPo)
For 1b 2 0 such that K K <
lim err erTM.N M.N (1"* =1 N
lim Drv ersc TV log(1 - (62)
N
where er fe(x) = / 1 e-
The error for optimal test 7 is stated at Corollary A.7 Likelihood ratio log is used for the optimal test to distinguish whether data Y in ensembles are sampled from i.i.d prior Fo. However, log L intractable since information about & and u, v is not given. Therefore, we need an alternative criterion for optimal test 7 that does not require prior knowledge to make it tractable with ensemble Q-learning- Intuitively, although we cannot know the exact form of P. and Po. its eigenvalue distribution can be described by ESD and Wigner's semicircle law. The expectation of log I. becomes KL divergence between P and Po, followed by the definition of log L. By integrating these intuitions, intractable criterion log L can be restated as KL divergence between ESD and Wigner's semicircle law, which we prove in Section
Proof of Lemma
A.2
The proof is exactly same as the Lemma 6.4 of 33
Proof of Lemma
A.3
For any event Fu. by Cauchy- Schwarr inequality,
Px(A)= / 1 (a(w)dP,(ww) / 1 (w)),,
/ RRaQu: v IndQun BE [T v (4))))
If lim sup Eq,, [4.]] 0o, then Qn implies Pa (An)) 0, concludes Pa < Qn
Proof of Lemma A.4 A.4 The proof is exactly same as the Theorem 4 of [
15

---

B Proofs
B.1
Technical Lemmas and Proof
Before proving our theoretical results, we present some lemmas to clear the deseription.
Lemma B.1 (Convergence). Let such that yy 1, then in the limit N - oo, where ~ denotes convergence in distribution,
log L(Y; N(( = log 26²), logg11 199)
(8)
The sign means Y Po. means Y P&
Lemma B.2 (Concentration), for fixed u", v., Y = V u + W, for every t 20, 0,
Pr[|log L(Y: :b) Ellog :(Y: 2 Nt) S 2xx2( Nexp(-NN/26K330
Lemma B.3 (Weyl's Lemma). If X is rotational invariance,
PIX]
((Tr(X),Tx(X)) (TX,),
Lemma B,4 (Vandermonde determinant). For GOEY OAO", the relationship between joint probability density function of matrix entity and eigenvalues is given as,
P(Y11, Y12,-.. .YN yY
P()1, X2, AN. .N. bx.(O))dddddg...da P(A1, ... AN, {0}) = P(Y,1, Y12, YYN)]J(( + X, (0)) where J(Y - X, (0}) dYiıdY1 dYNN htw/dd,db d..dg---ddd()))
Then vandermonde determinant become
J(Y - A. (O)) J(Y + X) = |aY/dA|
II A - Xn1 i
Proof of Lemma
B.1
The proof exactly same as the Theorem 3.1 of
Proof of Lemma
B.2
The proof is exactly same as the Lemma 3.14 of
Proof of Lemma
B.3
The proof exactly same as the Lemma 1.1 of
Proof of Lemma
B.4
The proof is exactly same as the procedure in Chapter 3 of [26
Proof of lemma A.6
Using Lemma B. and Lemma A.5 log can distinguish at log = as
N - 00 because of symmetry of Gaussian mean, i.e for Neyman-Pearson Leamma. Therefore, optimal test T (log > 0)
B.2 Proof of Main Theorem
Theorem 4.1. Following definition 3.2 as N + oo with probability at least 8, following test T(A) is optimal.
Ho Y Po T(A) (DK))(pp(A) Psc))) 2) 111 Pv
Proof. By Lemma B.3 and Lemma B.4 For spiked wishart model Y ~ Ps
(A) P()1, A2..... ;.N) = dOP(A), Av. (0))
- / doP(Yn Y12.... (120 (NN))J(Y X,H)
-
-IollPo(Y) / do, where / |d0 constant(volume)
16

---

If (iEE(o(()) <E,
Jp
(KI(PN, Po)
Pac, + |asl log
C> 0
|Jo|
= DF, Po) < M E + 1 log ClJ
=E, :1ppp > 0
|Jol
Let A log
Ellog L). = Nt. By Lemma B.2
Pr([A] 2 1) s 2 expp( - (2/K2)
Let 5 = 2 Then, with probability at least 8,
E | man, Amaz]
maz 2N log (2/8)
Since (Pr, Po) < E, then,
log L = Bllog L) + = P
Choose غ == = 8' > 0. Then,
log L < 8 <0
:. with probability at least
6, test T(A) is optimal test T as + 00
17

---

C Algorithm Details
C.1 Detailed Algorithm for SPQR
We describe a detailed version of SPQR for Algorithm 3 A detailed description for computing SPQR
loss shown Algorithm 4 Table shows functions used in implementing baseline ensemble Q-learning. For Figure 8 we visualize an illustrative example of SPQR.
Algorithm SPQR
Input: discount 7 E [0, 1), temperature parameter a, ensemble size N, SPQR loss gain (3, polyak т, batch size [B]
Initialize policy parameter 0, Q-function parameters { target Q-function parameters {ø} <-
@.);-11 online/offline replay buffer D
repeat
not offline dataset then
Take action from a, To (e((ss)), observe reward re, next state 8c--1
D + DU {18t, at rt, 11+1)
end if
aj rj {1)))111 from B + {(jj,,j,
D. Sample a mini-hatch
v(r,s)) +- t
Enstar (sQo a log a' ~ To(-ss") Compute Q-target
LSron + Lsron((0.)
Compute SPQR loss using Algorithm|
for = do
Lo: f [B] E(s.nn
(Qo, (8, (aarrra((aaa 8')² + BLSPQR
b Compute Q loss
Update Φ with gradient descent using Co. Update 6f + TØ: + (1 T)
end for
Lo B E.EB Finseval( (1Q (s, a)) log am
xo(-1a)
Compute policy loss
Update with gradient ascent using VoLo
until
Output: parameter ( (
Table 4: Target and evaluation functions used in Ensemble Q-learning algorithms
Algorithm
Ensur (s' (s',a')
Ens (Q (s,
SAC-Ens
N/ E (s',a')
N
+
Qa: (8,
REDQ SAC-Min CQL-Min
minigM Qoy M (I...,N
>
Qs, (s,
N i-
min; lo
min; I.... N Q0. (8,
min
min; N (8,
18

---

Volu
Volu
Sutt-Actoo
State Aotions
Ugenvaaiv Datribudon
G-ersenntlee
Independent O Gressemble
Y- P
Wi
G C-emsembre
Inputs
Impute
SPQR
Baseline
Figure 8: Illustrative example of SPQR compared to baseline methods.
C.2 Detailed Algorithm for SPQR Loss
detailed explanation for computing SPQR loss. We build symmetric random matrix with a randomly-orderes Q-value. Our goal is to generate symmetric Q-value matrix filled with an ensemble size of N. To avoid training or capturing the order of a matrix, we shuffle the filling order. This can be achieved by sorting the list of Q-networks 44. In order to create symmetric matrix of maximum size, the upper triangle part and the corresponding lower triangle part are equally filled.
Algorithmmeert
The size of the matrix is set as D = 1T+&N--1 J. D can be found by finding the largest integer that satisfies D'+D < N. To avoid numerical instability when computing KL divergence, we formulate 2
Wigner's 's semicircle distribution like s-soft, in which all probability distribution value is at least and we call it a soft Wigner's semicirele distribution, p gives weight between E-soft distribution and Wigner's semicircle distribution. We fix 0.5 and 0.01.
Algorithm 4 SPQR Loss
Input: List of Q-network 4
(ox (s',')
Y E DDD, D= 1-VI-1 1= -I--NN-
Yg 1Dptor if
Build random matrix
Yypps otherwise
Y 5- (Y- - -uy)/oy
Normalizing Compute eigenvalue
(A.) Eigen
Y)
JN
(esa(A) = N Σ, 5(1 Ac)
Define ESD where d dirac delta function
Pwigner (A) p p. v
2
2x
Define soft Wigner's semicirele distribution
(1 -p)e,
otherwise
(Ne) Pwigner (Xi)
LSPQR* + B E, E, Pesa(As) log
1
> Compute SPQR loss
Output: SPQR loss LSPQR
19

---

C.3 Eigenvalue Backpropagation
To use DxL (p (A), Psc (s))) as regularization loss, we need backpropagate loss gradient through eigenvalue. In Theorem C.I we construct backpropagation equation about eigenvalue refer to [6 and [14]. For practical implementation, using torch. eigvals and torch. eigvalsh gives numerically stable eigenvalue backpropagation by solving del(X X AT) = 0, not solving Xu Au geometrically.
Theorem C.1. Let L: Rd + R be loss function, and X UAUT with X Rmxm such that U = and A possessing diagonal structure. Using the notation A o B as Hadamard product of two matrices, which (Ao B)g Ais Big and Asym 2A" A), Adiag
I Then
dA = (UT UXUdXU)an )diag
du U(K (U dXU) 2ymm)
i
Kij
x; >ラ
0,
i=
The resulting partial derivatives are
0L
(UT dL aU
0L aA
KT
UT
0X
sym
diag
= Proof. - UAU _ then XU U'A Differentiation gives
dXU + XdU dU'A UdA
Define dC' UTAU so that dU UdC, then
dXU + UNCC UdCA
Multiplying U to both sides and rearranging gives
dCA - AdC + dA = dXU
Let Fig >, Xjs then
=
Rearranging gives,
EodC + dd U dxU
Since the diagonal of E are zero,
dA 10 (XU (UT d'U ) diag
So,
1
i i
dC = Koo (XUU = du = U(K (UT dxU)), where Kig =
A
2=
0,
i=j
=dA= (xU) (UT (UF)XU)) dings dU U(K (UFdXU))gn)
20

---

D Implementation Details
D.1 Online RL
We implement SPQR to SAC-Ens and REDQ on the REDQ code, SPQR to SAC-Min on the SAC- Min code, and SPQR CQL on the CQL code. For online MuloCo environment experiments, implement SPQR-SAC-Enn and SPQR-REDQ. We use s-v2 version of MuJoCo environments and train 12.5k timesteps for Hopper and 30k for Humanoid, Walker2d, and Ant. Hyperparameters shown in Table 5 are the same with REDQ implementation, including the size of a subset M = 2 for REDQ and SPQR-REDQ. For fair evaluation, we fix ensemble size N for each corresponding baseline algorithm, N = 20 for SAC-Ens and SPQR -OR-SSC-Ens. and N = 10 for REDQ and SPQR-REDO We use annealing beta with a linear decay to stabilize training.
Table 5: Hyperparameters used in the Mujoco environments
Hyperparameter
value
discount factor y
0.99
initial temperature parameter a
0.2
Q learning rate policy learning rate hidden layer width
3e-4
3e-4
256
hidden layer depth batch size [ polyak parameter optimizer
2
256
0.995
Adam
Table 6: Algorithm hyperparameters used in the Mujoco environments
Task
SPQR-SAC-EII (N, B)
SPQR-REDQ (N,
Hopper
20,0.5
10.0.5
Humanoid
20,2.0
10,2.0
Walker2d
20,0.1
10.0.1
Ant
20,2.0
10,2.0
D.2 Offline RL
SPQR-SAC- SAC-Min is implemented based SAC-Min and evaluated on D4RL MuJoCo Gym tasks with three environments, walker2d, halfcheetah, and hopper, each having separated six datasets. 1 1M samples generated from the
Each dataset is built with different behavior policy. random randomly initialized policy. expert uses 1M samples generated from a policy trained to completion with SAC. medium uses IM samples generated from policy that performs approximately 1/3 of the expert policy. medium-replay uses the replay buffer of policy that performs as much as the medium agent. full-replay uses 1M samples from the final replay buffer of the expert policy, medium-expert uses 50-50 split of medium and expert data, slightly less than 2M samples total. For fair evaluation with SAC-Min and EDAC, x-v2 datasets used for all tasks, and the same hyperparameter with SAC-Min and EDAC, as shown in Table 7
We use normalized score by calculating 100 raw__soom wandom__ccom
expert. repert_score- score random score' proposed evaluation metric from 13 Algorithmic hyperparameter / shown in Table g We fix over similar tasks to show the robustness of SPQR. Also, we use smaller ensemble size N for SPQR-SAC-ME rather than N for SAC-Min. For the hopper-* dataset, SPQR drastically reduces ensemble size N. using only 10% of N for SAC-Min. For SPQR-EDAC, SPOR-SAC-Min and = 0.5 for EDAC.
the same hyperparameter with
SPQR-CQL-Min is implemented based on CQL(H), called CQL-Min, and evaluated on Franaa Kitchen and Antmaze tasks, one of the most complicated tasks in the D4RL. environment. Since there no information about any hyperparameters for evaluating Franka Kitchen using CQL in the official
21

---

paper and code, we use hyperparameters given for evaluating Antmaze in the CQL paper, shown in 7 Like our efforts at SPQR-SAC-AR SAn-Min to test the SPQR's robustness, we fix all hyperparameters 8, as shown in Table 8
For all D4RL tasks, we use annealing beta with exponential decay to stabilize training.
Table 7: Hyperparameters used in the D4RL tasks
Hyperparameter
Gym
Franka Kitchen
Antmaze
discount factor
0.99
0.99
0.99
initial temperature parameter
1.0
1.0
1.0
Q learning rate policy learning rate hidden layer width hidden layer depth batch size [B] polyak parameter 7 optimizer lagrange threshold (for CQL)
3e-4
3e-4
3e-4
3e-4
le-4
1e-4
256
256
256
3
3
3
256
256
256
0.995
0,995
0,995
Adam
Adam
Adam
5.0
5.0
Table 8: Algorithmic hyperparameter used in the D4RL, Gym and Franka Kitchen tasks. SAC-Min and CQL-Min follow reported values. Refer 2 and [2]
Gym
SAC-Min (N)
SPQR-SAC-Min (N)
SPQR-SAC-Min (3)
walker2d-random
20
20
walker2d-medium
20
10
walker2d-expert
100
100
20
20
mulker2d-medium--rpphe
20
10
walker2d--uull-eepaay
20
20
10
serrhheeta--meddiue
10
3
halfchectah-exxe expert
10
10
0.1
halfcheetah-meche experrtt
10
10
halfcheetah- medium-replaa
10
10
halfcheetah-
Saahhetthh--ullelllteee full-replay
10
hopper-randee
500
50
hopper-medium
500
50
hopper-expert
500
200
Dopper-medium-expee
200
50
hopper-medium-i replay
200
50
hopper-full-aae full-replay
200
50
Franka Kitchen
CQL-Min (N)
(N)
SPQR-CQL-Min (3)
kitchen-complete kitchen-partial
2
2
2.0
kitchen-mixed
2
6
Antmaze
CQL-Min (N)
SPQR-CQL-Min (N)
SPQR-CQL-Min (8)
umaze
2
umaze-diverse
2
2.0
medium-play
2
medium-diverse
2
22

---

E Further Experimental Results
E.1 Evaluation Result with More Baselines
Table 9: Normalized average returns of
SPQR-EDAC and SPQR-CQL-Min on
D4RL Gym, Franka Kitchen, and Antmaze tasks, averaged over random seeds. Except for SPQR- the evaluated values of baselines are reported. Refer to 22 [20), and [21]. and reproduced. Reproduced values are indicated as italic font.
Gym
BC
CQL-Min
EDAC
SAC-Min
SPQR-SAC-Min
SPOR-EDAC
walker2d-rundom
1.3 0.1
7.0
16.6.:70
2.0
24,61::1 1.1
walker2d-medinn
11.0
74.5
0.8
87.9
9.0
94.8
walker2d-cxper
0.2
121.6
115.1=19
116
115.2
13.2
98.7
1.9
11
118.2
repply
9.8
32.6
87.1
0.7
2.5
89.6
walker2d
likkee22d-fllle -ull-ly
68.8
98.96
99.8
0.5
101.1
102.2 0.2
halfchectah-ranne -anndom
2.2
35.4
28.4
28.0
+..
43.2
44.4
63.9
67.5+
74.8
halfcheetah--xxpe expert
106.8
0.3
110.3
halfcheetah- medium-expert
1.6
62.4
106.3
20
114.0 1.6
allfcheetah--oe medium-repla,
3.1
46.2
613
63.9
69.7 = 1,4
halfcheetah- cheetah--ull--aeee full-replay
62.9
82.07
8..6
84.5
88.5 + 0.5
86.8 +: 0.2
hopper-random
3.7
7.0
25.3 10.4
31.3
35.6 1.4
hopper-medium
54.1
86.6
1.
1.3
1.3
103.8 -
hopper-expert
9.7
109.9
110.1
0.3
112.0
hopper-meding
5..9
111.0
110.7 = 0.1
110.1
112.5 0.3
111.7
pooper-meduum-replay
16.6 4.8
48.6
101.0 + 0.5
101.8 + 0.5
104.9 + 0.7
hopper-full-repie fell-replay
19.9 : 12.9
104.85
105.4 0.7
102.9
0.3
109.1 0.4
Average
49.9
70.9
85.2
84.5
89.6
Franka Kitchen
BC
SAC-Min
BEAR
IQL
CQL-Min
SPQR-CQL-Min
kitchen-complete
33.8
15.0
0.0
62.5
43.8
47.9 12.3
kitchen-partial
33.8
0.0
13.1
46.3
49.8
54,2 7.2
kitchen-mixed
47.5
2.5
47.2
51.0
51.0
55.6 1: 7.9
Average
38.4
5.8
20.1
53.3
48.2
52.6
Antmaze
BC
SAC-Min
BEAR
IQL
CQL-Min
SPQR-CQL-Min
65.0
0.0
73.0
87.5
74.0
93.3 4.7
umaze-diverse
55.0
0.0
61.0
62.2
84,0
80.0 0.0
medium-play
0.0
0.0
0.0
71.2
61.2
80.0 8.2
Average
40.0
0.0
44.7
73.6
73.1
84.4
E.2 Hypothesis Testing for Uniform Distribution
To check whether bias et, follows uniform distribution, it's sufficient check Q'(s, a) follows uniform distribution since (s,s, a) is constant for fixed 5, [27] propose powerful test, called x² test which determines whether given distribution follows uniform or not using hypothesis testing. 0.025 on 50 Q-ensemble trained in the D4RL dataset. Since
We perform x² test with significant level hopper-random dataset in 3M timesteps with 10000 data of D4RL ceoper-fulll-eppee the x² test be applied in discrete random variable, need to construct an approximative probabilistic mass function by binning. In Table 10 we show the acceptance rate of x2 test over 10000 data with different bin sizes. Especially in experiments with 20 bins, SPQR-SAC-Min shows better acceptance rate than SAC-Min approximately 40%.
Table 10: x² test
Algorithm
bin=10
bin=20
bin=25
bin=30
bin=50
SAC-Min
20.08%
32.12%
36.38%
38.54%
50.99%
SPQR-SAC-Min
33.67%
44.97%
49.78 %
51.12 %
63.56 %
23

---

E.3
Conservatism of ensemble and SPQR
We visualize how ensemble size N affects the conservatism, and how SPQR can be computa- tionally effective. As we described in Section 4.1 and Figure [ SPQR has conservatism power. Increasing ensemble size N for SAC-Min shows conservative behavior, as proven [22] and Figure 9 We train SAC-Min for 3M timesteps in halfcheetah-randon dataset with ensemble size N 3, 5, 10, 15, 20, 50. SAC-Min empirically shows conservative behavior as ensemble size increases. As result, SPQR can be computationally efficient algorithm when using SAC-Min.
150
Predictions
200
50-
N-5
N=10
N=15
N=50
timesteps
Figure 9: Mean of predicted Q-value with various N, averaged over seeds
E.4 Spike histogram visualization
We plot more detailed visualization of the spike histogram for Figure 10 spike histogram of EDAC and SPQR varies compared with SAC-Min.
We visualize how the
Migg sorc eeeccaa
Whg ShoN-sc-- expoe
60--
EIAC. #ig
250-
10--
sac Figain 2006
No.
3.16
11.4
100
1M
24
234
1118
335
100-
1.00
12.0
10.0
-20
-.00
130
1.0
425
215
430
Figure 10: Spike visualization SAC-Min, EDAC, and SPQR-3 00.01, 0.1, 1.0, 10.0, 100.0.
24

---

E,5 Bias of Q
In Figure I1. we plot the average normalized bias and standard deviation of normalized bias. Bias is calculated by Monte-Cart Carlo estimation. SPQR-* decreases average bias and standard deviation of bias from baseline algorithms. For all environments, standard deviation of bias than SAC-Ens. For Hopper and Humanoid tasks, UPQR-REDO shows underestimation, and for Walker2d and Ant tasks, SPQR-REDQ shows unbiased estimation, For Ant, SPQR-REDQ shows less bias than REDQ.
SACC-SAC-Ens shows smaller bias and
Hopper
Humandi
Walker2d
Antt
546
12
SAG
SPOR-GCC
SAC-Era
SPGR-SAC--Ge
SAG Era
SPOI-REEDGGH1
SP D'ORREDD-GG
5-
SPOR-RRED
05
0.-
-10
Spracrpe
Imwddace
Hopger
SPON REDGGO-22
SP RCDO 0u:23
SINCH HEED 01-20
SPCH RE.00 ovice
500
Avcrge meamm
-0-
2008
2000 kibro 40300 Y000)
J032 19:00 Saieo
2005
TSEE
Figure 11: Average of bias and std of bias for SPQR-SAC-Ens and SPQR-REDO, averaged over seeds. G represents the UTD ratio.
E.6 UTD Ablation Study
We visualize the performance of SPQR-SAC-Ens over variation of UTD ratio in Figure 12 SPQR- SAC-Ens shows performance improvement by increasing the UTD ratio with conservative and stable bias.
SPOR #VC2
en24 AMC Bic
200
200
2002 12000
4200
125.0
2000 1200
10.00
20000
4200
41200
6000
19000
Figure 12: Left 1: Average return of SPOR-SAC-EMI for UTD ratio 1,5, Left 2: Average of bias of SPQR-SAC-Ens for UTD ratio 1.5, Left 3: Standard deviation of bias of SPQR-SAC-EiN for UTD ratio 1,5. All experiments are averaged over seeds. G represents the UTD ratio.
25

---

E.7 Early Collapse
To show how SPQR affects the ensemble when the agent collects new tuple (s, a, 7; s') terms of exploration and variance, we evaluate the standard deviation with different 3.
As [30] pointed the early collapse of Q-values occurs, which means that in early stage of training. the variance of the Q-ensemble is almost zero since Q-values become similar to each other. To show how SPQR addresses the early collapse problem,
visualize the standard deviation of Q-ensemble in Figure 13] We train 10 Q-networks from the MuJoCo Hopper environment with the SAC-Min algorithm with the same hyperparameter. SPQR does collapse Q-values during training, preserving appropriate variance of Q-value, and higher values of / result in greater variance for = 0.3 and 3 2.0.
0=3.
6-2.0
std of Q-value
0.8
0.2
20000
40000
60000
80000
100000
120000
timesteps
Figure 13: Standard deviation of Q-ensemble with different B, average over seeds
E.8 Hyperparameter Sensitivity
We evaluate SPQR over various demonstrate hyperparameter sensitivity. We fix N 10 for SPQR-SAC-Min and evaluate it the D4RL Gym halfcheetah--randon dataset and halfcheetah- medium dataset. Evaluation results are shown Table II]
Table 11: Performance of SPQR-SAC-Min over various the D4RL halfcheetah-* dataset,
averaged over 4 seeds
dataset
A= 0.0
/ == 0.01
B 0.1
== 0.5
/ 1.0
8 == 2.0
8 5.0
3 10.0
random
28.0 0.9
6. 3.4
0.4
30.6 +
30.7
31.2 2.2
31.1 1.8
32.3
medium
67.5 1.2
70.0 0.5
71.7 0.6
70.1 + 1.0
70.5 + 1.7
69.2 1.3
7.90
70.4 + 1.3
26

---

E.9
Compare with MED-RL
We implement and evaluate another diversifying method for online RL tasks, MED-RL. We the Gini coefficient for MED-RL and use reported hyperparameters with UTD ratio 1. SPQR outperforms MED-RL for all tasks.
Horper
Humanood
Walkerd
MADEO CIDR MZD SEDO LITOM
Mas2201311 VED-N200O-/THH
IEDSTCDDGLTTH
100
50
tredapa
treatips
Figure 14: Average returns of REDQ, SPQR-REDQ and MED-RL with UTD ratio the MuloCo environment, averaged over 4 random seeds.
E.10
t-SNE Visualization
Figure 15 shows t-SNE 32 clustering visualization of Q-ensemble trained 3M timesteps by SPQR- We use t-SNE clustering for the last hidden layer
SAC-Min algorithm at malfcheetah-mediin---ee representation of Q-ensemble with ensemble size 10 and 256 batch data at wewfichhetta-mmiiu----eee tasks. The t-SNE result shows that Q-ensemble trained by SPQR-SAC-Min is well clustered.
Figure 15: t-SNE for SPQR-SACA -AC-Min at halfcheetab-meciim--eeee
27

---

E.11 Correlation Visualization
We visualize a detailed figure for the Rightmost figure of Figure 1 the Pearson correlation coefficient matrix for fully trained Qensemble and initial Q-ensemble in Figure [6 We train each Qeensemlee the D4RL hopper-random dataset. For effective visualization, we sample randomly 10 Q-networks among 50 Q-networks and compute the Pearson correlation coefficient between each sampled Q- network with 256 batch data in hopper-full-replay dataset. For the initial Q-ensemble, Q-networks are uncorrelated from each other. As we train Q-ensemble with SAC-Min for 3M timesteps, Q- networks become strongly correlated with each other.
4.204
0.20
5.14
2-
0.4
DAR
0a2
640
AJ8
0.85
337
8:39
3.89
16
13
Figure 16: Left: Q-ensemble trained 0 /timesteps. Right: Q-ensemble trained 3M timesteps
28


Reviews:
[{"review_id": "tMyr6uwDep", "reviewer": "Reviewer_KBA2", "summary": "Mitigating overestimation bias is crucial for deep reinforcement learning. Existing works about the ensemble techniques for Q-learning have been explored to leverage the diversity of multiple Q-functions. The authors argue that there has been no attempt to ensure ensemble independence from a theoretical standpoint. The authors introduce a regularization loss for Q-ensemble independence, based on random matrix theory, called Spiked Wishart Q-ensemble Independence Regularization (SPQR). The authors incorporate SPQR into online and offline ensemble Q-learning algorithms. Experimental results show that SPQR surpasses baseline algorithms in both online and offline RL benchmarks, demonstrating its effectiveness in addressing overestimation bias and improving performance.", "questions": "Please find my questions in the weakness section.", "limitations": "No.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Significance and Originality: The problem and viewpoint that the paper studies - how to address overestimation bias/out-of-distribution error in RL is an important problem, and there have been a number of works investigating how to tackle this problem based on ensemble learning. However, most of the methods assume the bias follows a uniform and independent distribution, which may not hold in practice. This paper aims to improve previous ensemble methods from this perspective with a theoretical guarantee of improved Q-ensemble independence. The viewpoint from the random matrix theory seems novel, and the authors also propose a practical and tractable implementation for the method, which makes it possible to be applied in standard high-dimensional tasks.\n\n- Quality: The authors have done a comprehensive experiments to evaluate the methods in different setupds, and the proposed SPQR methods provides improvements in different aspects.\n\n- Clarity: The paper is also well-written and easy to follow.\n\n", "weaknesses": "- It is claimed in the paper that the assumption that most previous works rely on may not be true (the i.i.d. assumption about the bias). Could the authors demonstrate this effect is generally invalid in most of the environments? It would be better to convince the readers (since previous methods also have great performance although this assumption may not hold in practice).\n\n- Results in Figure 5 about the validation for the independence analysis are interesting. Does this generally hold in different tasks?\n\n- The improvement in standard D4RL locomotion tasks is somewhat marginal (4% improvement) considering increasing computation costs. \n\n- The results in Figure 6 about the performance of SAC-Ens is very low. Could authors better explain this result?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your constructive and thoughtful feedback. We address your questions and concerns as follows.\n\nQ1. Verification of the invalidation of the i.i.d assumption in various environments.\n\nA1. \nAs you mentioned, we evaluate the acceptance ratio in various environments and datasets.\nAccording to the table below, SAC-Min and EDAC demonstrate a lower acceptance ratio than SPQR for various environments and offline datasets.\nWe evaluate each algorithm on *-full-replay (OOD data) with chi-square hypothesis testing for independence, with a significant level \\alpha=0.025. \nFor example, an agent trained by hopper-random is evaluated with hopper-full-replay, as we have noted on line 245.\n\nTable: independence testing acceptance ratio per environment\n| algorithm | hopper-random | halfcheetah-random | walker2d-random |\n|-----------|---------------|--------------------|-----------------|\n| SAC-Min   | 30.4%         | 13.3%              | 34.0%           |\n| EDAC      | 30.4%         | 6.7%               | 0.0%            |\n| SPQR      | 80.4%         | 51.0%              | 60.0%           |\n\nTable: independence testing acceptance ratio per dataset\n| algorithm | halfcheetah-random | halfcheetah-medium | halfcheetah-expert |\n|-----------|--------------------|--------------------|--------------------|\n| SAC-Min   | 13.3%              | 23.3%              | 32.0%              |\n| SPQR      | 51.0%              | 70.0%              | 74.0%              |\n\n\nQ2. Explain computational cost and performance improvement.\n\nA2.\nThe computational cost can be analyzed in terms of two aspects: memory usage and training time per epoch.\nFrom a memory usage perspective, we have less memory than SAC-Min because SPQR uses a smaller number of ensembles, N (we have provided the specific values of N in Table 8) by the conservative ensemble property of SPQR as shown in Figure 3 and Figure 8. \nThis significantly reduces the training time per epoch since lower memory usage also has a significant impact on time consumption.\nAs mentioned in line 294, SAC-Min requires 500 networks for some tasks in a hopper environment, whereas SPQR only needs 50, making it **significantly more computationally efficient**.\nFurthermore, computing the regularization loss does not significantly increase computational cost in terms of training time per epoch, by increasing **only 5% with the same ensemble size**, as mentioned in line 239.\n\nTable: average ensemble size decreasing rate per environment\n| environments    | walker2d | halfcheetah | hopper |\n|-----------------|----------|-------------|--------|\n| decreasing rate | 9.9% (33.3->30)    | 35% (10->6.5)        | 79% (350->75)   |\n\nIn addition, building upon our previous comment in line 290, we note that the performance improvements are **significantly greater** in the case of low dataset quality. \nAlthough there was a remarkable improvement in the low-quality dataset (*-random), the average improvement rate seems somewhat marginal because SAC-Min has already shown good performance in the high-quality dataset, such as *-expert series. \nTherefore, we can conclude that our algorithm outperforms the baseline by a large margin, even if its average value appears to be small.\n\nAs we provide the performance gain by dataset quality below, we can consistently confirm that our method shows a higher performance improvement ratio in the worse-quality dataset. \n\nTable: performance gain per dataset quality\n| random | medium | expert | medium-expert | medium-replay | full-replay |\n|--------|--------|--------|---------------|---------------|-------------|\n| 16%    | 7.0%   | 2.3%   | 3.2%          | 7.3%          | 3.3%        |\n\nIn conclusion, we want to emphasize that applying SPQR becomes more computationally efficient since it needs fewer networks and shows a larger performance gain when the dataset quality is worse.\n\nQ3. Explain the performance of SAC-Ens.\n\nA3.\nWhen compared to other baseline ensemble methods, SAC-Ens appears to perform poorly. \nGiven that SAC-Ens calculates an average Q-value among N networks, it is highly likely that its performance is similar to vanilla SAC that uses a single network (ensemble), as shown in Figure 4 (learning curve of DQN, DDQN, and Average DQN) and Figure 8 (sensitivity analysis of DQN, DDQN, and Average DQN) of the Maximin paper[1].\nWe also note that the reported learning curve of SAC-Ens is close to the performance of vanilla SAC as shown in Figure 1 of the REDQ paper[2]. \n\nMore importantly, using the average target Q-value (SAC-Ens) underperforms the minimum target Q-value (REDQ(UTD=1), SAC-Min), as reported in Figure 3 and Figure 4 (learning curve of Average DQN and Maxmin DQN) of the Maximin paper[1].\nThis occurs because using the average target Q-value suffers more from the overestimation bias harshly compared to using a minimum target Q-value, as illustrated in Figure 10 in the Appendix of our paper.\n\n[1] Lan, Qingfeng, et al. \"Maxmin q-learning: Controlling the estimation bias of q-learning.\" arXiv preprint arXiv:2002.06487 (2020).\n\n[2] Chen, Xinyue, et al. \"Randomized ensembled double q-learning: Learning fast without a model.\" arXiv preprint arXiv:2101.05982 (2021).\n\nWe hope that our explanation and additional experiments may address much of your concern."}, {"review_id": "HCOkpK3yk5", "reviewer": "Reviewer_JH5V", "summary": "This work proposes a spiked Wishart Q-ensemble independence regularization (SPQR) to improve the independence of ensembling in Q-learning. SPQR encourages the ensemble to be closer to an ideal independent ensemble by penalizing the KL divergence between the eigenvalue distribution of the current ensemble and an ideal one.", "questions": "See weaknesses above.", "limitations": "While I do not see potential negative societal impact, a discussion on limitations is missing from the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to follow. The paper provides nice evidence of lack of independence in Q-ensemble training in current methods. The regularization to increase the diversity is well motivated, and seems simple and practical. The empirical evaluation is comprehensive, though more baselines would make them more compelling.", "weaknesses": "There is quite a bit of prior work on ensembling in deep RL, for example bootstrapped DQN [1] and MSG [2], which are not discussed. In particular, [2] provides theoretical and empirical support for constructing independent ensembles. A discussion and comparison with MSG is warranted given the emphasis on constructing independent ensembles.\n\nWhile the emphasis of work is on improving ensembling methods in deep RL, recent methods have been able to forego ensembling while maintaining or improving performance over REDQ, for example DroQ [3] or RLPD [4]. It would be good to contextualize the improvements from SPQR by adding comparisons with these methods.\n\nMore comparisons with offline RL methods would help too, for example, IQL.\n\n[1] Deep Exploration via Bootstrapped DQN. Osband et al. \n[2] Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters. Ghasemipour et al. NeurIPS, 2022.\n[3] Dropout Q-Functions for Doubly Efficient Reinforcement Learning. Hiraoka et al.\n[4] Efficient Online Reinforcement Learning with Offline Data. Ball et al.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your detailed feedback. We address your questions and concerns as follows.\n\nQ1. Comparison with other RL algorithms.\n\nA1.\nWe appreciate for informing us about meaningful prior studies. We will try our best to compare and evaluate each previous study based on their conceptual and empirical performance perspectives. \n\n1. bootstrapped DQN\n\nBootstrapped DQN aims to enhance deep exploration in the Atari environment by using a multi-head ensemble in deep online RL. \nSPQR aims to improve Q-ensemble independence in both online and offline RL. \nBootstrapped DQN proposed a multi-head architecture for ensemble Q-learning, which was outlined and evaluated as multi-head MSG in the MSG paper. \nIntuitively, from the Q-ensemble independence perspective, separate N ensemble networks are preferable to a shared multi-head architecture. \nMoreover, empirical evidence indicates that deep ensemble MSG outperforms multi-head MSG. \nAs bootstrapped DQN suggested, we will evaluate SPQR using a multi-head architecture in addition.\n\n2. MSG\n\nThe Model Standard-deviation Gradient (MSG), proposes a pessimistic ensemble Q-learning algorithm using the Upper Confidence Bound (UCB) of independent Q-networks, without relying on a shared target Q-value. \nSPQR aims to enhance independence by regularizing the network while maintaining a shared target Q-value. \nThe performance of MSG and SPQR in the Antmaze environment can be compared, as reported in MSG. \nHowever, it should be noted that MSG uses the *-v0 environment, which needs to be reproduced. \nSince MSG and SPQR are orthogonal methodologies, we can combine them as we did for SPQR-EDAC. \nMSG claims that a shared target Q-value leads to optimism. \nWe can modify the shared Q-target value of SPQR to an independent target Q-value or a multi-head target Q-value, as MSG has proposed. \n\n3. DroQ\n\nThe objective of DroQ is to enhance the computational efficiency of the ensemble Q-learning using the dropout technique. \nSince the performance of DroQ and REDQ is similar, we have thought that a comparison between REDQ and SPQR is sufficient. \nWe will reproduce DroQ and evaluate an empirical comparison with SPQR for a revision of our paper. \n\n4. RLPD\n\nThe RLPD proposes an ensemble Q-learning algorithm that performs well by using an offline data buffer during online RL training. \nBy combining RLPD with SPQR, performance can be improved as RLPD employs the orthogonal methodology of SPQR. \n\n5. IQL\n\nTable 9 in Appendix E.1 compares SPQR-CQL with IQL in the Franka Kitchen and Antmaze environments. \nSince CQL outperforms IQL in the D4RL Gym environment, comparing SPQR with CQL suffices for the Gym environment. \n\nWe hope that our explanation may address much of your concern."}, {"review_id": "YtDYKU9DCt", "reviewer": "Reviewer_BeK9", "summary": "This paper proposes a new regularization loss that improves the independence of Q ensemble, thus improving the performance on online and offline DRL settings. The authors first point out that previous works with Q ensemble either rely on assumptions that are inaccurate in practice, or rely on heuristics to improve diversity of networks and no theoretical support. The authors provide theoretical results that support the proposed method, and also conducted empirical experiments, where the proposed method is applied to multiple recent online and offline algorithms, and tested on the MuJoCo online and d4rl offline benchmarks. On a high level, the regularization (Spiked Wishart Q-ensemble independence reuglarization (SPQR)) encourages the Q ensemble distribution to be closer to an ideal independent ensemble, resulting in a more diversified Q prediction values, lower bias and better performance. ", "questions": "- Main suggestion: fix the figures and make them more readable. \n- Question: what is the limitation of your work? Are these cases where the proposed method might fail, or type of algorithms that it cannot be applied to? ", "limitations": "The proposed method seems to be quite general and can be applied to other methods based on Q ensembles. However can be good to have some discussion on limitations in the paper. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**originality**\n- The theoretical results and the proposed algorithm, the findings can be seen as novel\n- Learning with Q ensembles has been studied in many previous works, but as the authors pointed out, they either rely on assumptions that are inaccurate in practice, or rely on heuristics, so the novelty is quite good here. \n\n**quality**\n- Overall the presentation is good, writing is clear\n\n**Clarity**\n- The motivation, connection to related works are all quite clear, and ample technical details are given, the authors also explained that they try to be fair in the comparisons and use the same hyperparameters. And the authors explained computation time, performance improvement, implementation difficulty quite clearly. \n\n**significance**\n- the analysis is nice, which shows that the proposed method seems to achieve the desired improved independence. \n- empirical results showing consistent performance improvement over the baseline, big improvement in some settings, slight improvements in others, but overall quite consistent. Consistent improvement over recent sota algorithms such as REDQ with small change in the code and low computation cost and no excessive fine-tuning is quite impressive.  \n- the theoretical part is interesting and quite important ", "weaknesses": "I don't have very major concerns, one thing that can be fixed is the text in all your figures, especially those in the main paper are just too small, please make the fontsize of legend, as well as axis labels and ticks and any other text in the figures bigger so they are easier to read. \n\nMinor issue: \n- line 81: SPQR analysis -> analyzes ?\n- Figure 3 caption can you add a short sentence on what does beta do? It is not explained until a later section and I got confused when reading to Figure 3. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your interest in our research and for providing us with constructive feedback. \nWe address your questions and concerns as follows.\n\nQ1. Typos, grammatical errors, and font size mistakes.\n\nA1. \nThank you for kindly pointing out our typos, grammar, and font size mistakes.\nWe will take care of all the mistakes in the revision.\n\nQ2. More detailed caption for Figure 3. \n\nA2. \nThank you for providing a detailed and thoughtful comment to improve the presentation of our paper.\nDuring the rebuttal phase, it is not possible to modify our paper itself. \nTherefore, we provide more specific explanations for other readers as follows.\n\nTo help to understand the independence and diversity of Q-ensemble more clearly, we mentioned in line 176 that beta can be simply considered as a loss gain (weight) and presented Figure 3 before providing a detailed explanation about SPQR and beta. \nWe agree with your comment and will include the statement “beta is the loss weight for the regularization for Q-ensemble independence. Higher beta represents highly-likely-to-independent”  in the caption for Figure 3 in the revision.\n\n\nQ3. Limitation of our work. \n\nA3. \nWe might need a hyperparameter tuning on the regularization weight beta. However, we can broadly apply a common fixed beta in most environments since beta is not a sensitive hyperparameter as we have already shown in Table 8 and Table 11 in the appendix.\n\nWe hope that our explanation may address much of your concern."}, {"review_id": "PT7xeeJHee", "reviewer": "Reviewer_ve3n", "summary": "The paper deals with the problem of alleviating overestimation bias in RL, using ensembles Q-functions. The authors argue that previous methods do not provide a theoretical guarantee of the independence of the members of the ensemble. To provide this they propose an approach based on random matrix theory, which, in practice, can be implemented as a regularization loss. \n\nThe approach is evaluated in a number of settings for RL in tasks in MuJoCo, D4RL Gym, Franka Kitchen and Antmaze. ", "questions": "The meaning of Figure 3 is not clear. It seems that the higher the \\beta, the lower the average Q value. But at this point in the paper, the approach had not yet been described, so it is not clear what the \\beta controls. Also, we don't know the correct value of Q. I can make some assumptions that the figure aims to show how the proposed technique improves on the overestimation bias, but this (if true) would need to be explained more clear.", "limitations": "No separate limitations section found in the paper. \n\nThe theoretical nature of the paper does not raise ethical concerns. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "\n+ The approach has a strong, non-trivial theoretical foundation.\n+ Experimental results show a significant improvement over the baselines. ", "weaknesses": "- Throughout the paper, there are many grammatical errors and awkward formulations. Some of these might hinder understanding, such as on page 5: \n\"high correlation occurs, which cannot be benefited by the ensemble method\". I assume it this case what the authors mean is \"benefit the ensemble method\". \nMany of these could be easily fixed with a grammar checker. \n\n- The text on figures 5 and 6 (and to a lesser degree, figure 4) are unreadably small. Graphs should be redrawn in such a way that the text is still readable at a normal printing size. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the careful reading and interesting review. We address your questions and concerns as follows.\n\nQ1. Detailed explanation about Figure 3.\n\nA1. \nThank you for providing a detailed and thoughtful comment to improve the presentation of our paper.\nDuring the rebuttal phase, it is not possible to modify our paper itself. \nTherefore, we provide more specific explanations for other readers as follows.\n\nTo help to understand the independence and diversity of Q-ensemble more clearly, we mentioned in line 176 that beta can be simply considered as a loss gain (weight) and presented Figure 3 before providing a detailed explanation about SPQR and beta. \nWe agree with your comment and will include the statement “beta is the loss weight for the regularization for Q-ensemble independence. Higher beta represents highly-likely-to-independent”  in the caption for Figure 3 in the revision.\n\nSection 4.1 aims to demonstrate the relationship between independence and conservatism is related, thereby enabling the control of conservatism through independence regularization.\nIt does not directly address the alleviation of overestimation bias, which requires a true Q-value. \nThe main purpose of this plot is to demonstrate the order of Q-values in terms of beta level, indicating that a higher independence regularization weight leads to a more conservative algorithm. \nThe value and scale themselves are not our primary concerns in this section.\nTherefore, appropriate conservatism can be achieved by tuning beta, which prevents underestimation due to extreme conservatism.\n\nQ2. Typos, grammatical errors, and font size mistakes.\n\nA2. \nThank you for kindly pointing out our typos, grammar, and font size mistakes.\nWe will handle all of these mistakes in the revision.\n\nWe hope that our explanation may address much of your concern."}, {"review_id": "w154yDVMuG", "reviewer": "Reviewer_ppMG", "summary": "The paper proposes the SPQR loss to regularize the independence of the Q-network ensemble. The authors apply the random matrix theory and a spiked random model to derive the KL loss between Wigner’s semicircle distribution and the empirical spectral density of eigenvalues. The authors show that the independence could be enforced by minimizing this KL divergence loss with theoretical guarantees by satisfying the testing hypothesis. Extensive experiments on both the online and offline RL tasks and several baselines demonstrate that SPQR could improve the performance of the current ensemble RL algorithms promisingly by increasing of independence of the ensemble networks.", "questions": "1.\tWhat is the purpose of the use of spiked random model in SPQR? Could the authors provide some illustration examples or give more explanations?\n2.\tHow to construct the symmetric Q-matrix in Algorithm 1?\n3.\tWhy is the GOE chosen for the study and why should we relax the definition of GOE?\n", "limitations": "NA.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper is well-organized and well-written.\n2.\tThe proposed SPQR is with a simple format and shows a promising improvement over the ensemble RL methods such as SAC-ens and REDQ.\n3.\tThe authors build the SPQR loss with mathematical tools and provide the theistical guarantee to ensure the independence of the ensembled Q-networks.\n4.\tExperiments on both the online and offline RL tasks are sufficient to validate the generality of the proposed SPQR and that it can improve the performance of ensemble RL methods.\n", "weaknesses": "1.\tDetailed explanations between the used math tools and the proposed method are missing, making the understanding hard for the readers. For example, the connection between the random matrix theory and a spiked random model is not explained well. What is the purpose of the use of a spiked random model in SPQR?\n2.\tThere lack details of in Algorithm 1. For example, how to construct the symmetric Q-matrix is not shown. The detailed version of the core algorithm is given in the appendix, which is strange.\n3.\tThe quality of the figures needs to be improved. For example, the text size is too small.\n4.\tThere are many typos.\n\na)\tIn Line 73, “tp” should be “to”.\n\nb)\tIn Line 81, “SPQR analysis Q-ensemble independence …”.\n\nc)\tIn Line 113, “Q-learning, We plot”.\n\nd)\tIn Line 206, “Proof of Theorem 4.1 are …”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your considerate feedback and emphasis on improving our work. We address your questions and concerns as follows.\n\nQ1. Explanations about the spiked random model and GOE.\n\nA1.\n\n**1. spiked random model**\n\nThe goal of the spiked random model is to detect/recover whether the given data is pure noise (random) or has a signal with noise (informative). \nAs we visualize in Figure 2, the pure random matrix and informative matrix show different behavior in the eigenvalue domain. \n\nBefore explaining the details, we can simply consider the spiked random model as the addition of an informative signal and the noise by the random matrix. \nIn Figure 2, we generate a random matrix where the diversity of the elements in the 1st and the 3rd subfigure is similar but all elements of the random matrix in the 3rd figure are slightly moved towards (1,1)-direction. \nThis difference is a clear difference between pure randomness and spiked randomness. \n\nSimilar to PCA, the most natural assumption for the presence of information in the data is that the information can be captured in the most significant eigenvalue. \nTo analyze the eigenvalue behavior of data, we need the random matrix theory. \nFor example, in Right 3-4 in Figure 2, some signal, u,v exists and in the eigenvalue domain, it is detected by the top eigenvalue (rightmost) when the other eigenvalues follow Wigner’s semicircle law based on random matrix theory. \nOn the other hand, in Left 1-2 in Figure 2, the elements of the random matrix show pure randomness without any signal, so its whole eigenvalues are in the Wigner’s semicircle. \n\nOur approach is to determine and alleviate non-independence among the Q-ensemble.\nInitially, we adapt the spike random model for SPQR to determine whether these data are collected independently. \nIn order to make the Q-ensemble independent, we optimize the test criterion of the spiked random model. \nThe test criterion has a form of KL loss between the eigenvalue distribution of observed data and pure random matrix, which is determined using random matrix theory.\n\n**2. GOE**\n\nIn many machine learning and deep learning literature, GOE is used as the most natural way to model and analyze general random data matrix, such as analyzing loss surface [1], representation of GAN data [2], optimization acceleration [3], and explorations [4]. \nWe choose GOE to model the behavior of an independent ensemble. \n\n**3. relax the definition of GOE**\n\nThank you for pointing out the sloppy statement. \nThe phrase ‘relaxed definition’ on line 141 should have been written as ‘simplified terms’.\nFor convenience, we intend to give a friendly definition to make it easier for GOE to understand. \nActually, the type of distribution (Gaussian, Rademacher, Poisson, etc..) of entities does not affect the eigenvalue distribution (of course Wigner’s law) of the random matrix, which is called ‘universality’ in the random matrix theory literature. as we have noted in line 205. \n\n[1] Baskerville, Nicholas P., et al. \"The loss surfaces of neural networks with general activation functions.\" Journal of Statistical Mechanics: Theory and Experiment 2021.6 (2021): 064001.\n\n[2] Seddik, Mohamed El Amine, et al. \"Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures.\" International Conference on Machine Learning. PMLR, 2020.\n\n[3] Lacotte, Jonathan, and Mert Pilanci. \"Optimal randomized first-order methods for least-squares problems.\" International Conference on Machine Learning. PMLR, 2020.\n\n[4] Sagun, Levent, et al. \"Explorations on high dimensional landscapes.\" arXiv preprint arXiv:1412.6615 (2014).\n\nQ2. Explanation about how to construct the symmetric Q-matrix.\n\nA2. \nIt was really hard to include the full version of the algorithm because of the page limitation.  However, we agree with your comment. \nIn the revised version, we will provide a more detailed explanation in the main text too. \n\nOur goal is to generate a symmetric Q-value matrix filled with an ensemble size of N.\nTo avoid training or capturing the order of a matrix, we shuffle the filling order. \nThis can be achieved by sorting the list of Q-networks l_k, as specified in Algorithm 3.\nIn order to create a symmetric matrix of maximum size, the upper triangle part and the corresponding lower triangle part are equally filled. \nThe size of the matrix is set as D=floor((sqrt(1+8*N)-1)/2), as described in Algorithm 3.\nD can be found by finding the largest integer that satisfies (D**2+D)/2 <= N. \n\nPlease refer to the illustration in the **attached pdf files of the global response** for a clear understanding of the process. \n\nQ3. Typos, grammatical errors, and font size mistakes.\n\nA3. \nThank you for kindly pointing out our typos, grammar, and font size mistakes.\nWe will handle all of these mistakes in the revision. \n\nWe hope that our explanation may address much of your concern."}]

Author rebuttal:
For reviewer ppMG, we attach a pdf file for illustration about constructing a symmetric matrix here.

Meta-review: This paper has been initially positively evaluated by reviewers, and the rebuttal helped to solve concerns about the empirical validation of the proposed method, which resulted in some reviewers increasing their scores. Overall, there is a clear consensus for accepting this paper.

I encourage the authors to address all the comments and to incorporate the recommended improvements in the final version.
Decision: Accept (poster)