SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dohyeok Lee $^{1}$ , Seungyub Han $^{1,2}$ , Taehyun Cho $^{1}$ , Jungwoo Lee $^{1,*}$
$^{1}$ Seoul National University,  $^{2}$ Hodoo AI Labs
{dohyeoklee,seungyubhan,talium,junglee}@snu.ac.kr
Alleviating overestimation bias is a critical challenge for deep reinforcement learning to achieve successful performance on more complex tasks or offline datasets containing out-of-distribution data. In order to overcome overestimation bias, ensemble methods for Q-learning have been investigated to exploit the diversity of multiple Q-functions. Since network initialization has been the predominant approach to promote diversity in Q-functions, heuristically designed diversity injection methods have been studied in the literature. However, previous studies have not attempted to approach guaranteed independence over an ensemble from a theoretical perspective. By introducing a novel regularization loss for  $Q$ -ensemble independence based on random matrix theory, we propose spiked Wishart Q-ensemble independence regularization (SPQR) for reinforcement learning. Specifically, we modify the intractable hypothesis testing criterion for the Q-ensemble independence into a tractable KL divergence between the spectral distribution of the Q-ensemble and the target Wigner's semicircle distribution. We implement SPQR in several online and offline ensemble Q-learning algorithms. In the experiments, SPQR outperforms the baseline algorithms in both online and offline RL benchmarks.
Reinforcement learning (RL), especially deep reinforcement learning (DRL) that contains high capacity function approximators such as deep neural networks, has shown considerable success in complex sequential decision making problems including robotics [1], video games [28], and strategy games [12]. Despite these promising results, modern DRL algorithms consistently suffer from sample-inefficiency and overestimation bias.
Vanilla off-policy Q-learning faces the overestimation issue in both online and offline reinforcement learning. The overestimation problem is specifically worse for out-of-distribution (OOD) state-action pairs. In order to address the overestimation problem, many existing methods utilize ensemble Q-learning. Ensemble Q-learning introduces multiple Q-networks to compute a Bellman target value to benefit from the diversity of their Q-values. Double Q-learning [15] proposes clipped Q-learning as the first method to adopt an ensemble to DRL. Along with double Q-learning, Ensemble DQN [3], Maxmin Q-learning [22], randomized ensembled double Q-learning (REDQ) [6], and SUNRISE [24] are proposed as efficient ensemble Q-learning algorithms on online RL tasks. Clipped SAC [2] is proposed for offline RL tasks to alleviate the overestimation bias of Q-values from OOD data. Despite the objective of ensemble Q-learning, which enables RL agents to access diverse Q-values for each state-action, existing ensemble Q-learning algorithms have only focused on the computing Bellman target over an ensemble. MED-RL [31] and EDAC [2] propose a diversifying loss for online and offline reinforcement learning, such as the Gini coefficient and the variance of the Q-value on
OOD data. However, these methods have a limitation in that the proposed losses do not guarantee independence from a theoretical perspective, but diversify Q-functions from heuristics and empirical studies although they have an i.i.d assumption. Most ensemble Q-learning algorithms assume that bias follows a uniform and independent distribution, which may not be true when using deep neural network approximation and a shared Bellman target. Hence we need to reinvestigate the notion of  $Q$ -ensemble independence more rigorously through a random matrix theory. The random matrix theory studies the statistical behavior of a random matrix and has a wide range of applications in diverse fields such as physics, engineering, economics, and machine learning as shown in [5, 25, 29].
In this paper, we propose spiked Wishart Q-ensemble independence regularization (SPQR), a tractable Q-ensemble independence regularization method based on the random matrix theory and a spiked random model perspective. SPQR is a regularization method that theoretically guarantees to improve the independence of a Q-ensemble during training. SPQR penalizes the KL divergence between the eigenvalue distribution of the Q-ensemble and an ideal independent ensemble. If a Q-ensemble becomes closer to the ideal independent ensemble, it satisfies the independence assumption of ensemble Q-learning. To show how effective SPQR is, we combine SPQR with various algorithms and online/offline RL tasks. SPQR outperforms baseline algorithms in MuJoCo, D4RL Gym, Franka Kitchen, and Antmaze. To summarize, our contributions are as follows:
1. We propose a tractable loss function for Q-ensemble independence without any prior assumption of Q-function distribution by adopting random matrix theory and a spiked random model for the first time in DRL algorithms.
2. It is empirically shown that the universality and performance gain of SPQR for various algorithms and tasks. SPQR is applied to SAC-Ens, REDQ, SAC-Min, CQL, and EDAC and evaluated in various complex and sparse tasks in online and offline RL, MuJoCo, D4RL Gym, Franka Kitchen, and Antmaze. To ensure our reproducibility and fair comparisons, we prodive our source code. $^{1}$
Ensemble Reinforcement Learning. Ensemble DQN [3] is the first method to adopt ensemble Q-learning for DRL. Ensemble DQN aims to reduce overestimation bias by computing a Bellman target as an average Q-value in the ensemble. Maxmin Q-learning [22] computes a Bellman target as a minimum Q-value over the Q-ensemble and shows significant performance enhancement based on the Atari benchmark. REDQ [6] constructs sample efficient ensemble Q-learning algorithm based on Maxmin Q-learning and SAC with a probabilistic ensemble by selecting a subset over Q-ensemble. Since REDQ combines the update-to-data (UTD) ratio, which is an orthogonal method with ensemble subset minimization, REDQ achieves state-of-the-art performance in the online MuJoCo environment. For offline reinforcement learning tasks, clipped Q-learning [2], denoted as SAC-Min is proposed, which computes the Bellman target as in Maxmin Q-learning. Diversification methods are proposed to improve the performance of ensemble Q-learning. MED-RL [31] optimizes the Gini coefficient to diversify an ensemble, and EDAC [2] optimizes the variance of the Q-value on OOD data to diversify an ensemble from the OOD data. SPQR proposes a Q-ensemble independence loss from a theoretical perspective, based on a spiked random model and random matrix theory to make ensemble Q-networks as independent as possible. In addition, conservative Q-learning (CQL) is proposed by [21], which penalizes the Q-values of OOD actions and performs well on numerous offline datasets.
Spiked Random Model. Signal recovery for a generative model using a spiked random model is analyzed by [8]. SPQR analyzes Q-ensemble independence for ensemble Q-learning in a spiked random model perspective. A universal data-driven optimal likelihood hypothesis test with linear spectral statistics (LSS) for a spiked random model as a central limit theorem for test statistics is given by [7, 18]. SPQR proposes a tractable universal data-driven test  $T(\lambda)$  to regularize the Q-ensemble independence.
$^{1}$ Code implementation is available at: https://github.com/dochyeoklee/SPQR
Figure 1: Left 1-4: Histogram visualization of Q-value distribution over networks in the ensemble. Each plot represents a Q-value histogram for one state-action data. The X-axis represents the Q-value of each Q-network and the Y-axis represents the number of Q-networks in the histogram bin. The red horizontal line represents a uniform distribution and the blue solid line represents kernel density estimation for a given histogram. Rightmost: Heatmap visualization of the Pearson correlation coefficient matrix between each Q-network in the ensemble. Detailed values and explanations are given in Appendix E.
In this section, we describe ensemble Q-learning algorithms and visualize an empirical study of their limitations. In order to overcome its limitations, we introduce the random matrix theory and a spiked random model to formulate the notion of Q-ensemble independence from a theoretical perspective.
We consider a Markov decision process (MDP), defined by a tuple  $(S, \mathcal{A}, P, r, \gamma)$ , where  $\mathcal{S}$  is a finite state space,  $\mathcal{A}$  is a finite action space,  $P: S \times \mathcal{A} \times S \to [0,1]$  is the state-action transition probability,  $r: S \times \mathcal{A} \to \mathbb{R}$  is the reward function, and  $\gamma \in [0,1)$  is the discount factor. We define a state-action value function, called Q-function,  $Q_{\phi}: S \times \mathcal{A} \to \mathbb{R}$ , represents an expectation of the discounted sum of rewards for the trajectory started from  $s, a$ , given by  $\mathbb{E}_{s_t, a_t}[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t)]$ .
3.1 Ensemble Q-learning
We first define bias  $e_{sa}^{i} = Q^{*}(s,a) - Q_{i}(s,a)$ , where  $Q^{*}(s,a)$  is the optimal Q-value given for  $(s,a)$  and  $Q_{i}(s,a)$  is the Q-value of the  $i$ -th networks in ensemble. Ensemble Q-learning assumes that the bias follows a zero mean, uniform, and independent distribution, i.e.,  $e_{sa}^{i} \sim Unif(-\tau, \tau)$  and is independent with respect to  $i$ , although REDQ does not assume uniform distribution. Given  $N$  Q-functions in the ensemble, there are three typical methods to compute a Bellman target: Ensemble DQN, Maxmin Q-learning, and REDQ.
Ensemble DQN. Ensemble DQN computes the Bellman target as the ensemble average Q-value. The bias and variance of the bias monotonically decrease as the ensemble size  $N$  increases, and they become 0 as  $N$  goes to infinity. We implement the SAC-style of Ensemble DQN, denoted as SAC-Ens.
Maxmin Q-learning. The Bellman target of Maxmin is to select the minimum Q-value within an ensemble, which can control the level of overestimation with a finite number of Q-functions. A modified version called SAC-Min is introduced specifically for offline RL tasks.
REDQ. REDQ computes the Bellman target as the minimum Q-value in a subset of the ensemble to become a sample-efficient algorithm.
Detailed explanations for Bellman targets and the evaluation function of each algorithm, denoted as  $Ens_{tar}$  and  $Ens_{eval}$ , are described in Appendix C.
To confirm the validity of uniform and independent assumptions of ensemble Q-learning, we plot Q-value distribution in Figure 1 and test whether each Q-distribution is uniform using the  $\chi^2$  test. We train 50 Q-networks 3M timesteps by the SAC-Min algorithm in the D4RL hopper-random dataset and evaluate it by the hopper-full-replay dataset. We also perform the  $\chi^2$  test for independence in Table 1. Out of 1000 batch data, only  $35.8\%$  can accept that the distribution is uniform and  $30.4\%$  can accept independence. Thus,  $e_{sa}^{i}\sim Unif(-\tau ,\tau)$  and i.i.d. assumption is empirically inaccurate and it is necessary to introduce the notion of Q-ensemble independence. More details on the  $\chi^2$  test are given in Appendix E.
Figure 2: Eigenvalue plot for the spiked Wishart model with Wigner's semicircle law. For effective visualization of the spiked model, we use a complex hermitian random matrix, GUE. Blue dots represent each data in a complex plane. The red line represents Wigner's semicircle distribution. The blue line in the histogram represents kernel density estimation. Left 1-2: Perturbation power is  $\psi = 0$ . Eigenvalue distribution follows Wigner's semicircle law. Right 3-4: Perturbation power is  $\psi = 10^{-5}$ . Eigenvalue distribution almost follows Wigner's semicircle law except for the largest eigenvalue, interpreted as a spike.
As [30] also points out, the early collapse of Q-values, which means that the Q-values in the ensemble become almost identical, occurs during the training. We empirically visualize a strong correlation among Q-networks in Figure 1. Although the main advantage of ensemble Q-learning is based on diverse Q-functions, network initialization is the only process to maintain diversity over an ensemble since the Bellman target is identical across all networks without diversification and independence regularization methods.
3.2 Random Matrix Theory
Random matrix theory studies the probabilistic properties of large matrices with random entries. Dyson [10] constructs some categories of the random matrix as a real symmetric matrix as Gaussian orthogonal ensemble (GOE), complex hermitian matrix as Gaussian unitary ensemble (GUE), and quaternion self-dual matrix as Gaussian symplectic ensemble (GSE), called Dyson's classification. In this paper, we restrict our interest to matrices in GOE.
Definition 3.1 (Gaussian Orthogonal Ensemble, [26]). Let random matrix  $W \in R^{N \times N}$ , its entities random variables  $w_{i}, P[W] = P(w_{1}, \ldots, w_{N^{2}})$  to be a joint probability density function of  $w_{i}$ . Define the set of  $W$  as GOE when these properties hold:
(1)  $w_{i}$  are independent with respect to  $i$
(2)  $W$  is a real symmetric matrix.
(3)  $W$  is rotational invariance, i.e.,  $P[W]dW = P[W']dW'$  if  $W' = UWU^{-1}$ ,  $U$  is orthogonal.
GOE is the most natural way to model random matrices in various applications, including machine learning. In simplified terms, we can consider GOE as a Wigner matrix, where entries are independent, symmetric, and follow Gaussian distribution. One of the main areas in random matrix theory is how eigenvalues or singular values of a large random matrix are distributed. More specifically, we only consider limiting properties of empirical spectral density (ESD), which is defined as, for random matrix  $W \in \mathbb{R}^{N \times N}$  and its eigenvalues  $\lambda_{i}, i = 1, \ldots, N$ ,
$$
p _ {W} (\lambda) = \frac {1}{N} \sum_ {i = 1} ^ {N} \delta \left(\lambda - \lambda_ {i}\right) \tag {1}
$$
The limits of the empirical spectral density of the GOE, called the Wigner's semicircle law, are given by [34]. For GOE  $\hat{W} \in \mathbb{R}^{N \times N}$  with variance of entries  $\sigma^2$ ,  $\lambda_1, \ldots, \lambda_N$  to be a eigenvalue of  $W = \frac{1}{\sqrt{N}} \hat{W}$  and empirical spectral distribution  $p_W = \frac{1}{N} \sum_{j=1}^{N} \delta(\lambda - \lambda_j)$ , then  $p_W$  converges to  $p_{sc}$  weakly almost surely, where
$$
p _ {s c} (\lambda) = \frac {1}{2 \pi \sigma^ {2}} \sqrt {4 \sigma^ {2} - \lambda^ {2}} \mathbb {1} _ {[ - 2 \sigma , 2 \sigma ]} \tag {2}
$$
We call  $p_{sc}$  as the Wigner's semicircle distribution. Wigner's semicircle law does not rely on the probability density of entries.
3.3 Spiked Random Model
Random matrix theory models random data well, but in reality, most data have both informative signals and random components. A spiked random model is a theory about constructing a model of perturbation with a data matrix. The objective of this theory is to separate true signals from perturbed ones with random noise. True signals show spiked phenomena that make the largest eigenvalue detach from the bulk of eigenvalues. A spiked model was first introduced by [17] as an application for principal component analysis. We restrict our analysis to a spiked Wishart model, the most natural methodology to model a general perturbation. A spiked Wishart model is defined as follows.
Definition 3.2. Let  $Y \in \mathbb{R}^{N \times M} \sim \mathbb{P}_{\psi}, u, v$  is unknown vector,  $Y$  is spiked Wishart model when
(1)  $Y = \sqrt{\frac{\psi}{N}} uv^T + W$ , which is equivalent to sample covariance matrix  $\hat{\Sigma} = I_N + \frac{\psi}{N} uu^T$
(2)  $u, v$  are i.i.d from zero mean priors  $u \sim P_u, v \sim P_v$ , support bounded in radius by  $K_u, K_v$
(3)  $W\sim \mathbb{P}_0$  is a matrix with i.i.d noise entries
(4)  $M / N\to \omega$
We will assume the spiked Wishart model with  $N = M$ , where we can ignore property 4. In Figure 2, we visualize how the spiked Wishart model works in both the data and spectral domain. For a detailed explanation, see Appendix A.
4 Q-ensemble Independence Regularization
In this section, we present a method of Q-ensemble independence regularization that is computationally feasible through the utilization of a spiked Wishart model. We bring the concept of ensemble Q-learning to the spiked Wishart model as  $Y \sim \mathbb{P}_{\psi}$  representing learned Q-value by Q-learning algorithm and  $W \sim \mathbb{P}_0$  representing the perfectly independent Q-ensemble. If  $W$  dominates  $Y$ ,  $Y$  follows independent assumptions of ensemble Q-learning. If not, a collapse of Q-value and high correlation occurs, which cannot be benefited by the ensemble method. By integrating with ensemble Q-learning and Q-ensemble independence testing, we propose a tractable Q-ensemble independence loss with gain  $\beta$ , which is further described in the following section. To help understand the independence and diversity of Q-ensemble more clearly, we mentioned that  $\beta$  can be simply considered as a loss gain (weight) for Q-ensemble independence regularization before providing a detailed explanation about SPQR and  $\beta$ . A higher  $\beta$  represents highly-likely-to-independent.
4.1 Diversity and Independence of Q-ensemble
To clarify our work, we distinguish between the notions of diversity and independence. Numerous ensemble Q-learning methods improve their performance by minimizing the overestimation bias and the variance of the Bellman target estimate. However, such works assume that the biases  $e_{sa}^{i}$  are zero-mean independent and identically distributed over  $i$ , although early collapse, a high correlation between Q-ensemble, occurs due to the shared Bellman target. On the other hand, diversification methods such as EDAC develop an ensemble diversification objective to reduce the number of ensemble networks required. In particular, EDAC increases the variance of the ensemble to penalize OOD actions. In our work, we focus on the independence of a Q-ensemble to reduce the effect of correlated behavior resulting from a shared Bellman target. Roughly speaking, diversification can be interpreted as flattening the distribution of a Q-ensemble, while independence regularization can be interpreted as promoting
Figure 3: Mean of predicted Q-value of SPQR-SAC-Min with various  $\beta$  on hopper-random dataset, averaged over 4 seeds.
an i.i.d. sampling procedure for a given distribution. We verify that our proposed regularization successfully exploits both the conservative behavior and the diversification of a Q-ensemble. The Q-ensemble independence regularization empirically shows a large performance gain by controlling the conservatism, as shown in Figure 3, similar to the performance improvement of the ensemble method caused by the conservatism. For further details, see Appendix E.
4.2 Tractable Loss for Q-ensemble Independence
We construct a tractable testing criterion to determine whether a distribution is independent in Theorem 4.1.  $D_{KL}(p_{\psi}(\lambda),p_{sc}(\lambda))$  does not require any prior assumption about  $\mathbb{P}_0$  and  $\mathbb{P}_{\psi}$  since in spectral domain, Wigner's semicircle law is universal for all random matrices. Proof of Theorem 4.1 is shown in Appendix B.
Theorem 4.1. Following definition 3.2, as  $N \to \infty$  with probability at least  $1 - \delta$ , following test  $T(\lambda)$  is optimal.
$$
T (\lambda) = \mathbb {1} (D _ {K L} (p _ {\psi} (\lambda), p _ {s c} (\lambda)) \geq \varepsilon) = \left\{ \begin{array}{l} H _ {0}: Y \sim \mathbb {P} _ {0} \\ H _ {1}: Y \sim \mathbb {P} _ {\psi} \end{array} \right.
$$
We now describe a practical Q-ensemble independence loss for ensemble Q-learning using the spiked Wishart model and Theorem 4.1. Let  $Y$  and  $u, v$  for the spiked Wishart model denote a Q-value matrix and an unknown signal. By the test  $T(\lambda)$ , if  $D_{KL}(p_{\psi}(\lambda), p_{sc}(\lambda)) < \varepsilon$  or is minimized, it is theoretically guaranteed that  $Y$  is a random matrix, i.e. a Q-ensemble is independently distributed.
In conclusion, we propose  $D_{KL}(p_{\psi}(\lambda), p_{sc}(\lambda))$  as a Q-ensemble independence regularization loss for ensemble Q-learning, without any prior assumption such as uniform distribution.
As described in Section 3.1, ensemble Q-learning has crucial but inaccurate assumptions about independence. To address the issue of insufficient independence, we propose SPQR: spiked Wishart Q-ensemble independence regularization for reinforcement learning, which we describe in Algorithm 1. Using the random matrix theory and a spiked random model lens, SPQR minimizes  $D_{KL}(p_{\psi}(\lambda), p_{sc}(\lambda))$  as a form of regularization loss, denoted as SPQR loss, on the baseline ensemble Q-learning algorithms.  $D_{KL}(p_{\psi}(\lambda), p_{sc}(\lambda))$  penalizes for going too far from an independent Q-ensemble, which means promoting Q-ensemble to follow  $\mathbb{P}_0$ . We visualize our concept in Figure 4. To compute the SPQR loss using  $p_{\psi}(\lambda)$ , we need to construct a symmetric Q-matrix  $Y$  filled with Q-values. Then, we normalize the Q-matrix  $Y$  with a zero-mean and unit variance to compute the Wigner's semicircle distribution  $p_{sc}(\lambda)$  with  $\sigma = 1$ . Finally, we compute eigenvalues with a numerically stable algorithm by constructing the ESD  $p_{\psi}(\lambda)$  and calculate the KL divergence between  $p_{\psi}(\lambda)$  and  $p_{sc}(\lambda)$ . The loss function  $\mathcal{L}_{SPQR}$  is computed as a expectation of  $D_{KL}(p_{\psi}(\lambda), p_{sc}(\lambda))$  over the batch data. A detailed explanation of implementations and algorithms is shown in Appendix C.
Algorithm 1 SPQR
1: Input:  $\gamma, \alpha, N, \beta, \tau, |B|$
2: Initialize policy  $\theta$ , Q-function  $\phi_i$ , target Q-function  $\phi_i'$ , replay buffer  $\mathcal{D}$
3: repeat
4:  $y(r,s^{\prime})\gets r + \gamma \Bigg(Ens_{tar}(\phi_{i}^{\prime}) - \alpha \log \pi_{\theta}(a^{\prime}|s^{\prime})\Bigg)$
5: for  $i = 1,\dots ,N$  do
6:  $\mathcal{L}_{SPQR} = D_{KL}(p_{\psi}(\lambda), p_{sc}(\lambda))$
7:  $\nabla_{\phi_i}\frac{1}{|B|}\sum (Q_{\phi_i}(s,a) - y(r,s'))^2 +\beta \mathcal{L}_{SPQR}$
8: end for
9:  $\nabla_{\theta}\frac{1}{|B|}\sum_{s\in B}\left(Ens_{eval}(\phi_i) - \alpha \log \pi_{\theta}(\tilde{a} |s)\right)$
10: until
Algorithm 2 SPQR Loss
1: Input: Q-network  $l_{k} = \{Q_{\phi_{i}}(s^{\prime},a^{\prime})\}_{i = 1}^{N}$
2:  $Y\in \mathbb{R}^{D\times D}$ $D = \lfloor \frac{\sqrt{1 + 8N} - 1}{2}\rfloor$
3:  $Y_{pq} \gets \left\{ \begin{array}{ll} l_{Dp + q}, & \text{if } p \geq q \\ Y_{qp}, & \text{otherwise} \end{array} \right.$
4:  $Y \gets (Y - \mu_Y) / \sigma_Y$
5:  $\{\lambda_i\}_{i = 1}^N\gets E i g e n(\frac{1}{\sqrt{N}} Y)$
6:  $p_{esd}(\lambda) = \frac{1}{N}\sum_{i}\delta (\lambda -\lambda_{i})$
7:  $p_{wigner}(\lambda) = \frac{\sqrt{4 - \lambda^2}}{2\pi}$
8:  $\mathcal{L}_{SPQR} \gets \frac{1}{|B|} \sum_{j} \sum_{i} p_{esd}(\lambda_i) \log \frac{p_{esd}(\lambda_i)}{p_{wigner}(\lambda_i)}$
9: Output: SPQR loss  $\mathcal{L}_{SPQR}$
Figure 4: Illustrative example of SPQR.
Figure 5: Illustration for building Q-matrix.
To reveal the validity and universality of SPQR from algorithms to tasks, we implement SPQR in various ensemble Q-learning algorithms and various online/offline RL tasks. We implement SPQR in SAC-Ens, REDQ, SAC-Min, and CQL, and evaluate them in online MuJoCo Gym tasks, offline D4RL Gym, Franka Kitchen, and Antmaze tasks. We denote the SPQR-implemented version of each ensemble algorithm as SPQR- $*$ , such as SPQR-SAC-Ens. All the following implementations use the same hyperparameter and experimental protocol with a baseline. For fair evaluation, the SPQR code is written on top of each official baseline code, REDQ $^1$ , EDAC $^2$ , and CQL $^3$ . Implementation of SPQR can be done by only 50 lines and training time only increases by  $5\%$ . We empirically show that SPQR outperforms baseline ensemble Q-learning algorithms in both online and offline RL tasks. See the implementation details in Appendix D.