Title: SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Reinforcement Learning

Ensemble Q-learning
Abstract: Alleviating overestimation bias is a critical challenge for deep reinforcement learning to achieve successful performance on more complex tasks or offline datasets containing out-of-distribution data. 
In order to overcome overestimation bias, ensemble methods for Q-learning have been investigated to exploit the diversity of multiple Q-functions. 
Since network initialization has been the predominant approach to promote diversity in Q-functions, heuristically designed diversity injection methods have been studied in the literature. 
However, previous studies have not attempted to approach guaranteed independence over an ensemble from a theoretical perspective. 
By introducing a novel regularization loss for Q-ensemble independence based on random matrix theory, we propose spiked Wishart Q-ensemble independence regularization (SPQR) for reinforcement learning. 
Specifically, we modify the intractable hypothesis testing criterion for the Q-ensemble independence into a tractable KL divergence between the spectral distribution of the Q-ensemble and the target Wigner's semicircle distribution. 
We implement SPQR in several online and offline ensemble Q-learning algorithms. 
In the experiments, SPQR outperforms the baseline algorithms in both online and offline RL benchmarks.

SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning

Abstract

Alleviating overestimation bias is a critical challenge for deep reinforcement learning to achieve successful performance on more complex tasks or offline datasets containing out-of-distribution data. In order to overcome overestimation bias, ensemble methods for Q-learning have been investigated to exploit the diversity of multiple Q-functions. Since network initialization has been the predominant approach to promote diversity in Q-functions, heuristically designed diversity injection methods have been studied in the literature. However, previous studies have not attempted to approach guaranteed independence over an ensemble from a theoretical perspective. By introducing a novel regularization loss for Q-ensemble independence based on random matrix theory, we propose spiked Wishart Q-ensemble independence regularization (SPQR) for reinforcement learning. Specifically, we modify the intractable hypothesis testing criterion for the Q-ensemble independence into a tractable KL divergence between the spectral distribution of the Q-ensemble and the target Wigner's semicircle distribution. We implement SPQR in several online and offline ensemble Q-learning algorithms. In the experiments, SPQR outperforms the baseline algorithms in both online and offline RL benchmarks.

1 Introduction

Reinforcement learning (RL), especially deep reinforcement learning (DRL) that contains high capacity function approximators such as deep neural networks, has shown considerable success in complex sequential decision making problems including robotics [1], video games [28], and strategy games [12]. Despite these promising results, modern DRL algorithms consistently suffer from sample-inefficiency and overestimation bias.

Vanilla off-policy Q-learning faces the overestimation issue in both online and offline reinforcement learning. The overestimation problem is specifically worse for out-of-distribution (OOD) state-action pairs. In order to address the overestimation problem, many existing methods utilize ensemble Q-learning. Ensemble Q-learning introduces multiple Q-networks to compute a Bellman target value to benefit from the diversity of their Q-values. Double Q-learning [15] proposes clipped Q-learning as the first method to adopt an ensemble to DRL. Along with double Q-learning, Ensemble DQN [3], Maxmin Q-learning [22], randomized ensembled double Q-learning (REDQ) [6], and SUNRISE [24] are proposed as efficient ensemble Q-learning algorithms on online RL tasks. Clipped SAC [2] is proposed for offline RL tasks to alleviate the overestimation bias of Q-values from OOD data. Despite the objective of ensemble Q-learning, which enables RL agents to access diverse Q-values for each state-action, existing ensemble Q-learning algorithms have only focused on the computing Bellman target over an ensemble. MED-RL [31] and EDAC [2] propose a diversifying loss for online and offline reinforcement learning, such as the Gini coefficient and the variance of the Q-value on

OOD data. However, these methods have a limitation in that the proposed losses do not guarantee independence from a theoretical perspective, but diversify Q-functions from heuristics and empirical studies although they have an i.i.d assumption. Most ensemble Q-learning algorithms assume that bias follows a uniform and independent distribution, which may not be true when using deep neural network approximation and a shared Bellman target. Hence we need to reinvestigate the notion of Q-ensemble independence more rigorously through a random matrix theory. The random matrix theory studies the statistical behavior of a random matrix and has a wide range of applications in diverse fields such as physics, engineering, economics, and machine learning as shown in [5, 25, 29].

In this paper, we propose spiked Wishart Q-ensemble independence regularization (SPQR), a tractable Q-ensemble independence regularization method based on the random matrix theory and a spiked random model perspective. SPQR is a regularization method that theoretically guarantees to improve the independence of a Q-ensemble during training. SPQR penalizes the KL divergence between the eigenvalue distribution of the Q-ensemble and an ideal independent ensemble. If a Q-ensemble becomes closer to the ideal independent ensemble, it satisfies the independence assumption of ensemble Q-learning. To show how effective SPQR is, we combine SPQR with various algorithms and online/offline RL tasks. SPQR outperforms baseline algorithms in MuJoCo, D4RL Gym, Franka Kitchen, and Antmaze. To summarize, our contributions are as follows:

We propose a tractable loss function for Q-ensemble independence without any prior assumption of Q-function distribution by adopting random matrix theory and a spiked random model for the first time in DRL algorithms.

It is empirically shown that the universality and performance gain of SPQR for various algorithms and tasks. SPQR is applied to SAC-Ens, REDQ, SAC-Min, CQL, and EDAC and evaluated in various complex and sparse tasks in online and offline RL, MuJoCo, D4RL Gym, Franka Kitchen, and Antmaze. To ensure our reproducibility and fair comparisons, we prodive our source code. 1

2 Related Works

Ensemble Reinforcement Learning. Ensemble DQN [3] is the first method to adopt ensemble Q-learning for DRL. Ensemble DQN aims to reduce overestimation bias by computing a Bellman target as an average Q-value in the ensemble. Maxmin Q-learning [22] computes a Bellman target as a minimum Q-value over the Q-ensemble and shows significant performance enhancement based on the Atari benchmark. REDQ [6] constructs sample efficient ensemble Q-learning algorithm based on Maxmin Q-learning and SAC with a probabilistic ensemble by selecting a subset over Q-ensemble. Since REDQ combines the update-to-data (UTD) ratio, which is an orthogonal method with ensemble subset minimization, REDQ achieves state-of-the-art performance in the online MuJoCo environment. For offline reinforcement learning tasks, clipped Q-learning [2], denoted as SAC-Min is proposed, which computes the Bellman target as in Maxmin Q-learning. Diversification methods are proposed to improve the performance of ensemble Q-learning. MED-RL [31] optimizes the Gini coefficient to diversify an ensemble, and EDAC [2] optimizes the variance of the Q-value on OOD data to diversify an ensemble from the OOD data. SPQR proposes a Q-ensemble independence loss from a theoretical perspective, based on a spiked random model and random matrix theory to make ensemble Q-networks as independent as possible. In addition, conservative Q-learning (CQL) is proposed by [21], which penalizes the Q-values of OOD actions and performs well on numerous offline datasets.

Spiked Random Model. Signal recovery for a generative model using a spiked random model is analyzed by [8]. SPQR analyzes Q-ensemble independence for ensemble Q-learning in a spiked random model perspective. A universal data-driven optimal likelihood hypothesis test with linear spectral statistics (LSS) for a spiked random model as a central limit theorem for test statistics is given by [7, 18]. SPQR proposes a tractable universal data-driven test T ( λ ) to regularize the Q-ensemble independence.

44

46

48

50

52

54

56

Q-value

0

1

2

3

4

5

6

Count

150

100

50

0

50

Q-value

0

1

2

3

4

5

6

7

Count

50

60

70

80

90

100

Q-value

0

2

4

6

8

10

12

Count

12

10

8

6

4

2

0

2

Q-value

0

1

2

3

4

5

6

7

8

Count

0

13

26

39

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

42

44

46

48

0.0

0.2

0.4

0.6

0.8

1.0

3 Backgrounds

In this section, we describe ensemble Q-learning algorithms and visualize an empirical study of their limitations. In order to overcome its limitations, we introduce the random matrix theory and a spiked random model to formulate the notion of Q-ensemble independence from a theoretical perspective.

We consider a Markov decision process (MDP), defined by a tuple ( S , A , P, r, γ ) , where S is a finite state space, A is a finite action space, P : S ×A×S → [0 , 1] is the state-action transition probability, r : S × A → R is the reward function, and γ ∈ [0 , 1) is the discount factor. We define a state-action value function, called Q-function, Q ϕ : S × A → R , represents an expectation of the discounted sum of rewards for the trajectory started from s, a , given by E s t ,a t [ ∑ ∞ t =0 γ t r ( s t , a t )] .

3.1 Ensemble Q-learning

We first define bias e i sa = Q ∗ ( s, a ) -Q i ( s, a ) , where Q ∗ ( s, a ) is the optimal Q-value given for ( s, a ) and Q i ( s, a ) is the Q-value of the i -th networks in ensemble. Ensemble Q-learning assumes that the bias follows a zero mean, uniform, and independent distribution, i.e., e i sa ∼ Unif ( -τ, τ ) and is independent with respect to i , although REDQ does not assume uniform distribution. Given N Q-functions in the ensemble, there are three typical methods to compute a Bellman target: Ensemble DQN, Maxmin Q-learning, and REDQ.

Ensemble DQN. Ensemble DQN computes the Bellman target as the ensemble average Q-value. The bias and variance of the bias monotonically decrease as the ensemble size N increases, and they become 0 as N goes to infinity. We implement the SAC-style of Ensemble DQN, denoted as SAC-Ens.

Maxmin Q-learning. The Bellman target of Maxmin is to select the minimum Q-value within an ensemble, which can control the level of overestimation with a finite number of Q-functions. A modified version called SAC-Min is introduced specifically for offline RL tasks.

REDQ. REDQ computes the Bellman target as the minimum Q-value in a subset of the ensemble to become a sample-efficient algorithm.

Detailed explanations for Bellman targets and the evaluation function of each algorithm, denoted as Ens tar and Ens eval , are described in Appendix C.

To confirm the validity of uniform and independent assumptions of ensemble Q-learning, we plot Q-value distribution in Figure 1 and test whether each Q-distribution is uniform using the χ 2 test. We train 50 Q-networks 3M timesteps by the SAC-Min algorithm in the D4RL hopper-random dataset and evaluate it by the hopper-full-replay dataset. We also perform the χ 2 test for independence in Table 1. Out of 1000 batch data, only 35.8% can accept that the distribution is uniform and 30.4% can accept independence. Thus, e i sa ∼ Unif ( -τ, τ ) and i.i.d. assumption is empirically inaccurate and it is necessary to introduce the notion of Q-ensemble independence. More details on the χ 2 test are given in Appendix E.

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.4

Re

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.4

Im

=0

2

0

2

4

6

8

eigenvalue

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Density

=0

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.4

Re

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.4

Im

=1e-05

2

0

2

4

6

8

eigenvalue

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Density

=1e-05

As [30] also points out, the early collapse of Q-values, which means that the Q-values in the ensemble become almost identical, occurs during the training. We empirically visualize a strong correlation among Q-networks in Figure 1. Although the main advantage of ensemble Q-learning is based on diverse Q-functions, network initialization is the only process to maintain diversity over an ensemble since the Bellman target is identical across all networks without diversification and independence regularization methods.

3.2 Random Matrix Theory

Random matrix theory studies the probabilistic properties of large matrices with random entries. Dyson [10] constructs some categories of the random matrix as a real symmetric matrix as Gaussian orthogonal ensemble (GOE), complex hermitian matrix as Gaussian unitary ensemble (GUE), and quaternion self-dual matrix as Gaussian symplectic ensemble (GSE), called Dyson's classification . In this paper, we restrict our interest to matrices in GOE.

Definition 3.1 (Gaussian Orthogonal Ensemble, [26]) . Let random matrix W ∈ R N × N , its entities random variables w i , P [ W ] = P ( w 1 , . . . , w N 2 ) to be a joint probability density function of w i . Define the set of W as GOE when these properties hold:

w i are independent with respect to i .

W is a real symmetric matrix.

W is rotational invariance, i.e., P [ W ] dW = P [ W ′ ] dW ′ if W ′ = UWU -1 , U is orthogonal.

GOE is the most natural way to model random matrices in various applications, including machine learning. In simplified terms, we can consider GOE as a Wigner matrix, where entries are independent, symmetric, and follow Gaussian distribution. One of the main areas in random matrix theory is how eigenvalues or singular values of a large random matrix are distributed. More specifically, we only consider limiting properties of empirical spectral density (ESD), which is defined as, for random matrix W ∈ R N × N and its eigenvalues λ i , i = 1 , . . . , N ,

The limits of the empirical spectral density of the GOE, called the Wigner's semicircle law , are given by [34]. For GOE ˆ W ∈ R N × N with variance of entries σ 2 , λ 1 , . . . , λ N to be a eigenvalue of

1

ˆ

N

W

=

√

W

and empirical spectral distribution weakly almost surely, where

1

N

N

∑

j

=1

p

W

=

δ

(

λ

-

λ

j

)

, then

p

W

converges to

p

sc

We call p sc as the Wigner's semicircle distribution. Wigner's semicircle law does not rely on the probability density of entries.

3.3 Spiked Random Model

Random matrix theory models random data well, but in reality, most data have both informative signals and random components. A spiked random model is a theory about constructing a model of perturbation with a data matrix. The objective of this theory is to separate true signals from perturbated ones with random noise. True signals show spiked phenomena that make the largest eigenvalue detach from the bulk of eigenvalues. A spiked model was first introduced by [17] as an application for principal component analysis. We restrict our analysis to a spiked Wishart model , the most natural methodology to model a general perturbation. A spiked Wishart model is defined as follows.

Definition 3.2. Let Y ∈ R N × M ∼ P ψ , u, v is unknown vector, Y is spiked Wishart model when

Y = √ ψ N uv T + W , which is equivalent to sample covariance matrix ˆ Σ = I N + ψ N uu T

u, v are i.i.d from zero mean priors u ∼ P u , v ∼ P v , support bounded in radius by K u , K v

W ∼ P 0 is a matrix with i.i.d noise entries

M/N → ω

We will assume the spiked Wishart model with N = M , where we can ignore property 4. In Figure 2, we visualize how the spiked Wishart model works in both the data and spectral domain. For a detailed explanation, see Appendix A.

4 Q-ensemble Independence Regularization

In this section, we present a method of Q-ensemble independence regularization that is computationally feasible through the utilization of a spiked Wishart model. We bring the concept of ensemble Q-learning to the spiked Wishart model as Y ∼ P ψ representing learned Q-value by Q-learning algorithm and W ∼ P 0 representing the perfectly independent Q-ensemble. If W dominates Y , Y follows independent assumptions of ensemble Q-learning. If not, a collapse of Q-value and high correlation occurs, which cannot be benefited by the ensemble method. By integrating with ensemble Q-learning and Q-ensemble independence testing, we propose a tractable Q-ensemble independence loss with gain β , which is further described in the following section. To help understand the independence and diversity of Q-ensemble more clearly, we mentioned that β can be simply considered as a loss gain (weight) for Q-ensemble independence regularization before providing a detailed explanation about SPQR and β . A higher β represents highly-likely-to-independent.

4.1 Diversity and Independence of Q-ensemble

To clarify our work, we distinguish between the notions of diversity and independence. Numerous ensemble Qlearning methods improve their performance by minimizing the overestimation bias and the variance of the Bellman target estimate. However, such works assume that the biases e i sa are zero-mean independent and identically distributed over i , although early collapse, a high correlation between Q-ensemble, occurs due to the shared Bellman target. On the other hand, diversification methods such as EDAC develop an ensemble diversification objective to reduce the number of ensemble networks required. In particular, EDAC increases the variance of the ensemble to penalize OOD actions. In our work, we focus on the independence of a Q-ensemble to reduce the effect of correlated behavior resulting from a shared Bellman target. Roughly speaking, diversification can be interpreted as flattening the distribution of a Q-ensemble, while independence regularization can be interpreted as promoting

0.0

0.5

1.0

1.5

2.0

2.5

3.0

timesteps

1e6

0

20

40

60

80

100

120

Q Predictions Mean

=0.01

=0.1

=1.0

=10.0

=100.0

an i.i.d. sampling procedure for a given distribution. We verify that our proposed regularization successfully exploits both the conservative behavior and the diversification of a Q-ensemble. The Q-ensemble independence regularization empirically shows a large performance gain by controlling the conservatism, as shown in Figure 3, similar to the performance improvement of the ensemble method caused by the conservatism. For further details, see Appendix E.

4.2 Tractable Loss for Q-ensemble Independence

We construct a tractable testing criterion to determine whether a distribution is independent in Theorem 4.1. D KL ( p ψ ( λ ) , p sc ( λ )) does not require any prior assumption about P 0 and P ψ since in spectral domain, Wigner's semicircle law is universal for all random matrices. Proof of Theorem 4.1 is shown in Appendix B.

Theorem 4.1. Following definition 3.2, as N →∞ with probability at least 1 -δ , following test T ( λ ) is optimal.

We now describe a practical Q-ensemble independence loss for ensemble Q-learning using the spiked Wishart model and Theorem 4.1. Let Y and u, v for the spiked Wishart model denote a Q-value matrix and an unknown signal. By the test T ( λ ) , if D KL ( p ψ ( λ ) , p sc ( λ )) < ε or is minimized, it is theoretically guaranteed that Y is a random matrix, i.e. a Q-ensemble is independently distributed.

In conclusion, we propose D KL ( p ψ ( λ ) , p sc ( λ )) as a Q-ensemble independence regularization loss for ensemble Q-learning, without any prior assumption such as uniform distribution.

4.3 Algorithm

As described in Section 3.1, ensemble Q-learning has crucial but inaccurate assumptions about independence. To address the issue of insufficient independence, we propose SPQR : spiked Wishart Q-ensemble independence regularization for reinforcement learning, which we describe in Algorithm 1. Using the random matrix theory and a spiked random model lens, SPQR minimizes D KL ( p ψ ( λ ) , p sc ( λ )) as a form of regularization loss, denoted as SPQR loss, on the baseline ensemble Q-learning algorithms. D KL ( p ψ ( λ ) , p sc ( λ )) penalizes for going too far from an independent Q-ensemble, which means promoting Q-ensemble to follow P 0 . We visualize our concept in Figure 4. To compute the SPQR loss using p ψ ( λ ) , we need to construct a symmetric Q-matrix Y filled with Q-values. Then, we normalize the Q-matrix Y with a zero-mean and unit variance to compute the Wigner's semicircle distribution p sc ( λ ) with σ = 1 . Finally, we compute eigenvalues with a numerically stable algorithm by constructing the ESD p ψ ( λ ) and calculate the KL divergence between p ψ ( λ ) and p sc ( λ ) . The loss function L SPQR is computed as a expectation of D KL ( p ψ ( λ ) , p sc ( λ )) over the batch data. A detailed explanation of implementations and algorithms is shown in Appendix C.

Algorithm 1 SPQR

1: Input: γ, α, N, β, τ, | B |

2: Initialize policy θ , Q-function ϕ i , target Q-function ϕ ′ i , replay buffer D

3: repeat

4: y ( r, s ′ ) ← r + γ ( Ens tar ( ϕ ′ i ) -α log π θ ( a ′ | s ′ ) )

5: for i = 1 , . . . , N do

6: L SPQR = D KL ( p ψ ( λ ) , p sc ( λ ))

8: end for

10: until

Algorithm 2 SPQR Loss

1: Input: Q-network l k = { Q ϕ i ( s ′ , a ′ ) } N i =1 √

2: Y ∈ R D × D , D = ⌊ 1+8 N -1 2 ⌋ :

3: Y pq ← { l Dp + q , if p ≥ q Y qp , otherwise

4: Y ← ( Y -µ Y ) /σ Y

7:

p

2

wigner

(

λ

) =

4

-

λ

2

π

8: L SPQR ← 1 | B | ∑ j ∑ i p esd ( λ i ) log p esd ( λ i ) p wigner ( λ i )

9: Output: SPQR loss L SPQR

5 Experimental Results

To reveal the validity and universality of SPQR from algorithms to tasks, we implement SPQR in various ensemble Q-learning algorithms and various online/offline RL tasks. We implement SPQR in SAC-Ens, REDQ, SAC-Min, and CQL, and evaluate them in online MuJoCo Gym tasks, offline D4RL Gym, Franka Kitchen, and Antmaze tasks. We denote the SPQR-implemented version of each ensemble algorithm as SPQR-∗ , such as SPQR-SAC-Ens. All the following implementations use the same hyperparameter and experimental protocol with a baseline. For fair evaluation, the SPQR code is written on top of each official baseline code, REDQ 1 , EDAC 2 , and CQL 3 . Implementation of SPQR can be done by only 50 lines and training time only increases by 5 %. We empirically show that SPQR outperforms baseline ensemble Q-learning algorithms in both online and offline RL tasks. See the implementation details in Appendix D.

Is SPQR really affecting independence? We perform independence hypothesis testing and visualize the spectral distribution to investigate whether SPQR really affects the independence of the Q-ensemble. First, we analyze the performance gain of SPQR from a spike reduction perspective. We train a Q-ensemble using SAC-Min and SPQR-SAC-Min on the hopper-random dataset and test it on the hopper-full-replay dataset. Figure 6 and Table 1 show the evaluation results for the spike-reducing phenomenon of SPQR, which deviates less from Wigner's semicircle law. Compared to SPQR-SAC-Min with β = 0 . 1 , SAC-Min has significantly more spikes, in particular, 2434 spikes compared to 2182 spikes over 25600 data. For extreme regularization gain, β = 100 , the spike is drastically reduced to 698, 71% less than the baseline algorithm. We also plot a spike histogram for another diversification method, EDAC. EDAC increases spikes to 54% more than SAC-Min, which can be interpreted as SPQR being the orthogonal method with EDAC.

We also demonstrate the χ 2 test with null hypothesis H 0 : P ( Q i , Q j ) = P ( Q i ) P ( Q j ) and significant level α = 0 . 025 for independence hypothesis testing on the same experimental setting above. Since SPQR promotes being "more" independent, we use the proportion of data that can accept an independent hypothesis as a measure of more or less independence. In Table 1, we report the proportion of trials over 1000 tests that accept the null hypothesis H 0 , meaning that the Q-ensemble is independent. These empirical studies can be interpreted as SPQR having Q-ensemble independence regularization power since the acceptance rate increases monotonically as β increases. Orthogonality between SPQR and EDAC is also found from an independence testing perspective. However, reducing spikes and being more independent does not guarantee optimal performance since there is an optimal conservatism for each task. We can find the proper conservatism by controlling the independence regularization power, β . More details can be found in Appendix E.

𝑙𝑙

𝑘𝑘

=

𝑄𝑄1

,

𝑄𝑄2

, … ,

𝑄𝑄

𝑁𝑁

...

shuffle

𝑙𝑙

𝑘𝑘

...

...

...

...

...

...

...

...

:

(Order of

𝑄𝑄

𝑖𝑖

is arbitrary)

fill symmetric matrix with

𝑙𝑙

𝑘𝑘

Count

2.20

2.15

2.10

2.05

2.00

1.95

1.90

0

100

200

300

400

500

600

Wigner's semi-circle(rescaled)

SAC-Min, # spike: 2434

EDAC, # spike: 3751

SPQR-SAC-Min-beta=100.0, # spike: 698

1.90

1.95

2.00

2.05

2.10

2.15

2.20

Table 1: Number of spikes, independence hypothesis testing acceptance ratio, and performance of SAC-Min, EDAC, and SPQRβ = 0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 , 100 . 0 . We reproduce EDACη = 1 . 0 . The reported value of EDACη = 1 . 0 is given in the Section H of [2]

Online Reinforcement Learning. We implement SPQR-SAC-Ens and SPQR-REDQ with UTD ratio 1 and 20, and compare them to SAC-Ens and REDQ to show how SPQR improves performance. We evaluate each algorithm on the OpenAI Gym MuJoCo tasks. Figure 7 shows the training curves for SAC-Ens, REDQ, SPQR-SAC-Ens, and SPQR-REDQ. Since the two main contributions of REDQ, UTD ratio, and subset minimization, are orthogonal methods to each other, we experiment with SPQR-REDQ for UTD ratio 1 and 20. A high UTD ratio can be interpreted as frequent gradient updates, thereby resulting in better performance. For each algorithm, we plot the average return as a solid curve and the standard deviation as a shaded area. The implementation of SPQR yields notable improvements in the performance of SAC-Ens and REDQ across all tasks, as well as under both UTD ratio 1 and 20 conditions. Specifically, SPQR-SAC-Ens performs 2 times better than SAC-Ens on the Hopper task. Also, SPQR-REDQ-UTD=1 and SPQR-REDQ-UTD=20 show 1.1 and 1.3 times better than REDQ-UTD=1 and REDQ-UTD=20 for the Walker2d and Ant tasks.















































































































































To investigate why SPQR performs better, we evaluate the bias of the Q-value computed by the Monte-Carlo algorithm. The average of the Q-bias decreases with SPQR, which can be interpreted as the relationship between the distribution of the ensemble becoming more independent. We also evaluate the ablation study for the UTD ratio. SPQR-SAC-Ens shows performance improvement and bias alleviation with low standard deviation as the UTD ratio increases although high UTD SAC causes large and unstable bias. In terms of diversity, we also evaluate the standard deviation with different β for SPQR. As β increases, the standard deviation for explored state-action also increases in all training timesteps. In summary, Q-ensemble independence is the key factor in improving the performance of SPQR. Further details and experimental results can be found in Appendix E.

Offline Reinforcement Learning. SAC-Min is formally constructed and evaluated in [2]. We follow the reported value and evaluation protocol from the paper. We consider baseline behavior cloning (BC), CQL, EDAC, and SAC-Min. The evaluation results of SPQR-SAC-Min compared with SAC-Min and other baselines are shown in Table 2. For most tasks, SPQR outperforms all baselines well. Specifically, for ∗ -random dataset, SPQR improves SAC-Min by about 16% when CQL and EDAC fail for ∗ -random dataset. It implies that SPQR is robust for the quality of data. Since SPQR gives more conservatism, which shows identical behavior with increasing ensemble size N , SPQR-SAC-Min needs a smaller number of ensemble networks to outperform SAC-Min. For the hopper-medium, SAC-Min uses 500 Q-networks and SPQR uses only 50 Q-networks, which is about 90% efficient in terms of computational cost. It implies that the independence regularization of the Q-ensemble becomes computationally efficient.

We also implement SPQR to CQL( H ), denoted as SPQR-CQL-Min. Referring to the implementation details in the paper and the original baseline code of CQL, the original CQL uses a twin Q-function trick, which is equivalent to SAC-Min with N = 2 . We modified CQL as CQL-Min to use N

Q-networks to compute the Bellman target. Hence, the original CQL can be called CQL-Min with N = 2 . We evaluate SPQR-CQL-Min on D4RL Franka Kitchen and Antmaze, one of the most complex and sparse tasks in D4RL.

In Table 2, we evaluate SPQR-CQL-Min with other baseline methods. SPQR-CQL-Min outperforms all baseline algorithms, notably 2.6 times better than BEAR. For Franka Kitchen tasks, SPQR consistently improves CQL by 10% for all three tasks, which can be interpreted as robustness over the quality of data. Surprisingly, SPQR achieves the maximum score on 3 different Antmaze tasks, which outperforms all existing algorithms. Further details and results are presented in Appendix D and E.

Can other diversification methods benefit from the SPQR? Since SPQR regularizes Q-ensemble independence, which is the orthogonal method with other diversification methods, we implement SPQR in EDAC to benefit from the combination of two orthogonal methods. We evaluate SPQREDAC for the tasks where EDAC outperforms the baseline algorithm SAC-Min. SPQR and SPQREDAC show large performance gains over SAC-Min, as shown in Table 3.

6 Conclusions

We propose SPQR, a Q-ensemble independence regularization framework for ensemble Q-learning by applying random matrix theory and a spiked random model. We theoretically propose a tractable loss for Q-ensemble independence. The proposed theoretical approach does not require any prior assumption on the Q-function distribution, such as uniform distribution or i.i.d. In addition, we present the difference between diversity and independence and empirically show the orthogonality of the diversification method and SPQR. Furthermore, SPQR serves as a conservatism control framework for ensemble Q-learning by effectively regularizing independence with various empirical evidence. We also empirically show that SPQR is computationally efficient and outperforms many existing ensemble Q-learning algorithms on various online and offline RL benchmark tasks.

References

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not as i say: Grounding language in robotic affordances, 2022.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In NeurIPS , pages 7436-7447, 2021.

Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning , pages 176-185. PMLR, 2017.

Jess Banks, Cristopher Moore, Roman Vershynin, Nicolas Verzelen, and Jiaming Xu. Information-theoretic bounds and phase transitions in clustering, sparse pca, and submatrix localization. IEEE Transactions on Information Theory , 64(7):4872-4894, 2018.

Nicholas P Baskerville, Diego Granziol, and Jonathan P Keating. Appearance of random matrix theory in deep learning. Physica A: Statistical Mechanics and its Applications , 590:126742, 2022.

Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations , 2021.

Hye Won Chung and Ji Oon Lee. Weak detection of signal in the spiked wigner model. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1233-1241. PMLR, 09-15 Jun 2019.

Jorio Cocola, Paul Hand, and Vlad Voroninski. Nonasymptotic guarantees for spiked matrix recovery with generative priors. Advances in Neural Information Processing Systems , 33: 15185-15197, 2020.

Momar Dieng. On the distribution of eigenvalues and their spacings: a rapid introduction to random matrix theory.

Freeman J Dyson. The threefold way. algebraic structure of symmetry groups and ensembles in quantum mechanics. Journal of Mathematical Physics , 3(6):1199-1215, 1962.

Ahmed El Alaoui El Abidi. Detection limits and fluctuation results in some spiked random matrix models and pooling of discrete data . University of California, Berkeley, 2018.

FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science , 378(6624):1067-1074, 2022.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Mike Giles. An extended collection of matrix derivative results for forward and reverse mode automatic differentiation. 2008.

Hado Hasselt. Double q-learning. Advances in neural information processing systems , 23, 2010.

Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix backpropagation for deep networks with structured layers. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 2965-2973, 2015. doi: 10.1109/ICCV.2015.339.

Iain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics , 29(2):295-327, 2001.

Ji Hyung Jung, Hye Won Chung, and Ji Oon Lee. Detection of signal in the spiked rectangular models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5158-5167. PMLR, 18-24 Jul 2021.

Robert W Keener. Theoretical statistics: Topics for a core course . Springer, 2010.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. CoRR , abs/2110.06169, 2021. URL https://arxiv.org/abs/2110.06169 .

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33: 1179-1191, 2020.

Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations , 2020.

Lucien LeCam. Locally asymptotically normal families of distributions. Univ. California Publ. Statist. , 3:37-98, 1960.

Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6131-6141. PMLR, 18-24 Jul 2021.

Zhenyu Liao and Romain Couillet. The dynamics of learning: A random matrix approach. In International Conference on Machine Learning , pages 3072-3081. PMLR, 2018.

M.L. Mehta. Random matrices and the statistical theory of energy levels, 1967.

Karl Pearson. X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , 50(302):157-175, 1900.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. CoRR , abs/1911.08265, 2019.

Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures. In International Conference on Machine Learning , pages 8573-8582. PMLR, 2020.

Hassam Sheikh, Kizza Frisbee, and Mariano Phielipp. DNS: Determinantal point process based neural network sampler for ensemble reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 19731-19746. PMLR, 17-23 Jul 2022.

Hassam Sheikh, Mariano Phielipp, and Ladislau Boloni. Maximizing ensemble diversity in deep reinforcement learning. In International Conference on Learning Representations , 2022.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Aad W Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Eugene P. Wigner. Characteristic vectors of bordered matrices with infinite dimensions. Annals of Mathematics , 62(3):548-564, 1955. ISSN 0003486X.

A Review for Detection Theory and Spiked Random Model

We call strong detection when we perfectly detect the presence of a spike signal and weak detection when we detect the presence of a spike signal with some error by hypothesis testing. A formal definition for a strong detection can be constructed under the notion of contiguity by [23]. The contiguity of the spiked Wishart model is proven by the second moment method, which means that we cannot distinguish without error whether data is pure noise or contains some signal.

Definition A.1 (Contiguity) . Let P n and Q n be two sequences of probability measures defined on the sequence of measurable spaces (Ω n , F n ) . Q n is contiguous with respect to the sequence P n if P n ( A n ) → 0 implies Q n ( A n ) → 0 as n → ∞ for every sequence of measurable sets A n ∈ F n , denoted as Q n ◁ P n . The sequences P n and Q n are mutually contiguous if both Q n ◁ P n and P n ◁ Q n holds, denoted as Q n ◁▷ P n

Lemma A.2 (Le Cam's First Lemma) . Let P n and Q n be two sequences of probability measures defined on the sequence of measurable spaces (Ω n , F n ) . Then the following statements are equivalent: 1. Q n ◁ P n

If dP n /dQ n Q n - - → U along a subsequence, then P ( U > 0) = 1

If dQ n /dP n P n - - → V along a subsequence, then E [ V ] = 1

For any statistics T n : Ω n ↦→ R k , If T n P n - - → 0 , then T n Q n - - → 0

Lemma A.3 (Second moment method) . Let P n and Q n be two sequences of probability measures defined on the sequence of measurable spaces (Ω n , F n ) , and Radon-Nikodym derivative of P n with respect to Q n as L n ≡ dP n /dQ n . If lim sup E Q n [ L 2 n ] < ∞ , then P n ◁ Q n

Lemma A.4. For spiked wishart model following definition 3.2, K 4 u K 4 v ψ 2 < 1 , the families of distributions P 0 and P ψ ,indexed by M,N , are mutually contiguous in the limit N →∞ , M/N → ω

̸

Hypothesis Testing for Weak Detection. To distinguish whether spiked Wishart model Y follows P ψ or P 0 , detection error is inevitable since contiguity holds. Even if the perfect test is not available, it needs to be better than a random guess, which is achievable by constructing a hypothesis test. Let (Ω , F , ˆ ρ ) be a probability space, ( X , B ) be a topological space with Borel σ -algebra B , and Θ to be a set of parameters. For observed random variable X : Ω ↦→X with X ∼ P θ , θ ∈ Θ , and fixed ˆ θ = 0 , Objective of hypothesis testing is to distinguish between two hypotheses, called null hypothesis H 0 and alternative hypothesis H 1

Constructing a measurable function T : X ↦→ { 0 , 1 } to represent test, T ( X ) = 0 means accept H 0 and T ( X ) = 1 means accept H 1 . The error of test T is the sum of Type-I error, false positive, and Type-II error, false negative, stated as,

Optimal test T ∗ is defined as minimizing Type-II error subject to the constraint that Type-I error is no greater than a predetermined level α . T ∗ must have a form of likelihood ratio test by following Neyman-Pearson Lemma and propositions from [19].

Lemma A.5 (Neyman-Pearson Lemma) . Let null hypothesis H 0 with probability measure p 0 and density f 0 , alternative hypothesis H 1 with probability measure p 1 and density f 1 , density ratio r ( x ) = f 1 ( x ) /f 0 ( x ) , and constant k ∈ R such that p 0 { r ( x ) > k } ≤ α for a given significance level α . There exist an optimal test function T ∗ : R D → R , with a mild continuity assumption, such that

T ∗ ( X ) = ✶ ( f 1 ( x ) /f 0 ( x ) > k ) = ✶ ( r ( x ) > k ) To construct an optimal test by applying Neyman-Pearson Lemma, we define a likelihood ratio, Radon-Nikodym derivative between P ψ and P 0 as L ( Y ; ψ ) by the following equations. For convience, let Hamiltonian H : R N + M → R be as follows:

where ⊗ represents a tensor product, L ( Y ; ψ ) has form of

The optimal test for a spiked Wishart model is given as follows:

Lemma A.6. Let us assume that a spiked Wishart model is defined in definition 3.2. Let null hypothesis H 0 : Y ∼ P 0 , alternative hypothesis H 1 : Y ∼ P ψ , likelihood ratio L between P ψ and P 0 , and define tests T : R N × M →{ 0 , 1 } . By Lemma A.5, optimal test is,

Proof of A.6 is given in Section B.

Corollary A.7. For optimal test T ∗ , Error of T ∗ is

For ψ ≥ 0 such that K 4 u K 4 v ψ 2 < 1

where erfc ( x ) = 2 √ π ∫ ∞ x e -t 2 dt

The error for optimal test T ∗ is stated at Corollary A.7. Likelihood ratio log L is used for the optimal test to distinguish whether data Y in ensembles are sampled from i.i.d prior P 0 . However, log L is intractable since information about ψ and u, v is not given. Therefore, we need an alternative criterion for optimal test T ∗ that does not require prior knowledge to make it tractable with ensemble Q-learning. Intuitively, although we cannot know the exact form of P ψ and P 0 , its eigenvalue distribution can be described by ESD and Wigner's semicircle law. The expectation of log L becomes KLdivergence between P ψ and P 0 , followed by the definition of log L . By integrating these intuitions, intractable criterion log L can be restated as KL divergence between ESD and Wigner's semicircle law, which we prove in Section B.

Proof of Lemma A.2. The proof is exactly same as the Lemma 6.4 of [33]

Proof of Lemma A.3. For any event A ∈ F n , by Cauchy-Schwarz inequality,

∴ If lim sup E Q n [ L 2 n ] < ∞ , then Q n ( A n ) → 0 implies P n ( A n ) → 0 , concludes to P n ◁ Q n ■

Proof of Lemma A.4.

The proof is exactly same as the Theorem 4 of [4] ■

■

B Proofs

B.1 Technical Lemmas and Proof

Before proving our theoretical results, we present some lemmas to clear the description.

Lemma B.1 (Convergence) . Let ψ ≥ 0 such that K 4 u K 4 v ψ 2 < 1 , then in the limit N →∞ , where ⇝ denotes convergence in distribution,

The sign + means Y ∼ P 0 , -means Y ∼ P ψ

Lemma B.2 (Concentration) . for fixed u ∗ , v ∗ , Y = √ ψ N u ∗ v ∗ T + W , for every t ≥ 0 ,

Lemma B.3 (Weyl's Lemma) . If X is rotational invariance, P [ X ] = g ( Tr ( X ) , T r ( X 2 ) , . . . , T r ( X N ))

Lemma B.4 (Vandermonde determinant) . For GOE Y = O Λ O T , the relationship between joint probability density function of matrix entity and eigenvalues is given as,

Then vandermonde determinant become

Proof of Lemma B.1. The proof is exactly same as the Theorem 3.1 of [11] ■

Proof of Lemma B.2. The proof is exactly same as the Lemma 3.14 of [11] ■

Proof of Lemma B.3.

The proof is exactly same as the Lemma 1.1 of [9] ■

Proof of Lemma B.4. The proof is exactly same as the procedure in Chapter 3 of [26]

■

Proof of lemma A.6. Using Lemma B.1 and Lemma A.5, log L can distinguish at log L = 0 as N →∞ because of symmetry of Gaussian mean, i.e k = 1 for Neyman-Pearson Leamma. Therefore, optimal test T ∗ = ✶ (log L > 0) ■ B.2 Proof of Main Theorem

Theorem 4.1. Following definition 3.2, as N →∞ with probability at least 1 -δ , following test T ( λ ) is optimal.

Proof. By Lemma B.3 and Lemma B.4, For spiked wishart model Y ∼ P ψ

If D KL ( p ψ ( λ ) , p sc ( λ )) < ε ,

Let ∆ = log L -E [log L ] , ˆ t = Nt . By Lemma B.2,

Let δ = 2exp( -ˆ t 2 / 2 ψNK 2 u K 2 v ) . Then, with probability at least 1 -δ ,

Since D KL ( P ψ , P 0 ) < ˜ ε , then,

Choose ˜ ε = -∆ -δ ′ , δ ′ > 0 . Then,

∴ with probability at least 1 -δ , test T ( λ ) is optimal test T ∗ as N →∞

■

C Algorithm Details

C.1 Detailed Algorithm for SPQR

We describe a detailed version of SPQR for Algorithm 3. A detailed description for computing SPQR loss is shown in Algorithm 4. Table 4 shows functions used in implementing baseline ensemble Q-learning. For Figure 8, we visualize an illustrative example of SPQR.

Algorithm 3 SPQR

Input: discount γ ∈ [0 , 1) , temperature parameter α , ensemble size N , SPQR loss gain β , polyak τ , batch size | B |

Initialize policy parameter θ , Q-function parameters { ϕ i } N i =1 , target Q-function parameters { ϕ ′ i ← ϕ i } N i =1 , online/offline replay buffer D repeat if D not offline dataset then Take action from a t ∼ π θ ( ·| s t ) , observe reward r t , next state s t +1 D ← D ∪ { ( s t , a t , r t , s t +1 ) } end if B ←{ ( s j , a j , r j , s ′ j ) } | B | j =1 from D ▷ Sample a mini-batch y ( r, s ′ ) ← r + γ ( Ens tar ( { Q ϕ ′ i ( s ′ , a ′ ) } N i =1 ) -α log π θ ( a ′ | s ′ ) ) , a ′ ∼ π θ ( ·| s ′ ) ▷ Compute Q-target L SPQR ←L SPQR ( { ϕ i } N i =1 ) ▷ Compute SPQR loss using Algorithm 4 for i = 1 , . . . , N do L ϕ i ← 1 | B | ∑ ( s,a,r,s ′ ) ( Q ϕ i ( s, a ) -y ( r, s ′ )) 2 + β L SPQR ▷ Compute Q loss Update ϕ i with gradient descent using ∇ ϕ i L ϕ i Update ϕ ′ i ← τϕ ′ i +(1 -τ ) ϕ i end for L θ ← 1 | B | ∑ s ∈ B ( Ens eval ( { Q ϕ i ( s, ˜ a ) } N i =1 ) -α log π θ (˜ a | s ) ) , ˜ a ∼ π θ ( ·| s ) ▷ Compute policy loss Update θ with gradient ascent using ∇ θ L θ until Output: parameter θ, { ϕ i } N i =1 , { ϕ ′ i } N i =1

C.2 Detailed Algorithm for SPQR Loss

Algorithm 4 gives a detailed explanation for computing SPQR loss. We build a symmetric random matrix with a randomly-ordered Q-value. Our goal is to generate a symmetric Q-value matrix filled with an ensemble size of N. To avoid training or capturing the order of a matrix, we shuffle the filling order. This can be achieved by sorting the list of Q-networks l k . In order to create a symmetric matrix of maximum size, the upper triangle part and the corresponding lower triangle part are equally filled. The size of the matrix is set as D = ⌊ √ 1+8 N -1 2 ⌋ . D can be found by finding the largest integer that satisfies D 2 + D 2 ≤ N . To avoid numerical instability when computing KL divergence, we formulate Wigner's semicircle distribution as like ε -soft, in which all probability distribution value is at least ε , and we call it a soft Wigner's semicircle distribution. ρ gives weight between ε -soft distribution and Wigner's semicircle distribution. We fix ρ = 0 . 5 and ε = 0 . 01 .

Algorithm 4 SPQR Loss

▷ Build random matrix

▷ Normalizing

▷ Compute eigenvalue

▷ Define ESD where δ is dirac delta function

▷ Define soft Wigner's semicircle distribution

▷ Compute SPQR loss

C.3 Eigenvalue Backpropagation

To use D KL ( p ψ ( λ ) , p sc ( λ )) as a regularization loss, we need to backpropagate loss gradient through eigenvalue. In Theorem C.1, we construct a backpropagation equation about eigenvalue refer to [16] and [14]. For practical implementation, using torch.eigvals and torch.eigvalsh gives numerically stable eigenvalue backpropagation by solving det ( X -λI ) = 0 , not solving Xu = λu geometrically.

Theorem C.1. Let L : R d → R be a loss function, and X = U Λ U T with X ∈ R m × m , such that U T U = I and Λ possessing diagonal structure. Using the notation A ◦ B as a Hadamard product of two matrices, which ( A ◦ B ) ij = A ij B ij and A sym = 1 2 ( A T + A ) , A diag = I ◦ A Then

̸

The resulting partial derivatives are

Proof. Let X = U Λ U T , U T U = I , then XU = U Λ . Differentiation gives

Define dC = U T dU , so that dU = UdC , then

Multiplying U T to both sides and rearranging gives

Let E ij = λ j -λ i , then

Rearranging gives,

Since the diagonal of E are zero,

So,

̸

■

D Implementation Details

D.1 Online RL

We implement SPQR to SAC-Ens and REDQ on the REDQ code, SPQR to SAC-Min on the SACMin code, and SPQR to CQL on the CQL code. For online MuJoCo environment experiments, we implement SPQR-SAC-Ens and SPQR-REDQ. We use ∗ -v2 version of MuJoCo environments and train 12.5k timesteps for Hopper and 30k for Humanoid, Walker2d, and Ant. Hyperparameters shown in Table 5 are the same with REDQ implementation, including the size of a subset M = 2 for REDQ and SPQR-REDQ. For fair evaluation, we fix ensemble size N for each corresponding baseline algorithm, N = 20 for SAC-Ens and SPQR-SAC-Ens, and N = 10 for REDQ and SPQR-REDQ. We use annealing beta with a linear decay to stabilize training.

D.2 Offline RL

SPQR-SAC-Min is implemented based on SAC-Min and evaluated on D4RL MuJoCo Gym tasks with three environments, walker2d, halfcheetah, and hopper, each having separated six datasets. Each dataset is built with a different behavior policy. random uses 1M samples generated from the randomly initialized policy. expert uses 1M samples generated from a policy trained to completion with SAC. medium uses 1M samples generated from a policy that performs approximately 1/3 of the expert policy. medium-replay uses the replay buffer of a policy that performs as much as the medium agent. full-replay uses 1M samples from the final replay buffer of the expert policy. medium-expert uses a 50-50 split of medium and expert data, slightly less than 2M samples total. For fair evaluation with SAC-Min and EDAC, ∗ -v2 datasets are used for all tasks, and the same hyperparameter with SAC-Min and EDAC, as shown in Table 7.

We use normalized score by calculating 100 × raw _ score -random _ score expert _ score -random _ score , proposed evaluation metric from [13]. Algorithmic hyperparameter β is shown in Table 8. We fix β over similar tasks to show the robustness of SPQR. Also, we use a smaller ensemble size N for SPQR-SAC-Min rather than N for SAC-Min. For the hopper-∗ dataset, SPQR drastically reduces ensemble size N , using only 10% of N for SAC-Min. For SPQR-EDAC, we use the same hyperparameter with SPQR-SAC-Min and η = 0 . 5 for EDAC.

SPQR-CQL-Min is implemented based on CQL( H ), called CQL-Min, and evaluated on D4RL Franka Kitchen and Antmaze tasks, one of the most complicated tasks in the D4RL environment. Since there is no information about any hyperparameters for evaluating Franka Kitchen using CQL in the official

paper and code, we use hyperparameters given for evaluating Antmaze in the CQL paper, shown in Table 7. Like our efforts at SPQR-SAC-Min to test the SPQR's robustness, we fix all hyperparameters β , as shown in Table 8. For all D4RL tasks, we use annealing beta with an exponential decay to stabilize training.

E Further Experimental Results

E.1 Evaluation Result with More Baselines

E.2 Hypothesis Testing for Uniform Distribution

To check whether bias e i sa follows a uniform distribution, it's sufficient to check Q i ( s, a ) follows a uniform distribution since Q ∗ ( s, a ) is constant for fixed s, a . [27] propose a powerful test, called χ 2 test which determines whether a given distribution follows uniform or not using hypothesis testing. We perform a χ 2 test with significant level α = 0 . 025 on 50 Q-ensemble trained in the D4RL hopper-random dataset in 3M timesteps with 10000 data of D4RL hopper-full-replay dataset. Since the χ 2 test can be applied in a discrete random variable, we need to construct an approximative probabilistic mass function by binning. In Table 10, we show the acceptance rate of χ 2 test over 10000 data with different bin sizes. Especially in experiments with 20 bins, SPQR-SAC-Min shows a better acceptance rate than SAC-Min approximately 40%.

E.3 Conservatism of ensemble and SPQR

We visualize how ensemble size N affects the conservatism, and how SPQR can be computationally effective. As we described in Section 4.1 and Figure 3, SPQR has conservatism power. Increasing ensemble size N for SAC-Min shows conservative behavior, as proven in [22] and Figure 9. We train SAC-Min for 3M timesteps in a halfcheetah-random dataset with ensemble size N = 3 , 5 , 10 , 15 , 20 , 50 . SAC-Min empirically shows conservative behavior as ensemble size increases. As a result, SPQR can be a computationally efficient algorithm when using SAC-Min.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

timesteps

1e6

0

50

100

150

200

250

Q Predictions Mean

N=3

N=5

N=10

N=50

E.4 Spike histogram visualization

We plot a more detailed visualization of the spike histogram for Figure 10. We visualize how the spike histogram of EDAC and SPQR varies compared with SAC-Min.

2.20

2.15

2.10

2.05

2.00

1.95

1.90

0

100

200

300

400

500

600

Count

Wigner's semi-circle(rescaled)

SAC-Min, # spike: 2434

EDAC, # spike: 3751

1.90

1.95

2.00

2.05

2.10

2.15

2.20

2.20

2.15

2.10

2.05

2.00

1.95

1.90

Wigner's semi-circle(rescaled)

SAC-Min, # spike: 2434

SPQR-SAC-Min-beta=0.01, # spike: 2360

1.90

1.95

2.00

2.05

2.10

2.15

2.20

2.20

2.15

2.10

2.05

2.00

1.95

1.90

0

100

200

300

400

500

600

Count

Wigner's semi-circle(rescaled)

SAC-Min, # spike: 2434

SPQR-SAC-Min-beta=0.1, # spike: 2182

1.90

1.95

2.00

2.05

2.10

2.15

2.20

2.20

2.15

2.10

2.05

2.00

1.95

1.90

Wigner's semi-circle(rescaled)

SAC-Min, # spike: 2434

SPQR-SAC-Min-beta=1.0, # spike: 1844

1.90

1.95

2.00

2.05

2.10

2.15

2.20

2.20

2.15

2.10

2.05

2.00

1.95

1.90

0

100

200

300

400

500

600

Count

Wigner's semi-circle(rescaled)

SAC-Min, # spike: 2434

SPQR-SAC-Min-beta=10.0, # spike: 1603

1.90

1.95

2.00

2.05

2.10

2.15

2.20

2.20

2.15

2.10

2.05

2.00

1.95

1.90

Wigner's semi-circle(rescaled)

SAC-Min, # spike: 2434

SPQR-SAC-Min-beta=100.0, # spike: 698

1.90

1.95

2.00

2.05

2.10

2.15

2.20

E.5 Bias of Q

In Figure 11, we plot the average normalized bias and standard deviation of normalized bias. Bias is calculated by Monte-Carlo estimation. SPQR-∗ decreases average bias and standard deviation of bias from baseline algorithms. For all environments, SPQR-SAC-Ens shows a smaller bias and standard deviation of bias than SAC-Ens. For Hopper and Humanoid tasks, SPQR-REDQ shows underestimation, and for Walker2d and Ant tasks, SPQR-REDQ shows unbiased estimation. For Ant, SPQR-REDQ shows less bias than REDQ.











































    











































































   













































































    





































































   





















E.6 UTD Ablation Study

We visualize the performance of SPQR-SAC-Ens over variation of UTD ratio in Figure 12. SPQRSAC-Ens shows performance improvement by increasing the UTD ratio with a conservative and stable bias.







































































































E.7 Early Collapse

To show how SPQR affects the ensemble when the agent collects new tuple ( s, a, r, s ′ ) in terms of exploration and variance, we evaluate the standard deviation with different β .

As [30] pointed out, the early collapse of Q-values occurs, which means that in an early stage of training, the variance of the Q-ensemble is almost zero since Q-values become similar to each other. To show how SPQR addresses the early collapse problem, we visualize the standard deviation of Q-ensemble in Figure 13. We train 10 Q-networks from the MuJoCo Hopper environment with the SAC-Min algorithm with the same hyperparameter. SPQR does not collapse Q-values during training, preserving appropriate variance of Q-value, and higher values of β result in greater variance for β = 0 . 3 and β = 2 . 0 .

0

20000

40000

60000

80000

100000

120000

timesteps

0.2

0.4

0.6

0.8

1.0

1.2

std of Q-value

=0.0

=0.3

=2.0

E.8 Hyperparameter Sensitivity

We evaluate SPQR over various β to demonstrate hyperparameter sensitivity. We fix N = 10 for SPQR-SAC-Min and evaluate it on the D4RL Gym halfcheetah-random dataset and halfcheetahmedium dataset. Evaluation results are shown in Table 11.

E.9 Compare with MED-RL

We implement and evaluate another diversifying method for online RL tasks, MED-RL. We use the Gini coefficient for MED-RL and use reported hyperparameters with UTD ratio 1. SPQR outperforms MED-RL for all tasks.































































































































































E.10 t-SNE Visualization

Figure 15 shows t-SNE [32] clustering visualization of Q-ensemble trained 3M timesteps by SPQRSAC-Min algorithm at halfcheetah-medium-expert. We use t-SNE clustering for the last hidden layer representation of Q-ensemble with ensemble size 10 and 256 batch data at halfcheetah-medium-expert tasks. The t-SNE result shows that Q-ensemble trained by SPQR-SAC-Min is well clustered.

75

50

25

-25

-50

-75

50

-25

25

50

75

E.11 Correlation Visualization

We visualize a detailed figure for the Rightmost figure of Figure 1, the Pearson correlation coefficient matrix for a fully trained Q-ensemble and initial Q-ensemble in Figure 16. We train each Q-ensemble in the D4RL hopper-random dataset. For effective visualization, we sample randomly 10 Q-networks among 50 Q-networks and compute the Pearson correlation coefficient between each sampled Qnetwork with 256 batch data in a hopper-full-replay dataset. For the initial Q-ensemble, Q-networks are uncorrelated from each other. As we train Q-ensemble with SAC-Min for 3M timesteps, Qnetworks become strongly correlated with each other.

38

10

43

16

27

24

32

28

1

11

38

10

43

16

27

24

32

28

1

11

1

0.0073

1

-0.056

0.08

1

-0.4

-0.37

0.12

1

-0.48

-0.14

-0.1

0.071

1

0.33

-0.34

0.43

-0.07

-0.22

1

0.48

0.57

0.35

-0.39

-0.38

0.21

1

-0.73

-0.21

0.076

0.57

0.49

-0.28

-0.4

1

-0.033

0.2

0.26

0.18

0.25

-0.014

0.23

0.2

1

0.55

0.2

0.021

-0.3

-0.57

0.12

0.39

-0.49

-0.1

1

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

20

12

21

19

10

18

7

35

28

4

20

12

21

19

10

18

7

35

28

4

1

0.93

1

0.67

0.78

1

0.9

0.92

0.79

1

0.94

0.89

0.57

0.82

1

0.94

0.94

0.72

0.93

0.92

1

0.84

0.82

0.63

0.82

0.83

0.86

1

0.93

0.9

0.6

0.85

0.93

0.94

0.86

1

0.94

0.94

0.7

0.91

0.94

0.97

0.87

0.94

1

0.97

0.94

0.68

0.93

0.94

0.96

0.86

0.93

0.96

1

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the SPQR loss to regularize the independence of the Q-network ensemble. The authors apply the random matrix theory and a spiked random model to derive the KL loss between Wigner’s semicircle distribution and the empirical spectral density of eigenvalues. The authors show that the independence could be enforced by minimizing this KL divergence loss with theoretical guarantees by satisfying the testing hypothesis. Extensive experiments on both the online and offline RL tasks and several baselines demonstrate that SPQR could improve the performance of the current ensemble RL algorithms promisingly by increasing of independence of the ensemble networks.\n\nSTRENGTHS:\n1.\tThis paper is well-organized and well-written.\n2.\tThe proposed SPQR is with a simple format and shows a promising improvement over the ensemble RL methods such as SAC-ens and REDQ.\n3.\tThe authors build the SPQR loss with mathematical tools and provide the theistical guarantee to ensure the independence of the ensembled Q-networks.\n4.\tExperiments on both the online and offline RL tasks are sufficient to validate the generality of the proposed SPQR and that it can improve the performance of ensemble RL methods.\n\nWEAKNESSES:\n1.\tDetailed explanations between the used math tools and the proposed method are missing, making the understanding hard for the readers. For example, the connection between the random matrix theory and a spiked random model is not explained well. What is the purpose of the use of a spiked random model in SPQR?\n2.\tThere lack details of in Algorithm 1. For example, how to construct the symmetric Q-matrix is not shown. The detailed version of the core algorithm is given in the appendix, which is strange.\n3.\tThe quality of the figures needs to be improved. For example, the text size is too small.\n4.\tThere are many typos.\n\na)\tIn Line 73, “tp” should be “to”.\n\nb)\tIn Line 81, “SPQR analysis Q-ensemble independence …”.\n\nc)\tIn Line 113, “Q-learning, We plot”.\n\nd)\tIn Line 206, “Proof of Theorem 4.1 are …”.\n\nQUESTIONS:\n1.\tWhat is the purpose of the use of spiked random model in SPQR? Could the authors provide some illustration examples or give more explanations?\n2.\tHow to construct the symmetric Q-matrix in Algorithm 1?\n3.\tWhy is the GOE chosen for the study and why should we relax the definition of GOE?\n\nLIMITATIONS:\nNA.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The paper deals with the problem of alleviating overestimation bias in RL, using ensembles Q-functions. The authors argue that previous methods do not provide a theoretical guarantee of the independence of the members of the ensemble. To provide this they propose an approach based on random matrix theory, which, in practice, can be implemented as a regularization loss. 

The approach is evaluated in a number of settings for RL in tasks in MuJoCo, D4RL Gym, Franka Kitchen and Antmaze.

STRENGTHS:
+ The approach has a strong, non-trivial theoretical foundation.
+ Experimental results show a significant improvement over the baselines.

WEAKNESSES:
- Throughout the paper, there are many grammatical errors and awkward formulations. Some of these might hinder understanding, such as on page 5: 
"high correlation occurs, which cannot be benefited by the ensemble method". I assume it this case what the authors mean is "benefit the ensemble method". 
Many of these could be easily fixed with a grammar checker. 

- The text on figures 5 and 6 (and to a lesser degree, figure 4) are unreadably small. Graphs should be redrawn in such a way that the text is still readable at a normal printing size.

QUESTIONS:
The meaning of Figure 3 is not clear. It seems that the higher the \beta, the lower the average Q value. But at this point in the paper, the approach had not yet been described, so it is not clear what the \beta controls. Also, we don't know the correct value of Q. I can make some assumptions that the figure aims to show how the proposed technique improves on the overestimation bias, but this (if true) would need to be explained more clear.

LIMITATIONS:
No separate limitations section found in the paper. 

The theoretical nature of the paper does not raise ethical concerns.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new regularization loss that improves the independence of Q ensemble, thus improving the performance on online and offline DRL settings. The authors first point out that previous works with Q ensemble either rely on assumptions that are inaccurate in practice, or rely on heuristics to improve diversity of networks and no theoretical support. The authors provide theoretical results that support the proposed method, and also conducted empirical experiments, where the proposed method is applied to multiple recent online and offline algorithms, and tested on the MuJoCo online and d4rl offline benchmarks. On a high level, the regularization (Spiked Wishart Q-ensemble independence reuglarization (SPQR)) encourages the Q ensemble distribution to be closer to an ideal independent ensemble, resulting in a more diversified Q prediction values, lower bias and better performance.\n\nSTRENGTHS:\n**originality**\n- The theoretical results and the proposed algorithm, the findings can be seen as novel\n- Learning with Q ensembles has been studied in many previous works, but as the authors pointed out, they either rely on assumptions that are inaccurate in practice, or rely on heuristics, so the novelty is quite good here. \n\n**quality**\n- Overall the presentation is good, writing is clear\n\n**Clarity**\n- The motivation, connection to related works are all quite clear, and ample technical details are given, the authors also explained that they try to be fair in the comparisons and use the same hyperparameters. And the authors explained computation time, performance improvement, implementation difficulty quite clearly. \n\n**significance**\n- the analysis is nice, which shows that the proposed method seems to achieve the desired improved independence. \n- empirical results showing consistent performance improvement over the baseline, big improvement in some settings, slight improvements in others, but overall quite consistent. Consistent improvement over recent sota algorithms such as REDQ with small change in the code and low computation cost and no excessive fine-tuning is quite impressive.  \n- the theoretical part is interesting and quite important\n\nWEAKNESSES:\nI don't have very major concerns, one thing that can be fixed is the text in all your figures, especially those in the main paper are just too small, please make the fontsize of legend, as well as axis labels and ticks and any other text in the figures bigger so they are easier to read. \n\nMinor issue: \n- line 81: SPQR analysis -> analyzes ?\n- Figure 3 caption can you add a short sentence on what does beta do? It is not explained until a later section and I got confused when reading to Figure 3.\n\nQUESTIONS:\n- Main suggestion: fix the figures and make them more readable. \n- Question: what is the limitation of your work? Are these cases where the proposed method might fail, or type of algorithms that it cannot be applied to?\n\nLIMITATIONS:\nThe proposed method seems to be quite general and can be applied to other methods based on Q ensembles. However can be good to have some discussion on limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a spiked Wishart Q-ensemble independence regularization (SPQR) to improve the independence of ensembling in Q-learning. SPQR encourages the ensemble to be closer to an ideal independent ensemble by penalizing the KL divergence between the eigenvalue distribution of the current ensemble and an ideal one.\n\nSTRENGTHS:\nThe paper is easy to follow. The paper provides nice evidence of lack of independence in Q-ensemble training in current methods. The regularization to increase the diversity is well motivated, and seems simple and practical. The empirical evaluation is comprehensive, though more baselines would make them more compelling.\n\nWEAKNESSES:\nThere is quite a bit of prior work on ensembling in deep RL, for example bootstrapped DQN [1] and MSG [2], which are not discussed. In particular, [2] provides theoretical and empirical support for constructing independent ensembles. A discussion and comparison with MSG is warranted given the emphasis on constructing independent ensembles.\n\nWhile the emphasis of work is on improving ensembling methods in deep RL, recent methods have been able to forego ensembling while maintaining or improving performance over REDQ, for example DroQ [3] or RLPD [4]. It would be good to contextualize the improvements from SPQR by adding comparisons with these methods.\n\nMore comparisons with offline RL methods would help too, for example, IQL.\n\n[1] Deep Exploration via Bootstrapped DQN. Osband et al. \n[2] Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters. Ghasemipour et al. NeurIPS, 2022.\n[3] Dropout Q-Functions for Doubly Efficient Reinforcement Learning. Hiraoka et al.\n[4] Efficient Online Reinforcement Learning with Offline Data. Ball et al.\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nWhile I do not see potential negative societal impact, a discussion on limitations is missing from the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMitigating overestimation bias is crucial for deep reinforcement learning. Existing works about the ensemble techniques for Q-learning have been explored to leverage the diversity of multiple Q-functions. The authors argue that there has been no attempt to ensure ensemble independence from a theoretical standpoint. The authors introduce a regularization loss for Q-ensemble independence, based on random matrix theory, called Spiked Wishart Q-ensemble Independence Regularization (SPQR). The authors incorporate SPQR into online and offline ensemble Q-learning algorithms. Experimental results show that SPQR surpasses baseline algorithms in both online and offline RL benchmarks, demonstrating its effectiveness in addressing overestimation bias and improving performance.\n\nSTRENGTHS:\n- Significance and Originality: The problem and viewpoint that the paper studies - how to address overestimation bias/out-of-distribution error in RL is an important problem, and there have been a number of works investigating how to tackle this problem based on ensemble learning. However, most of the methods assume the bias follows a uniform and independent distribution, which may not hold in practice. This paper aims to improve previous ensemble methods from this perspective with a theoretical guarantee of improved Q-ensemble independence. The viewpoint from the random matrix theory seems novel, and the authors also propose a practical and tractable implementation for the method, which makes it possible to be applied in standard high-dimensional tasks.\n\n- Quality: The authors have done a comprehensive experiments to evaluate the methods in different setupds, and the proposed SPQR methods provides improvements in different aspects.\n\n- Clarity: The paper is also well-written and easy to follow.\n\nWEAKNESSES:\n- It is claimed in the paper that the assumption that most previous works rely on may not be true (the i.i.d. assumption about the bias). Could the authors demonstrate this effect is generally invalid in most of the environments? It would be better to convince the readers (since previous methods also have great performance although this assumption may not hold in practice).\n\n- Results in Figure 5 about the validation for the independence analysis are interesting. Does this generally hold in different tasks?\n\n- The improvement in standard D4RL locomotion tasks is somewhat marginal (4% improvement) considering increasing computation costs. \n\n- The results in Figure 6 about the performance of SAC-Ens is very low. Could authors better explain this result?\n\nQUESTIONS:\nPlease find my questions in the weakness section.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}