SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Choose  $\tilde{\varepsilon} = -\Delta - \delta'$ ,  $\delta' > 0$ . Then,
$$
\log L <   - \delta^ {\prime} <   0
$$
$\therefore$  with probability at least  $1 - \delta$ , test  $T(\lambda)$  is optimal test  $T^{*}$  as  $N \to \infty$
C.1 Detailed Algorithm for SPQR
We describe a detailed version of SPQR for Algorithm 3. A detailed description for computing SPQR loss is shown in Algorithm 4. Table 4 shows functions used in implementing baseline ensemble Q-learning. For Figure 8, we visualize an illustrative example of SPQR.
Algorithm 3 SPQR
Input: discount  $\gamma \in [0,1)$ , temperature parameter  $\alpha$ , ensemble size  $N$ , SPQR loss gain  $\beta$ , polyak  $\tau$ , batch size  $|B|$   
Initialize policy parameter  $\theta$ , Q-function parameters  $\{\phi_i\}_{i=1}^N$ , target Q-function parameters  $\{\phi_i'\gets \phi_i\}_{i=1}^N$ , online/offline replay buffer  $\mathcal{D}$   
repeat  
if  $\mathcal{D}$  not offline dataset then  
Take action from  $a_t \sim \pi_\theta(\cdot|s_t)$ , observe reward  $r_t$ , next state  $s_{t+1}$ $\mathcal{D} \gets \mathcal{D} \cup \{(s_t, a_t, r_t, s_{t+1})\}$   
end if  
 $B \gets \{(s_j, a_j, r_j, s_j')\}_{j=1}^{|B|}$  from  $\mathcal{D}$ $\begin{array}{l} y(r, s') \gets r + \gamma \left( Enstar(\{Q_{\phi_i'}(s', a')\}_{i=1}^N) - \alpha \log \pi_\theta(a'|s') \right), a' \sim \pi_\theta(\cdot|s') \end{array}$ $\mathcal{L}_{SPQR} \gets \mathcal{L}_{SPQR}(\{\phi_i\}_{i=1}^N)$   
for  $i = 1, \ldots, N$  do  
 $\mathcal{L}_{\phi_i} \gets \frac{1}{|B|} \sum_{(s,a,r,s')} (Q_{\phi_i}(s,a) - y(r,s'))^2 + \beta \mathcal{L}_{SPQR}$   
Update  $\phi_i$  with gradient descent using  $\nabla_{\phi_i} \mathcal{L}_{\phi_i}$   
Update  $\phi_i' \gets \tau \phi_i' + (1 - \tau) \phi_i$   
end for  
 $\mathcal{L}_{\theta} \gets \frac{1}{|B|} \sum_{s \in B} (Eneval(\{Q_{\phi_i}(s,\tilde{a})\}_{i=1}^N) - \alpha \log \pi_\theta(\tilde{a}|s)), \tilde{a} \sim \pi_\theta(\cdot|s)$   
Update  $\theta$  with gradient ascent using  $\nabla_{\theta} \mathcal{L}_{\theta}$   
until  
Output: parameter  $\theta, \{\phi_i\}_{i=1}^N, \{\phi_i'\}_{i=1}^N$
Table 4: Target and evaluation functions used in Ensemble Q-learning algorithms
Algorithm	Enstar(\{Qφi′(s′,a′)N i=1}	Enseval(\{Qφi(s,tildea)N i=1}
SAC-Ens	1/N ∑i=1N Qφi′(s′,a′)	1/N ∑i=1N Qφi(s,tildea)
REDQ	mini∈M Qφi′(s′,a′), M⊂{1,...,N}	1/N ∑i=1N Qφi(s,tildea)
SAC-Min	mini=1,...,N Qφi′(s′,a′)	mini=1,...,N Qφi(s,tildea)
CQL-Min	mini=1,...,N Qφi′(s′,a′)	mini=1,...,N Qφi(s,tildea)
Figure 8: Illustrative example of SPQR compared to baseline methods.
C.2 Detailed Algorithm for SPQR Loss
Algorithm 4 gives a detailed explanation for computing SPQR loss. We build a symmetric random matrix with a randomly-ordered Q-value. Our goal is to generate a symmetric Q-value matrix filled with an ensemble size of N. To avoid training or capturing the order of a matrix, we shuffle the filling order. This can be achieved by sorting the list of Q-networks  $l_{k}$ . In order to create a symmetric matrix of maximum size, the upper triangle part and the corresponding lower triangle part are equally filled. The size of the matrix is set as  $D = \lfloor \frac{\sqrt{1 + 8N} - 1}{2} \rfloor$ . D can be found by finding the largest integer that satisfies  $\frac{D^2 + D}{2} \leq N$ . To avoid numerical instability when computing KL divergence, we formulate Wigner's semicircle distribution as like  $\varepsilon$ -soft, in which all probability distribution value is at least  $\varepsilon$ , and we call it a soft Wigner's semicircle distribution.  $\rho$  gives weight between  $\varepsilon$ -soft distribution and Wigner's semicircle distribution. We fix  $\rho = 0.5$  and  $\varepsilon = 0.01$ .
Algorithm 4 SPQR Loss
Input: List of Q-network  $l_{k} = \{Q_{\phi_{i}}(s^{\prime},a^{\prime})\}_{i = 1}^{N}$
$$
Y \in \mathbb {R} ^ {D \times D}, D = \left\lfloor \frac {\sqrt {1 + 8 N} - 1}{2} \right\rfloor :
$$
$$
Y _ {p q} \leftarrow \left\{ \begin{array}{l l} l _ {D p + q}, & \text {i f} p \geq q \\ Y _ {q p}, & \text {o t h e r w i s e} \end{array} \right.
$$
$$
Y \leftarrow (Y - \mu_ {Y}) / \sigma_ {Y}
$$
$$
\left\{\lambda_ {i} \right\} _ {i = 1} ^ {N} \leftarrow E i g e n \left(\frac {1}{\sqrt {N}} Y\right)
$$
$$
p _ {e s d} (\lambda) = \frac {1}{N} \sum_ {i} \delta (\lambda - \lambda_ {i})
$$
$$
p _ {w i g n e r} (\lambda) = \left\{ \begin{array}{l l} \rho \frac {\sqrt {4 - \lambda^ {2}}}{2 \pi}, & \text {i f} \lambda \leq 2 \\ (1 - \rho) \varepsilon , & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\mathcal {L} _ {S P Q R} \leftarrow \frac {1}{| B |} \sum_ {j} \sum_ {i} p _ {e s d} (\lambda_ {i}) \log \frac {p _ {e s d} (\lambda_ {i})}{p _ {w i g n e r} (\lambda_ {i})}
$$
Output: SPQR loss  $\mathcal{L}_{SPQR}$
Build random matrix
$\triangleright$  Compute eigenvalue
$\triangleright$  Define ESD where  $\delta$  is dirac delta function
$\triangleright$  Define soft Wigner's semicircle distribution
$\triangleright$  Compute SPQR loss
C.3 Eigenvalue Backpropagation
To use  $D_{KL}(p_{\psi}(\lambda), p_{sc}(\lambda))$  as a regularization loss, we need to backpropagate loss gradient through eigenvalue. In Theorem C.1, we construct a backpropagation equation about eigenvalue refer to [16] and [14]. For practical implementation, using torch.eigvals and torch.eigvalsh gives numerically stable eigenvalue backpropagation by solving  $\det(X - \lambda I) = 0$ , not solving  $X u = \lambda u$  geometrically.
Theorem C.1. Let  $L: \mathbb{R}^d \to \mathbb{R}$  be a loss function, and  $X = U\Lambda U^T$  with  $X \in \mathbb{R}^{m \times m}$ , such that  $U^T U = I$  and  $\Lambda$  possessing diagonal structure. Using the notation  $A \circ B$  as a Hadamard product of two matrices, which  $(A \circ B)_{ij} = A_{ij}B_{ij}$  and  $A_{sym} = \frac{1}{2}(A^T + A)$ ,  $A_{diag} = I \circ A$  Then
$$
d \Lambda = \left(U ^ {T} d X U\right) _ {d i a g}
$$
$$
d U = U \left(K ^ {T} \circ \left(U ^ {T} d X U\right) _ {s y m}\right)
$$
$$
K _ {i j} = \left\{ \begin{array}{l l} \frac {1}{\lambda_ {i} - \lambda_ {j}}, & i \neq j \\ 0, & i = j \end{array} \right.
$$
The resulting partial derivatives are
$$
\frac {\partial L}{\partial X} = U \left\{\left. \left(K ^ {T} \circ \left(U ^ {T} \frac {\partial L}{\partial U}\right) _ {s y m}\right) + \left(\frac {\partial L}{\partial \Lambda}\right) _ {d i a g} \right\} U ^ {T} \right.
$$
Proof. Let  $X = U\Lambda U^T$ ,  $U^T U = I$ , then  $XU = U\Lambda$ . Differentiation gives
$$
d X U + X d U = d U \Lambda + U d \Lambda
$$
Define  $dC = U^T dU$ , so that  $dU = UdC$ , then
$$
d X U + U \Lambda d C = U d C \Lambda + U d \Lambda
$$
Multiplying  $U^T$  to both sides and rearranging gives
$$
d C \Lambda - \Lambda d C + d \Lambda = U ^ {T} d X U
$$
Let  $E_{ij} = \lambda_j - \lambda_i$ , then
$$
d C \Lambda - \Lambda d C = E \circ d C
$$
Rearranging gives,
$$
E \circ d C + d \Lambda = U ^ {T} d X U
$$
Since the diagonal of  $E$  are zero,
$$
d \Lambda = I \circ (U ^ {T} d X U) = (U ^ {T} d X U) _ {\text {d i a g}}
$$
$$
d C = K \circ (U ^ {T} d X U) \Rightarrow d U = U (K \circ (U ^ {T} d X U)), \text {w h e r e} K _ {i j} = \left\{ \begin{array}{l l} \frac {1}{\lambda_ {i} - \lambda_ {j}}, & i \neq j \\ 0, & i = j \end{array} \right.
$$
$$
\therefore d \Lambda = I \circ (U ^ {T} d X U) = (U ^ {T} d X U) _ {\text {d i a g}}, \quad d U = U (K ^ {T} \circ (U ^ {T} d X U) _ {\text {s y m}})
$$
D.1 Online RL
We implement SPQR to SAC-Ens and REDQ on the REDQ code, SPQR to SAC-Min on the SAC-Min code, and SPQR to CQL on the CQL code. For online MuJoCo environment experiments, we implement SPQR-SAC-Ens and SPQR-REDQ. We use  $*$ -v2 version of MuJoCo environments and train 12.5k timesteps for Hopper and 30k for Humanoid, Walker2d, and Ant. Hyperparameters shown in Table 5 are the same with REDQ implementation, including the size of a subset  $M = 2$  for REDQ and SPQR-REDQ. For fair evaluation, we fix ensemble size  $N$  for each corresponding baseline algorithm,  $N = 20$  for SAC-Ens and SPQR-SAC-Ens, and  $N = 10$  for REDQ and SPQR-REDQ. We use annealing beta with a linear decay to stabilize training.
Table 5: Hyperparameters used in the Mujoco environments
Hyperparameter	value
discount factor γ	0.99
initial temperature parameter α	0.2
Q learning rate	3e-4
policy learning rate	3e-4
hidden layer width	256
hidden layer depth	2
batch size |B|	256
polyak parameter τ	0.995
optimizer	Adam
Table 6: Algorithm hyperparameters used in the Mujoco environments
Task	SPQR-SAC-Ens (N,β)	SPQR-REDQ (N,β)
Hopper	20,0.5	10,0.5
Humanoid	20,2.0	10,2.0
Walker2d	20,0.1	10,0.1
Ant	20,2.0	10,2.0
D.2 Offline RL
SPQR-SAC-Min is implemented based on SAC-Min and evaluated on D4RL MuJoCo Gym tasks with three environments, walker2d, halfcheetah, and hopper, each having separated six datasets. Each dataset is built with a different behavior policy. random uses 1M samples generated from the randomly initialized policy. expert uses 1M samples generated from a policy trained to completion with SAC. medium uses 1M samples generated from a policy that performs approximately 1/3 of the expert policy. medium-replay uses the replay buffer of a policy that performs as much as the medium agent. full-replay uses 1M samples from the final replay buffer of the expert policy. medium-expert uses a 50-50 split of medium and expert data, slightly less than 2M samples total. For fair evaluation with SAC-Min and EDAC,  $*$ -v2 datasets are used for all tasks, and the same hyperparameter with SAC-Min and EDAC, as shown in Table 7.
We use normalized score by calculating  $100 \times \frac{\text{raw_score} - \text{random_score}}{\text{expert_score} - \text{random_score}}$ , proposed evaluation metric from [13]. Algorithmic hyperparameter  $\beta$  is shown in Table 8. We fix  $\beta$  over similar tasks to show the robustness of SPQR. Also, we use a smaller ensemble size  $N$  for SPQR-SAC-Min rather than  $N$  for SAC-Min. For the hopper-* dataset, SPQR drastically reduces ensemble size  $N$ , using only  $10\%$  of  $N$  for SAC-Min. For SPQR-EDAC, we use the same hyperparameter with SPQR-SAC-Min and  $\eta = 0.5$  for EDAC.
SPQR-CQL-Min is implemented based on CQL( $\mathcal{H}$ ), called CQL-Min, and evaluated on D4RL Franka Kitchen and Antmaze tasks, one of the most complicated tasks in the D4RL environment. Since there is no information about any hyperparameters for evaluating Franka Kitchen using CQL in the official
paper and code, we use hyperparameters given for evaluating Antmaze in the CQL paper, shown in Table 7. Like our efforts at SPQR-SAC-Min to test the SPQR's robustness, we fix all hyperparameters  $\beta$ , as shown in Table 8. For all D4RL tasks, we use annealing beta with an exponential decay to stabilize training.
Table 7: Hyperparameters used in the D4RL tasks
Hyperparameter	Gym	Franka Kitchen	Antmaze
discount factor γ	0.99	0.99	0.99
initial temperature parameter α	1.0	1.0	1.0
Q learning rate	3e-4	3e-4	3e-4
policy learning rate	3e-4	1e-4	1e-4
hidden layer width	256	256	256
hidden layer depth	3	3	3
batch size |B|	256	256	256
polyak parameter τ	0.995	0.995	0.995
optimizer	Adam	Adam	Adam
lagrange threshold (for CQL)	-	5.0	5.0
Table 8: Algorithmic hyperparameter used in the D4RL Gym and Franka Kitchen tasks. SAC-Min and CQL-Min follow reported values. Refer to [2] and [21]
Gym	SAC-Min (N)	SPQR-SAC-Min (N)	SPQR-SAC-Min (β)
walker2d-random	20	20
walker2d-medium	20	10
walker2d-expert	100	100
walker2d-medium-expert	20	20
walker2d-medium-replay	20	10
walker2d-full-replay	20	20
halfcheetah-random	10	3
halfcheetah-medium	10	3
halfcheetah-expert	10	10	0.1
halfcheetah-medium-expert	10	10
halfcheetah-medium-replay	10	10
halfcheetah-full-replay	10	3
hopper-random	500	50
hopper-medium	500	50
hopper-expert	500	200
hopper-medium-expert	200	50
hopper-medium-replay	200	50
hopper-full-replay	200	50
Franka Kitchen	CQL-Min (N)	SPQR-CQL-Min (N)	SPQR-CQL-Min (β)
kitchen-complete	2	6
kitchen-partial	2	6	2.0
kitchen-mixed	2	6
Antmaze	CQL-Min (N)	SPQR-CQL-Min (N)	SPQR-CQL-Min (β)
umaze	2	6
umaze-diverse	2	6
medium-play	2	3	2.0
medium-diverse	2	3
E Further Experimental Results
E.1 Evaluation Result with More Baselines
Table 9: Normalized average returns of SPQR-SAC-Min, SPQR-EDAC and SPQR-CQL-Min on D4RL Gym, Franka Kitchen, and Antmaze tasks, averaged over 4 random seeds. Except for SPQR*, the evaluated values of baselines are reported. Refer to [2],[20], and [21], and reproduced. Reproduced values are indicated as italic font.
Gym	BC	CQL-Min	EDAC	SAC-Min	SPQR-SAC-Min	SPQR-EDAC
walker2d-random	1.3 ± 0.1	7.0	16.6 ± 7.0	21.7 ± 0.0	24.6 ± 1.1	-
walker2d-medium	70.9 ± 11.0	74.5	92.5 ± 0.8	87.9 ± 0.2	98.4 ± 2.0	94.8 ± 1.0
walker2d-expert	108.7 ± 0.2	121.6	115.1 ± 1.9	116.7 ± 0.4	115.2 ± 1.3	-
walker2d-medium-expert	90.1 ± 13.2	98.7	114.7 ± 0.9	116.7 ± 0.4	118.2 ± 0.7	-
walker2d-medium-replay	20.3 ± 9.8	32.6	87.1 ± 2.3	78.7 ± 0.7	87.8 ± 2.5	89.6 ± 1.8
walker2d-full-replay	68.8 ± 17.7	98.96	99.8 ± 0.7	94.6 ± 0.5	101.1 ± 0.4	102.2 ± 0.2
halfcheetah-random	2.2 ± 0.0	35.4	28.4 ± 1.0	28.0 ± 0.9	33.5 ± 2.5	-
halfcheetah-medium	43.2 ± 0.6	44.4	65.9 ± 0.6	67.5 ± 1.2	74.8 ± 1.3	-
halfcheetah-expert	91.8 ± 1.5	104.8	106.8 ± 3.4	105.2 ± 2.6	112.8 ± 0.3	110.3 ± 0.2
halfcheetah-medium-expert	44.0 ± 1.6	62.4	106.3 ± 1.9	107.1 ± 2.0	114.0 ± 1.6	-
halfcheetah-medium-replay	37.6 ± 2.1	46.2	61.3 ± 1.9	63.9 ± 0.8	69.7 ± 1.4	-
halfcheetah-full-replay	62.9 ± 0.8	82.07	84.6 ± 0.9	84.5 ± 1.2	88.5 ± 0.5	86.8 ± 0.2
hopper-random	3.7 ± 0.6	7.0	25.3 ± 10.4	31.3 ± 0.0	35.6 ± 1.4	-
hopper-medium	54.1 ± 3.8	86.6	101.6 ± 0.6	100.3 ± 0.3	100.2 ± 1.3	103.8 ± 0.1
hopper-expert	107.7 ± 9.7	109.9	110.1 ± 0.1	110.3 ± 0.3	112.0 ± 0.2	-
hopper-medium-expert	53.9 ± 4.7	111.0	110.7 ± 0.1	110.1 ± 0.3	112.5 ± 0.3	111.7 ± 0.0
hopper-medium-replay	16.6 ± 4.8	48.6	101.0 ± 0.5	101.8 ± 0.5	104.9 ± 0.7	-
hopper-full-replay	19.9 ± 12.9	104.85	105.4 ± 0.7	102.9 ± 0.3	109.1 ± 0.4	-
Average	49.9	70.9	85.2	84.5	89.6	-
Franka Kitchen	BC	SAC-Min	BEAR	IQL	CQL-Min	SPQR-CQL-Min
kitchen-complete	33.8	15.0	0.0	62.5	43.8	47.9 ± 12.3
kitchen-partial	33.8	0.0	13.1	46.3	49.8	54.2 ± 7.2
kitchen-mixed	47.5	2.5	47.2	51.0	51.0	55.6 ± 7.9
Average	38.4	5.8	20.1	53.3	48.2	52.6
Antmaze	BC	SAC-Min	BEAR	IQL	CQL-Min	SPQR-CQL-Min
umaze	65.0	0.0	73.0	87.5	74.0	93.3 ± 4.7
umaze-diverse	55.0	0.0	61.0	62.2	84.0	80.0 ± 0.0
medium-play	0.0	0.0	0.0	71.2	61.2	80.0 ± 8.2
Average	40.0	0.0	44.7	73.6	73.1	84.4
E.2 Hypothesis Testing for Uniform Distribution
To check whether bias  $e_{sa}^{i}$  follows a uniform distribution, it's sufficient to check  $Q^{i}(s,a)$  follows a uniform distribution since  $Q^{*}(s,a)$  is constant for fixed  $s,a$ . [27] propose a powerful test, called  $\chi^2$  test which determines whether a given distribution follows uniform or not using hypothesis testing. We perform a  $\chi^2$  test with significant level  $\alpha = 0.025$  on 50 Q-ensemble trained in the D4RL hopper-random dataset in 3M timesteps with 10000 data of D4RL hopper-full-replay dataset. Since the  $\chi^2$  test can be applied in a discrete random variable, we need to construct an approximative probabilistic mass function by binning. In Table 10, we show the acceptance rate of  $\chi^2$  test over 10000 data with different bin sizes. Especially in experiments with 20 bins, SPQR-SAC-Min shows a better acceptance rate than SAC-Min approximately  $40\%$ .
Table 10:  ${\chi }^{2}$  test
Algorithm	bin=10	bin=20	bin=25	bin=30	bin=50
SAC-Min	20.08%	32.12%	36.38%	38.54%	50.99%
SPQR-SAC-Min	33.67%	44.97%	49.78%	51.12%	63.56%
E.3 Conservatism of ensemble and SPQR
We visualize how ensemble size  $N$  affects the conservatism, and how SPQR can be computationally effective. As we described in Section 4.1 and Figure 3, SPQR has conservatism power. Increasing ensemble size  $N$  for SAC-Min shows conservative behavior, as proven in [22] and Figure 9. We train SAC-Min for 3M timesteps in a halfcheetah-random dataset with ensemble size  $N = 3,5,10,15,20,50$ . SAC-Min empirically shows conservative behavior as ensemble size increases. As a result, SPQR can be a computationally efficient algorithm when using SAC-Min.
Figure 9: Mean of predicted Q-value with various  $N$ , averaged over 4 seeds
E.4 Spike histogram visualization
We plot a more detailed visualization of the spike histogram for Figure 10. We visualize how the spike histogram of EDAC and SPQR varies compared with SAC-Min.
Figure 10: Spike visualization for SAC-Min, EDAC, and SPQR-  $\beta = 0.01$  ,0.1,1.0,10.0,100.0.
E.5 Bias of Q
In Figure 11, we plot the average normalized bias and standard deviation of normalized bias. Bias is calculated by Monte-Carlo estimation. SPQR- $*$  decreases average bias and standard deviation of bias from baseline algorithms. For all environments, SPQR-SAC-Ens shows a smaller bias and standard deviation of bias than SAC-Ens. For Hopper and Humanoid tasks, SPQR-REDQ shows underestimation, and for Walker2d and Ant tasks, SPQR-REDQ shows unbiased estimation. For Ant, SPQR-REDQ shows less bias than REDQ.
Figure 11: Average of bias and std of bias for SPQR-SAC-Ens and SPQR-REDQ, averaged over 4 seeds. G represents the UTD ratio.
E.6 UTD Ablation Study
We visualize the performance of SPQR-SAC-Ens over variation of UTD ratio in Figure 12. SPQR-SAC-Ens shows performance improvement by increasing the UTD ratio with a conservative and stable bias.
Figure 12: Left 1: Average return of SPQR-SAC-Ens for UTD ratio 1,5, Left 2: Average of bias of SPQR-SAC-Ens for UTD ratio 1,5, Left 3: Standard deviation of bias of SPQR-SAC-Ens for UTD ratio 1,5. All experiments are averaged over 4 seeds. G represents the UTD ratio.
E.7 Early Collapse
To show how SPQR affects the ensemble when the agent collects new tuple  $(s, a, r, s')$  in terms of exploration and variance, we evaluate the standard deviation with different  $\beta$ .
As [30] pointed out, the early collapse of Q-values occurs, which means that in an early stage of training, the variance of the Q-ensemble is almost zero since Q-values become similar to each other. To show how SPQR addresses the early collapse problem, we visualize the standard deviation of Q-ensemble in Figure 13. We train 10 Q-networks from the MuJoCo Hopper environment with the SAC-Min algorithm with the same hyperparameter. SPQR does not collapse Q-values during training, preserving appropriate variance of Q-value, and higher values of  $\beta$  result in greater variance for  $\beta = 0.3$  and  $\beta = 2.0$ .
Figure 13: Standard deviation of Q-ensemble with different  $\beta$ , average over 3 seeds
E.8 Hyperparameter Sensitivity
We evaluate SPQR over various  $\beta$  to demonstrate hyperparameter sensitivity. We fix  $N = 10$  for SPQR-SAC-Min and evaluate it on the D4RL Gym halfcheetah-random dataset and halfcheetah-medium dataset. Evaluation results are shown in Table 11.
Table 11: Performance of SPQR-SAC-Min over various  $\beta$  on the D4RL halfcheetah-\* dataset, averaged over 4 seeds
dataset	β = 0.0	β = 0.01	β = 0.1	β = 0.5	β = 1.0	β = 2.0	β = 5.0	β = 10.0
random	28.0 ± 0.9	30.6 ± 3.4	32.4 ± 0.4	30.6 ± 1.3	30.7 ± 1.8	31.2 ± 2.2	31.1 ± 1.8	32.3 ± 2.4
medium	67.5 ± 1.2	70.0 ± 0.5	71.7 ± 0.6	70.1 ± 1.0	70.5 ± 1.7	69.2 ± 1.3	70.0 ± 0.9	70.4 ± 1.3
E.9 Compare with MED-RL
We implement and evaluate another diversifying method for online RL tasks, MED-RL. We use the Gini coefficient for MED-RL and use reported hyperparameters with UTD ratio 1. SPQR outperforms MED-RL for all tasks.
Figure 14: Average returns of REDQ, SPQR-REDQ and MED-RL with UTD ratio 1 on the MuJoCo environment, averaged over 4 random seeds.
E.10 t-SNE Visualization
Figure 15 shows t-SNE [32] clustering visualization of Q-ensemble trained 3M timesteps by SPQR-SAC-Min algorithm at halfcheetah-medium-expert. We use t-SNE clustering for the last hidden layer representation of Q-ensemble with ensemble size 10 and 256 batch data at halfcheetah-medium-expert tasks. The t-SNE result shows that Q-ensemble trained by SPQR-SAC-Min is well clustered.
Figure 15: t-SNE for SPQR-SAC-Min at halfcheetah-medium-expert
E.11 Correlation Visualization
We visualize a detailed figure for the Rightmost figure of Figure 1, the Pearson correlation coefficient matrix for a fully trained Q-ensemble and initial Q-ensemble in Figure 16. We train each Q-ensemble in the D4RL hopper-random dataset. For effective visualization, we sample randomly 10 Q-networks among 50 Q-networks and compute the Pearson correlation coefficient between each sampled Q-network with 256 batch data in a hopper-full-replay dataset. For the initial Q-ensemble, Q-networks are uncorrelated from each other. As we train Q-ensemble with SAC-Min for 3M timesteps, Q-networks become strongly correlated with each other.
Figure 16: Left: Q-ensemble trained 0 timesteps. Right: Q-ensemble trained 3M timesteps