SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We analyze the diversity of state-action gathered by ensemble policy on the basis of Shannon entropy  $\mathcal{H}[\cdot]$ , a commonly used diversity measure [32; 34; 51], which is a measure of the dispersion of a distribution. A high entropy of the state-action visit distribution implies that the trajectory distribution is more dispersed, which means more diverse trajectories. Thus, to achieve diverse trajectories, we maximize the expected return while maximizing the entropy of the state-action visit distribution:
$$
\pi^ {*} = \arg \max  _ {\pi \in \Pi} J (\pi) + \alpha \mathcal {H} [ \rho^ {\pi} ] \tag {17}
$$
Where  $\rho^{\pi}$  deduce the state-action visit distribution induced by the ensemble policy  $\pi$  and  $\alpha$  is the weighting factor. The diversity of trajectories gathered by the ensemble policy comes from two
Figure 1: Learning curves for 4 MuJoCo continuous control tasks. For better visualization, the curves are smoothed uniformly. The bolded line represents the average evaluation over 10 seeds. The shaded region represents a standard deviation of the average evaluation over 10 seeds.
Figure 2: Learning curves for 5 continuous control tasks on DMControl suite. For better visualization, the curves are smoothed uniformly. The bolded line represents the average evaluation over 10 seeds. The shaded region represents the standard deviation of the average evaluation over 10 seeds.
components: the discrepancy of sub-policies and the diversity of trajectories gathered by the sub-policies. To illustrate that,
Lemma 1 (Ensemble Sample Diversity Decomposition) Given the state-action visit distribution of the ensemble policy  $\rho$ . The entropy of this distribution is  $\mathcal{H}(\rho)$ . Notice that this term can be decomposed into two parts:
$$
\mathcal {H} (\rho) = \mathbb {E} _ {z} \left[ \mathcal {D} _ {\mathcal {K L}} \left(\rho (s, a | z _ {k}) | | \rho (s, a)\right) \right] + \mathcal {H} (\rho | z) \tag {18}
$$
Where the first term is the state-action visit distribution discrepancy between the sub-policies and the ensemble policy induced by  $\mathcal{KL}$ -divergence measure. The second term implies the diversity of state-action visited by sub-policies which depends on which algorithm is used for the sub-policy, such as TD3 [14], SAC [22], RND [24], etc.
As shown in this inequality,  $\mathcal{H}(\rho)$  is irrelevant of ensemble size  $N$ . Therefore, the ensemble size may improve the diversity of the ensemble policy by influencing the discrepancy of sub-policies indirectly, which implies that the diversity is not guaranteed with the increased ensemble size. Our method increases the discrepancy of sub-policies which theoretically improve the sample diversity of the ensemble policy.
This section presents the experimental setup used to evaluate our proposed TEEN and assesses its performance in the MuJoCo environments.
**Environments.** We evaluate our algorithm on several continuous control tasks from MuJoCo control suite [25] and DeepMind Control Suite [26]. We conduct experiments on 4 control tasks in MuJoCo control suite namely HalfCheetah-v3, Hopper-v3, Walker2d-v3, Ant-v3. In Halfcheetah-v3, the task is to control a cheetah with only one forefoot and one hind foot to run forward. In Hopper-v3, we control a single-legged robot to hop forward and keep balance when hopping. Walker2d-v3 is a bipedal robot environment, where we train the agent to walk or run forward. The target of Ant-v3 is to train a quadrupedal agent to stay balance and move forward. These four environments are all challenging continuous control tasks in MuJoCo. On DeepMind Control Suite, we evaluate our algorithm in 5 environments: cheetah-run, finger-spin, fish-swim, walker-walk, walker-run.
(1) TEEN
(2) TD3
(3) SAC
(4) RND
(5) SUNRISE
Figure 3: Measuring the exploration region. The points represent region explored by each method in 10 episodes. All the states get dimension reduction by the same t-SNE transformation for better visualization.
(c) Performance
(e) Performance
(a) Performance
(b) Average Bias
(d) Average Bias
(f) Average  $\mathcal{H}[\rho]$
Figure 4: TEEN ablation results for Ant environment. The first column shows the effect of ensemble size  $N$  on both performance and estimation bias. The second column shows the effect of target value number  $M$  on both performance and estimation bias. The third column shows the effect of weight parameter  $\alpha$  on both performance and the entropy of trajectories.
Baselines. Beyond simply comparing our approach with the foundational algorithm TD3 [14], a leading-edge deterministic policy gradient reinforcement learning algorithm, we also contrast our method with other efficient exploration algorithms across three distinct categories. We examine curiosity-based exploration baselines, such as RND [24], which promotes efficient exploration through intrinsic curiosity rewards. Regarding maximum entropy-based exploration, we consider SAC [22], an off-policy deep reinforcement learning algorithm that offers sample-efficient exploration by maximizing entropy during training. For ensemble exploration, we look at SUNRISE [16], a unified framework designed for ensemble-based deep reinforcement learning.
Evaluation. We follow the standard evaluation settings, carrying out experiments over one million (1e6) steps and running all baselines with ten random seeds per task to produce the main results. We maintain consistent learning rates and update ratios across all baselines. Further details regarding these parameters can be found in the Appendix B.2. For replication of TD3, SAC and SUNRISE, we use the code the author released for each baseline without any modifications to ensure the performance. For the reproduction of RND, we follow to the repository provided in the original code. For a fair
Table 1: Performance on MuJoCo Control Suite at 500K and 1M timesteps. The results show the average returns with mean and variance over 10 runs. The maximum value for each task is bolded.
500K Step	HalfCheetah	Hopper	Walker2d	Ant
TEEN	10139±623	3548±100	4034±547	5009±649
TD3	8508±601	3330±135	3798±511	4179±809
SAC	9590±419	3332±223	3781±521	3302±798
RND	9185±813	2814±536	2709±1341	3666±685
SUNRISE	4955±1249	3426±99	3782±516	1964±1020
1M Step
TEEN	11914 ± 448	3687 ± 57	5099±513	5930 ± 486
TD3	9759±778	3479±147	4229±468	5142 ± 940
SAC	11129±420	3484±128	4349±567	5084±901
RND	10629±942	3148±143	4197±791	4990±789
SUNRISE	6269±1809	3644±75	4819±398	3523±1430
comparison, we use fully connected network with 3 layers as the architecture of the critic networks and actor networks. The learning rate for each network is set to be  $3e - 4$  with Adam [52] as the optimizer. The implementation details for each baseline can be found in Appendix B.2.
We conducted experiments to evaluate our algorithm's performance in addressing three key research questions (RQ). Specifically, we sought to determine whether TEEN outperforms the current state-of-the-art across multiple environments and achieves a broader range of exploration region, as well as how the ensemble size and discrepancy of sub-policies (defined in Eq. 9) impact TEEN's performance.
RQ1: Whether TEEN outperforms the current state-of-the-art across multiple environments? Table 1 and Table 2 report the average returns with mean and variance of evaluation roll-outs across all algorithms. The learning curves during training on MuJoCo control suite are shown in Figure 1. Learning curves for DeepMind Control suite can be found in Figure 2. On MuJoCo continuous control suite, TEEN shows superior performance across all environments which implies that by ensemble sub-policies with weak exploration performance, such as TD3, the exploration performance of the ensemble policy enhances evenly exceeding algorithms with strong exploration performance, SAC, RND, etc. While purely ensembling multiple models may not certainly improve the performance. Although SUNRISE improves the exploration performance by UCB exploration [53], it does not take effect when the uncertainty shows little relevance to the environment, and even degrades the performance (as shown in HalfCheetah-v3). TEEN can be seen as an immediate solution for diverse exploration within ensemble reinforcement learning by directly enforcing an ensemble policy to discover a broader range of trajectories. We further conduct experiments to validate that TEEN achieves a broader exploration region (Figure 3). For a fair comparison, all the algorithms are trained in Ant-v3 with the same seed at half of the training process. In order to get reliable results, the states explored are gathered in 10 episodes with different seeds. We apply the same t-SNE transformation to the states explored by all of the algorithms for better visualization.
RQ2: How the ensemble size  $N$  and number of target values  $M$  impact TEEN's performance? We make experiments in Ant-v3 environment from OpenAI gym. As shown in Figure 3, when using  $M = 2$  as the number of target values, the estimate of the target value is underestimated and the estimation bias increases with the increase of  $M$ . This underestimation is mitigated with an increase in ensemble size  $N$  resulting in better performance. Thus, we choose 10 to be the ensemble size for all the environments. To enhance reliability, we conducted ablation studies in various MuJoCo
Table 2: Performance on DeepMind Control Suite at 500K and 1M timesteps. The results show the average returns with mean and variance over 10 runs. The maximum value for each task is bolded.
500K Step	Cheetah-Run	Finger-Spin	Fish-Swim	Walker-Walk	Walker-Run
TEEN	837±23	928±22	381±71	970±6	754±33
TD3	784±50	891±24	260±77	968±5	592±87
SAC	791±46	897±30	155±34	972±4	688±55
RND	408±159	894±32	254±42	963±20	180±107
SUNRISE	606±37	885±38	201±34	954±16	406±99
1M Step
TEEN	891±7	976±12	519±79	974±5	800±16
TD3	863±29	965±22	367±113	972±5	683±69
SAC	871±23	971±22	202±50	976±3	746±42
RND	841±35	960±24	385±66	975±3	606±110
SUNRISE	703±26	920±40	285±46	969±4	553±54
environments, obtaining consistent results. The results for other environments can be found in the Appendix C.
RQ3: How the discrepancy of sub-policies impact TEEN's performance? In order to validate the benefits of trajectory-aware exploration, we conduct experiments in the Ant-v3 environment, progressively increasing the weight parameter denoted as  $\alpha$ . We adjust the value of the weight  $\alpha$  from  $[0, 0.2, 0.5, 1]$ . With  $\alpha = 0$ , we show the performance without trajectory-aware exploration. As the results presented in Figure 3, with adequate reward scale, TEEN improves the performance with the use of trajectory-aware exploration. The effectiveness of the approach fluctuates with changes in reward scale. For a large  $\alpha$  value, TEEN aims to enhance the diversity of trajectories. However, as  $\alpha$  increases, TEEN fails to effectively utilize reward signals, resulting in sub-optimal performance. In the experiment, we used  $\alpha = 0.2$  for the MuJoCo control suites. However, the reward scales in the DeepMind control suites are significantly smaller than those in the MuJoCo control suites. In the MuJoCo control suite, the accumulated reward exceeds 5,000 in the Ant environment, while in the DeepMind control suite, the maximum accumulated reward for all tasks is 1,000. As a result, we utilized a smaller  $\alpha = 0.02$  for the DeepMind control suite, considering the smaller reward scale. We also conducted ablation studies with varying reward scales across different environments, please refer to Appendix C for more details.
This paper presents a theoretical analysis of the ensemble policy's diversity exploration problem, based on trajectory distribution. We innovatively discover that the diversity of the ensemble policy can only be enhanced by increasing the Kullback-Leibler (KL) divergence of the sub-policies when entropy is used as the measure of diversity. Guided by this analysis, we introduce the Trajectories-aware Ensemble Exploration (TEEN) method — a novel approach that encourages diverse exploration by improving the diversity of trajectories. Our experiments demonstrate that TEEN consistently elevates the performance of existing off-policy RL algorithms such as TD3, SAC, RND, and ensemble RL algorithms like SUNRISE. There are still important issues to be resolved, e.g., TEEN is sensitive to the reward scale resulting in poor performance without a proper parameter setting. We hope these will be addressed by the community in the future.
This work was funded by the National Key R&D Program of China (2021YFC2800500).