Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control
Authors: Chao Li
Chen GONG
Qiang He
Xinwen Hou
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Ensemble Exploration
Control Tasks
TL;DR: This paper presents a novel approach called Trajectories-awarE Ensemble exploratioN (TEEN) that encourages diverse exploration of ensemble policy.

Abstract: The combination of deep reinforcement learning (DRL) with ensemble methods has been proved to be highly effective in addressing complex sequential decision-making problems. This success can be primarily attributed to the utilization of multiple models, which enhances both the robustness of the policy and the accuracy of value function estimation. However, there has been limited analysis of the empirical success of current ensemble RL methods thus far. Our new analysis reveals that the sample efficiency of previous ensemble DRL algorithms may be limited by sub-policies that are not as diverse as they could be. Motivated by these findings, our study introduces a new ensemble RL algorithm, termed \textbf{T}rajectories-awar\textbf{E} \textbf{E}nsemble exploratio\textbf{N} (TEEN). The primary goal of TEEN is to  maximize the expected return while promoting more diverse trajectories. Through extensive experiments, we demonstrate that TEEN not only enhances the sample diversity of the ensemble policy compared to using sub-policies alone but also improves the performance over ensemble RL algorithms. On average, TEEN outperforms the baseline ensemble DRL algorithms by 41\% in performance on the tested representative environments.

Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control
Chao Li Chen Gong" Qiang He
Xinwen Hou" !
Institute of Automation, Chinese Academy of Sciences, China
University of Virginia, USA
Ruhr University Bochum, Germany University of Chinese Academy of Sciences, China {lichao2021, xinwen. xinwen.hou)@ia.co www.hhhooaaac............ fzv6en@virginia. edu, qianghe97@gaiil.com
Abstract
The combination of deep reinforcement learning (DRL) with ensemble methods has been proved to be highly effective in addressing complex sequential decision- making problems. This success can be primarily attributed to the utilization of multiple models, which enhances both the robustness of the policy and the accuracy of value function estimation. However, there has been limited analysis of the empirical success of current ensemble RL methods thus far. Our new analysis reveals that the sample efficiency of previous ensemble DRL algorithms may limited by sub-policies that are not as diverse as they could be. Motivated by these findings, our study introduces a new ensemble RI. algorithm, termed Trajectories-awark Ensemble exploratioN (TEEN). The primary goal of TEEN is to maximize the expected return while promoting more diverse trajectories. Through extensive experiments, we demonstrate that TEEN not only enhances the sample diversity of the ensemble policy compared to using sub-policies alone but also improves the performance over ensemble RL algorithms. On average, TEEN outperforms the baseline ensemble DRL algorithms by 41 % in performance the tested representative environments.
1 Introduction
Deep Reinforcement Learning (DRL) has demonstrated significant potential addressing range of complex sequential decision-making problems, such video games 2 3 4 autonomous driving 5 (6) robotic control tasks 7 8) board games [9] [0], etc. Although the combination of high-capacity function approximators, such as deep neural networks (DNNs), enables DRL to solve more complex tasks, two notorious problems impede the widespread use of these DRL in real-world domains. i) function approximation error: Q-learning algorithm converges to sub-optimal solutions due to the error propagation [II] 12 13). c.g., the maximization of random function approximation errors accumulate into consistent overestimation bias [14. ii) sample inefficiency, model-free algorithm is notorious for requiring sample -eeestttttraiiingg of neural networks requires large number of samples which is hard acquire in real-world scenarios.
Ensemble reinforcement learning shows great potential solving the issues mentioned above by combining multiple models of the value function and (or) policy [14] improve the accuracy and diversity. For instance, Max-Min Q-learning [II] reduces estimation bias by utilizing an ensemble of estimates and selecting the minimum value as the target estimate.
13
16
17
18
19 20) to
Bootstrapped DQN [15] and SUNRISE [16 train an ensemble of value functions and policies,
"Coresponning Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

leveraging the uncertainty estimates of value functions to enrich the diversity of learning experiences. However, we claim that existing ensemble methods do not effectively address the crucial issue of diverse exploration, which is essential for enhancing sample efficiency. Although randomly initializing sub-policies shares some similarities to adding random noise with the ensemble policy, is not sufficient for significantly improving the sample diversity of the ensemble policy. Thus, how to enhance the exploration diversity of ensemble policy remains an open question.
This paper presents a novel approach called Trajectories-awarE Ensemble exploratioN (TEEN) that encourages diverse exploration of ensemble policy by encouraging diverse behaviors through explo- ration of the state-action visit distribution measure space. The state-action visit distribution measure quantifies the frequency with which a particular state-action pair is visited when using a certain policy, Thus, policies with diverse state-action visit distributions induce different trajectories [2]. Moreover, value functions are learned from the trajectories generated by the policy, rather than the actions the policy may take. While previous research [22
23 24 has emphasized promoting diverse actions, this approach does not always lead to diverse trajectories. This paper highlights the importance of diverse trajectories in improving sample efficiency, which TEEN achieves.
Our contributions can be summarized in three aspects. (I) We propose Trajectories-awarB Ensemble exploratioN (TEEN) algorithm, a highly sample-efficient ensemble reinforcement learning algorithm. TEEN trains sub-policies to exhibit greater diversity of behaviors, accomplished by considering the distribution of trajectories. (2) Theoretical analysis confirms that our algorithm facilitates more diverse exploration by enhancing the varied behaviors of the sub-policies. (3) Extensive experimental results show that our method outperforms or achieves a similar level of performance as the current state-of-the-art across multiple environments, which include both MuJoCo control 25 tasks and DMControl tasks [26). Specifically, on average across the tasks we investigated, TEEN surpasses the baseline ensemble DRL algorithm by 41% and the state-of-the-art off-policy DRL algorithms by 7.3% in terms of performance. Furthermore, additional experiments demonstrate that our algorithm samples from a diverse array of trajectories.
2 Related Work
Ensemble methods have been widely utilized in DRL to serve unique purposes. Overestimation bias in value function severely undermines the performance of Q-learning algorithms, and a variety of research endeavors on the ensemble method have been undertaken to alleviate this issue [8 19 (7 11, (28). TD3 [4] clips Double Q-learning to control the overestimation bias, which significantly improves the performance of DDPG (29] Maxmin Q-learning [I]] directly mitigates the overestimation bias by using a minimization over multiple action-value estimates. Ensemble methods also play role in encouraging sample efficient exploration (30] For example, Bootstrapped DQN [5] leverages uncertainty estimates for efficient (and deep) exploration. On top of that, REDQ 117 enhances the sample efficiency by using update-to-data ratio >> 1. SUNRISE [16 turns to an ensemble policy to consider for compelling exploration.
Ensemble Methods.
Diverse Policy. A range of works have been proposed for diverse policy, and these can generally be classified into three categories [3] The first category involves directly optimizing policy diversity during pre- pre-training without extrinsic rewards. Fine- fine-tuning with rewards is then applied to complete the downstream tasks. These methods (32 B3) 34 351 train the policy by maximizing the mutual information between the latent variable and the trajectories, resulting in multiple policies with diverse trajectories. The second category addresses constraint optimization problems, where either diversity is optimized subject to quality constraints, or the reverse is applied quality is optimized subject to diversity constraints [36, 37 38 39]. The third category comprises quality diversity methods [31] 40 41), which prioritize task-specific diversity instead of task-agnosti diversity. These methods utilize scalar functions related to trajectories and simultaneously optimize both quality and
diversity.
Efficient Exploration. Efficient exploration methods share similar methods to diverse policy methods in training the agent, wherein exploration by diverse policies can enhance the diversity of experiences, ultimately promoting more efficient exploration. [42]. Other prior works focusing on efficient exploration consider diverse explorations within single policy, with entropy-based methods being the representative approach [2) 43). These methods strive to simultaneously maximize the expected cumulative reward and the entropy of the policy, thus facilitating diverse exploration. Additionally,
2

---

another representative category of research is the pariosity-driiver method 224 23 44]. These methods encourage the agent to curiously explore the environment by quantifying the state with "novelty". such as state visit counts 23 model prediction error [24], etc.
Preliminaries
3.1 Markoy Decision Process
Markov Decision Process (MDP) can be described as tuple (S, (S,A,P,r P.r.7) 7), where and A represent the state and action spaces, respectively. The function P maps a state-action pair to the next state with a transition probability in the range [0, 1], i.e., P: S A x S - [0, 1]. The reward function r S x A + IR denotes the reward obtained by performing an action in a particular state Additionally, 7 E (0, 1) is the discount factor. The goal of reinforcement learning algorithms is to find a policy x² that maximizes the discounted accumulated reward J (TT) defined as:
((x) Es De (iron a:)]
(1)
3.2 Overestimation Bias in Deep Deterministic Policy Gradient
The Deep Deterministic Policy Gradient (DDPG) algorithm
45 29] forms the foundation of various
continuous control tasks such as TD3 [14 and SAC 22 By enabling policy output produce deterministic actions, DDPG offers a powerful algorithmic solution to these tasks. DDPG utilizes two neural networks: the actor and critic networks. The actor network is responsible for generating the policy, while the critic network evaluates the value function. In deep reinforcement learning (DRL), the policy [[ass) and the value function Q(s, a) are expressed by deep neural networks parameterized with Φ and respectively. DDPG suggests updating the policy through deterministic policy gradient, as follows:
VaJ(o) E.pP, VaQ (s, (a a))a=n(s) V (x)
In order approximate the discounted cumulative reward (as specified in EI. the function (s,, a) is updated by minimizing the temporal difference (TD) errors [46] between the estimated value of the next state s' and the current state
0" = arg min E (r(s. a) + 7Z (x, as)
In policy evaluation, Q function is used approximate the value by taking action in state s. The Q function is updated by Bellman equation [47] with bootstrapping, where we minimize the expected Temporal-Differenc Difference (TD) [46] error 8(s, a) between value and the target estimate in a mini-batch of samples B,
8(s, a) = Eg ((s.a) a) + max Q(5' a') Q().
(4)
This process can lead to a consistent overestimation bias as the holding of Jensen's ' inequality Specifically, the overestimation of value evaluation arises due to the following process:
48
Eg [max Q(s', ,a')a ]] max Ex(((s' a'EAA
(5)
e'
4 Methodology
This section describes Trajectories-awarE Ensemble exploratioN (TEEN) method that achieves diverse exploration for agents. TEEN aims to enforce efficient exploration by encouraging diverse behaviors of an ensemble of N sub- sub-policies parameterized by [ TEEN initially formulates a discrepancy measure and enforces the discrepancy among the sub-policies while simultaneously maximizing expected return. Next, we present how to solve this optimization problem indirectly by utilizing the mutual information theory. Finally, we conduct theoretical analysis to ensure whether the diversity of the samples collected by the ensemble policy is enhanced by solving this optimization problem. We summarize TENN in Algorithm I

---

4.1 Discrepancy Measure
We assess the diverse behaviors of policies by examining the diversity of trajectories generated by their interactions with the environments. The state-action visit distribution, indicating the probability of encountering a specific state-action pair when initiating the policy from a starting state, encapsulates the diversity of these trajectories directly. Thus, TEEN measures the diverse behaviors of policies by directly assessing their state-action visit distribution. Formally, given an ensemble of N sub-policies #N we p* induce the date-action visit distribution deduced by policy and use for the ensemble policy. We have,
To; #1,-..; ...; Tky
Σ
p T
For convenience. use the conditioned state-action visit probability, denoted as p(s, a|2), to represent the respective state-action visit distribution of each sub-policy. This can be expressed as follows,
p The (s, 0) p(s,a|z))
Where { is the latent variable representing each policy. Our discrepancy measure K.C-divergence an asymmetric measure the difference between two probability distributions. Given this discrepancy measure, we define the difference of policies as the K.C-divereeece of the conditioned state-action visit distribution with the state-action visit distribution.
Definition 1 (Difference between Policies) Given an ensemble of N policies { Wl,..., TT, .... TN the difference between policy m, and other policies can be defined by the conditioned state-action occupation discrepancy:
D -1 [lp] Dxc [p(s, al al2x))ll((ss)) a)]
(8)
Consequently, we improve the difference between the conditioned state-action visit distribution to enforce diverse behaviors of sub-polices. While directly optimizing the Eq. (8) can be hard: Obtaining the state-action visit distribution can be quite challenging. In response to this, Liu, H., al., as referenced in citation [49], have suggested the use of K-Nearest Neighbors (KNN) as approximate measure for this distribution. We notice that can be expressed by the form of entropy discrepancy:
E, [Dxc (p(s, dl=)]lp(ss, a))] = H(pp) - H(ppz)
(9)
In which the H[- is the Shannon Entropy with base e, H(p) is the entropy of the state-action visit distribution. With the mutual information theory, we can transform this optimization target as follows:
H(p) H(p==) H()) H((pp)
(10)
Thus, we turn to this equivalent optimization target. The first term encourages high entropy of p(2). We fix p(2) to be uniform in this approach by randomly selecting one of the sub-policies to explore. We have H(2) = N EN log)(>*) log N, which is constant. The second term emphasises the sub-policy is easy to infer given state-action pair. We approximate this posterior with a learned discriminator qç( (=|s,, a) and optimize the variation lower bound.
H(=) H({pp) log N + E.,.. (log p(=ss p(=)s,a))
log N E,,a (pxc (2 a) a)) (zls, + E. (log ( (2(s,
(11)
2 log N E..a. (log (z))
Where we use non-negativity of KL-divergence, that is Dic Dxclp(z]s, (2)gg a)] with respect to parameter Ç to tighten the variation lower bound:
We minimize
v (eEe,a a) (z|s, a))] -E VV log q: (2)
(12)
Consequently, we have the equivalent policy gradient with regularizer:
π² arg max (TT) adE (, (z|s, a))]
(13)

---

4.2 Trajectories-aware Ensemble Exploration
We select TD3 14] as the algorithm of sub-policies which is not a powerful exploration algorithm. We then show how to ensemble algorithm with poor exploration performance and encourage efficient ensemble exploration by solving this optimization problem. We consider an ensemble of N TD3 agents i.e., {Qa,, , t k== where 02 and Φu: denote the parameters of k-th Q-function and sub-policy. Combined with deterministic policy gradient method gradient with regularizer.
29 we update the each sub-policy by policy
V Vrotaa((nn) = Ex- Va (Q" a) a log &e (zx a)) a))la (o) V Toy (3)]
(14)
Recurrent Optimization. Updating all sub-policies every time step in parallel may suffer from the exploration degradation problem [35. Consider state-action pair (s, a1), which is frequently visited by sub-policy ти and another state-action pair (s, a2) which is visited by multiple sub sub-policies such that we have (x( s, a1) 9, 9<(2 (s a2). Under this circumstance, the constraint encourage sub-policy тa to execute action while preventing from executing action 02 which leads to the problem that some explored state-actions are prevented from being visited by the sub- sub-policies. To tackle this challenge, we employ recurrent training method for our sub-policies. In particular, given an ensemble N sub-policies, we randomly select a single sub-policy, denoted as тk, every T episodes. We then concentrate our efforts on regulating the behavior of the selected sub-policy, rather than all sub-policies simultaneously.
Gradient Clip. TEEN uses (ogp q,(2s, a))l as a constraint to optimize the policy, while the gradient of this term is extremely large when the probability (c(2|8, a) is small. And state-action with small probability to infer the corresponding sub-policy z implies that the state-action is rarely visited by sub-policy z but frequently visited by other sub-policies. The target of the constraint is to increase the discrepancy of the sub-policies, while making the sub-policy visit this state-action will instead reduce the discrepancy. Further, for a state-action with large probability &: [2[2s, a), continuing to increase this probability will prevent the sub-policy from exploring other possible state aate-actions. Thus, we use the clipped constraint and the main objective we propose is the following:
well = arg max J(x + a., [log clip(g (z|s, a),s, -))
(15)
where is a hyper-parameter, and we set e = 0.1 for all of our experiments.
4.3 Controlling the Estimation Bias
In Q-value estimation, share some similarities with Max-Min DQN | and Averaged DQN [50] While Max-Min DQN takes the minimum over an ensemble of value functions as the target en- couraging underestimation, Averaged DQN reduces the variance of estimates by using the mean of the previously learned Q-values. We combine these two techniques and apply both mean and min operators to reduce overestimation bias and variance. Instead of using the mean of the previously learned Q-values, we use the mean value on N actions of N sub-policies. TEEN then uses the minimum over a random subset M of M Q-functions as the target estimate.
Strrget = r(8, +
min 1=1,2...,,
Σ (((x, 1=1
(16)
N
We give some analytical evidences from the perspective of order statistics to show how we control the estimation bias by adjusting M and N.
Theorem 1. Let X1, X2,... ..., XN be an infinite sequence of i.i.d. random variables with a probability density function (PDF) of f(x) and cumulative distribution function (CDF) of F(z). Denote = E and x² = v[X,]- Let X < X2.N < X..N I XNAN be the order statistics corresponding {X,)N. Denote PDF and CDF of the k-th order statistic Xa.N as feI and FREN respectively. The following statements hold.
ii p (N-1)a S E[X S S4, N4,N1L E[ S [[X V2N-I
5

---

Algorithm Trajectories-Awar Ensemble Exploration algorithm (TEEN)
1: Initialize N policy parameters 6k. N Q-function parameters kk. & = 1, 1,2, N
2: Initialize target Q-function parameters 6% 02, 1, 1.2.... N.
3: Initialize discriminator
4: Initialize empty replay buffer D
5: Initialize recurrent interval d
6: for each iterations do
7: Randomly sample policy TT
8: for each time step do
9: Choose one action with noise Tox (s) +E, (N(,~
10:
Collect state 81+1 and reward r, from the environment by taking action az
Store transition T (st, at, st+1. in replay buffer +- T.
12:
end for
13:
if t mod d then
14:
Randomly select one sub-policy Té.
end if
16:
for each gradient step do
17:
Sample random minibatch B = {T=] ~ D Randomly select set M of M distinct indices from {1, J..
18:
19:
Compute the Q target y by Eq. (T6)
20:
For all of the value functions, update 0g using gradient decent
21:
0o
Σ
Qy-Qo (s,
(s.0,
For sub-policy Tे. update o, by gradient decent
23:
V Φe B (B,0,,,"")BB
(a log (: (2)ls,))) (s, (s)
Update target Q-function by 8% p0p + (1 (1- p)0 25: Update discriminator qe by Eq.
26:
end for
27:
end for
(i) Let NEN
LEN X then, E[X] p4 Var[X] NN 0
Points 1 of the Theorem indicates that the minimum value, which implies that we can control the estimation from over estimates to under or proper estimates. The second point indicates that the mean over an ensemble of values reduces the variance. Thus, by combining these two operators, we control the estimation bias and the variance.
ensemble of values reduces the expected
4.4 Theoretical Analysis
We analyze the diversity of state-actions gathered by ensemble policy on the basis of Shannon entropy H[-). commonly used diversity measure (32, 34 [5T), which is measure the dispersion of distribution. A high entropy of the state-action visit distribution implies that the trajectory distribution more dispersed, which means more diverse trajectories. Thus, to achieve diverse trajectories, we maximize the expected return while maximizing the entropy of the sate-action visit distribution:
arg max J (T) + aHHp""]
(17)
well
Where nd deduce the state-action visit distribution induced by the ensemble policy æ and the weighting factor. The diversity of trajectories gathered by the ensemble policy comes from two

---

TD3
SAC
SUNRISE
Hal/Cheefab--v3
4000
Hopper-s3
Walker2d-c3
Average Return
6000
10000
4000
5000
2000
2000
0.5
1.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
Time Steps (1c6)
Time Steps (1c6)
Time Steps (1c6)
Time Steps (1c6)
Figure I: Learning curves for MuJoCo continuous control tasks. For better visualization,the curves are smoothed uniformly, The bolded line represents the average evaluation over 10 seeds. The shaded region represents a standard deviation of the average evaluation over 10 seeds.
TEEN
TD3
SAC
RND
SUNRISE
cheetob: run
finger spin
flst sista
walkee-wllk
Walker-rua
Averags Return
1000
400
500
50I
500
200
150
0.5
0.5 Time Steps (1.6)
1.0
Time Steps (1c6)
Time Steps (1e6)
Time Steps (1e6)
Timse Steps (1e6)
Figure 2: Learning curves for 5 continuous control tasks DMControl suite. For better visualization, the curves are smoothed uniformly. The bolded line represents the average evaluation over 10 seeds. The shaded region represents the standard deviation of the average evaluation over 10 seeds.
components: the discrepancy of sub- sub-policies and the diversity of trajectories gathered by the sub- policies. To illustrate that,
Lemma (Ensemble Sample Diversity Decomposition) Given the state-action visit distribution of the ensemble policy p. The entropy of this distribution H()). Notice that this term can be decomposed into two parts:
(18) Where the first term the state-action visit distribution discrepancy between the sub-policies and the ensemble policy induced by K.C-divergence measure. The second term implies the diversity of state-action visited by sub-policies which depends on which algorithm is used for the sub-policy, such as TD3 [1]. SAC (22), RND [24), etc.
H(pp) = E, [,,[(((((s, (la))) + H(ppz)
As shown in this inequality, H(P) is irrelevant of ensemble size N. Therefore, the ensemble size may improve the diversity of the ensemble policy by influencing the discrepancy of sub-policies indirectly, which implies that the diversity is not guaranteed with the increased ensemble size. Our method increases the discrepancy of sub-policies which theoretically improve the sample diversity of the ensemble policy.
5 Experimental Setup
This section presents the experimental setup used to evaluate our proposed TEEN and assesses its performance in the MuJoCo environments.
Environments. We evaluate our algorithm on several continuous control tasks from MuJoCo control suite [25] and DeepMind Control Suite (26). We conduct experiments on control tasks in MuJoCo control suite namely HalfCheetah-v3, H3pper-v3, Walker2d-v3, Ant-v3. In Halfcheetah-ve -33, the task to control cheetah with only one forefoot and one hind foot run forward. Hopper-v3, we control single-legged robot to hop forward and keep balance when hopping. Walker2d-v3 is bipedal robot environment, where we train the agent to walk or run forward. The target of Ant-v3 is to train quadrupedal agent to stay balance and move forward. These four environments are all challenging continuous control tasks MuJoCo. DeepMind Control Suite, we evaluate our algorithm in 5 environments: chectah-run. finger-spin, fish-swim, walker-walk, walker-run.
7

---

(I) TEEN
(2) TD3
(3) SAC
RND
(5) SUNRISE
Figure 3: Measuring the exploration region. The points represent region explored by each method in 10 episodes. All the states get dimension reduction by the same t-SNE transformation for better visualization.
6000
6000
6000
TEEN.-00.6
TEEN@-0.2
5000
TEEN.-
Average Return
Average Return
Avurage Return
TEINo-10
4000
3000
3000
3000
2000
TEEN(M-LN-2)
2000
TEENOM-2N-10)
2000
TEEN(M-LN-S.
TEENIM I0)
1000
TEENIM-2N-10)
TEENIM-4N-10)
0
TD3
TD3
0.00
0.00
1.00
0.50
0.75
Time Steps (le6)
Time Steps (le6)
Time Steps (1e6)
(a) Performance
(c) Performance
(e) Performance
Average Normolized Bins
0.2
Average Normalizzd Bias
1.0
Averige H(p)
-0.4
TELN.0002
-0.8
TEENM 2N
-0.8
PRENMM-3NN
a--5
TEENM-ZN-10)
TRENM-4N-100
TELN,-118
0.4
0
0.25
0.50
0.75
1.00
0.80
0.25
0.50
0.75
1.0
0.00
0.25
0.50
0.75
1.00
Time Steps (1e6)
Time Steps (1c6)
Time Steps (1e6)
(b) Average Bias
(d) Average Bias
(f) Average H[pp]
Figure 4: TEEN ablation results for Ant environment. The first column shows the effect of ensemble
size N both performance and estimation bias. The second column shows the effect of target value number M on both performance and estimation bias. The third column shows the effect of weight parameter on both performance and the entropy of trajectories.
Baselines. Beyond simply comparing our approach with the foundational algorithm TD3 [4] leading-edge deterministic policy gradient reinforcement learning algorithm, we also contrast our method with other efficient exploration algorithms across three distinet categories. We examine curiosity-based exploration baselines, such as RND 124] which promotes efficient exploration through intrinsic curiosity rewards. Regarding maximum entropy-based exploration, we consider SAC [22] off-policy deep reinforcement learning algorithm that offers sample-efficien exploration by maximizing entropy during training. For ensemble exploration, we look at SUNRISE [16), unified framework designed for ensemble-based deep reinforcement learning.
Evaluation. We follow the standard evaluation settings, carrying out experiments over one million (1e6) steps and running all baselines with ten random seeds per task to produce the main results. We maintain consistent learning rates and update ratios across baselines. Further details regarding these parameters can be found in the Appendx B.2 For replication of TD3, SAC and SUNRISE, we use the code the author released for each baseline without any modifications to ensure the performance. For the reproduction of RND, we follow the repository provided in the original code. For fair

---

Table I: Performance on MuJoCo Control Suite at 500K and 1M timesteps. The results show the average returns with mean and variance over runs. The maximum value for each task is bolded.
500K Step
HaliChectah
Hopper
Walker2d
Ant
TEEN
601
3548=100
4034::547
5009:1699
TD3
85081601
3330
3798:151
417911809
SAC
959011419
3332.1223
3781
3302.1798
RND
9183=813
2814::36
2709#1341
3666=1685
SUNRISE
4955=1249
3426#99
3782=516
1964=1020
IM Step
TEEN
11914 448
3687 57
509915533
5930 + 486
TD3
9759=778
147
4229+-468
5142 + 940
SAC
11129++420
3484+128
4349+:567
5084+901
RND
10629-9442
144
4197:1911
4990-1889
SUNRISE
6269-1809
3644175
48191139
3523 1430
comparison, we use fully connected network with layers as the architecture of the critic networks and actor networks. The learning rate for each network is set to be 3e 4 with Adam [52] as the optimizer. The implementation details for each baseline can be found in Appendx B.2
6 Experimental Results
We conducted experiments to evaluate our algorithm's performance addressing three key research questions (RQ). Specifically, we sought to determine whether TEEN outperforms the current state- of-the-art across multiple environments and achieves boarder range of exploration region, as well as how the ensemble size and discrepancy of sub-policies (defined in Eq. 9) impact TEEN's performance.
RQI: Whether TEEN outperforms the current state-of-the-art across multiple environments? Table and Table report the average returns with mean and variance of evaluation roll-outs all algorithms. The learning curves during training on MuJoCo control suite are shown in Figure] Learning curves for Deep.Mnnd Control suite can be found in Figure On MuJoCo continuous control suite, TEEN shows superior performance across all environments which implies that by ensemble sub-policies with weak exploration performance, such as TD3, the exploration performance of the ensemble policy enhances evenly exceeding algorithms with strong exploration performance, SAC, RND, etc. While purely ensembling multiple models may not certainly improve the performance. Although SUNRISE improves the exploration performance by UCB exploration [53]. it does not take effect when the uncertainty shows little relevance to the environment, and even degrades the performance (as shown in HalfCheetah-v3). TEEN can be seen an immediate solution for diverse exploration within ensemble reinforcement learning by directly enforcing an ensemble policy to discover broader range of trajectories. We further conduct experiments to validate that TEEN achieves a boarder exploration region (Figure D.. For a fair comparison, all the algorithms are trained in Ant-v3 with the same seed at half of the training process. In order to get reliable results, the states explored are gathered in 10 episodes with different seeds. We apply the same t-SNE transformation to the states explored by all of the algorithms for better visualization.
RQ2: How the ensemble size N and number of target values M impact TEEN's performance? We make experiments in Ant-v3 enviromment from OpenAl gym. As shown in Figure 3 when using M as the number of target values, the estimate of the target value underestimated and estimation bias increases with the increase of M. This underestimation is mitigated with increase in ensemble size N resulting in better performance. Thus, we choose 10 to the ensemble size for all the environments. To enhance reliability, conducted ablation studies in various MuJoCo

---

Table 2: Performance on DeepMind Control Suite at 500K and 1M timesteps. The results show the average returns with mean and variance over runs. The maximum value for each task is bolded.
500K Step
Cheetah-Run
Finger-Spin
Fish-Swim
Walker-Wan Walk
Walker-Run
TEEN
837--23
928++22
381+71
970+-6
754+33
TD3
784-150
891:124
260.1.7
968.1.5
592.187
SAC
791-146
897.130
155:134
972.14
688.155
RND
159+159
894#:32
254::42
963+22
180
SUNRISE
606:137
885
201
954#16
406
IM Step
TEEN
89117
9761:12
519:179
974:55
800:1:66
TD3
863++299
965+-22
367++113
972++
683++69
SAC
871+23
971#-22
202+:50
9764+3
746++42
RND
841:135
9601124
3851166
975113
606:1: 110
SUNRISE
703:126
9201140
2851446
96914
5531154
environments, obtaining consistent results. The results for other environments can be found in the Appendix C
RQ3: How the discrepancy of sub-policies impact TEEN's performance? In order to validate the benefits of traee exploration, conduct experiments in the Ant-v3 environment, progressively increasing the weight parameter denoted as a. We adjust the value of the weight a from [0, 0.2, 0.5, 1]. With a = O, show the performance without trajectory-aware exploration. As the results presented in Figure3i with adequate reward scale, TEEN improves the performance with exploration. The effectiveness of the approach fluctuates with changes in
the use of trajectory-awart reward scale. For a large value. TEEN aims to enhance the diversity of trajectories. However, as increases, TEEN fails effectively utilize reward signals, resulting in sub-optimal performance.. In the experiment, we used = 0.2 for the MuloCo control suites. However, the reward scales in the DeepMind control suites are significantly smaller than those in the MuJoCo control suites. In the MuloCo control suite, the accumulated reward exceeds 5,000 in the Ant environment, while in the DeepMind control suite, maximum accumulated reward for all tasks 1,000. a result, we utilized smaller a 0.02 for the DeepMind control suite, considering the smaller reward scale. We also conducted ablation studies with varying reward scales across different environments, please refer to Appendix C for more details.
Conclusion
This paper presents a theoretical analysis of the ensemble policy's diversity exploration problem, based on trajectory distribution. We innovatively discover that the diversity of the ensemble policy can only be enhanced by increasing the Kullback-Leibler (KL) divergence of the sub-policies when entropy is used the measure of diversity. Guided by this analysis, introduce the Trajectories- aware Ensemble Exploration (TEEN) method - a novel approach that encourages diverse exploration by improving the diversity of trajectories. Our experiments demonstrate that TEEN consistently elevates the performance of existing off-policy RL algorithms such as TD3, SAC, RND, and ensemble RL algorithms like SUNRISE. There are still important issues be resolved, e.g., TEEN sensitive to the reward scale resulting in poor performance without proper parameter setting. We hope these will be addressed by the community in the future.
Acknowledgments
This work was funded by the National Key R&DD Program of China (2021YFC2800500)
10

---

References
[1] Sutton, R. S., A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[2] Mnih, V., K. Kavukcuoglu, D. Silver, et al. Human-level control through deep reinforcement learning. nature, 518(7540))522-533 2015.
[3] Mnih, V., A. P. Badia, M. Mirza, et al. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, vol. 48 of JMLR Workshop and Conference Proceedings, pages 1928-1937. PMLR, 2016.
[4] Gong, C., Y. Bai, X. Hou, et al. Stable training of bellman error in reinforcement learning. In Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18-22, 2020, Proceedings, Part V 27. pages 439-448. Springer, 2020.
[5] Ye, Y., X. Zhang, J. Sun. Automated vehicle's behavior decision making using deep rein- forcement learning and high-fidelity simulation environment. Transportation Research Part C: Emerging Technologies, 107:155- 170, 2019.
[6] Gong, C., Z. Yang, Y. Bai, et al. Mind your data! hiding backdoors in offline reinforcement learning datasets. arXiv preprint arXiv:2210.04683 2022.
[7] Andrychowicz. M., F. Wolski, A. Ray, et al. Hindsight experience replay. Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017.
[8] Lillierap, T. P., Hunt, A. Pritzel, et al. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
[9] Silver, D., T. Hubert, Schrittwieser, et al. general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 11111140011100 144, 2018.
[10] Silver, D., J. Schrittwieser, K. Simonyan, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-33) 359, 2017.
[11] Lan, Q. Y. Pan, A. Fyshe, et al. Maxmin q-learning: Controlling the estimation bias of learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[12) He, Q., X. Hou. WD3: taming the estimation bias in deep reinforcement learning. In 32nd IEEE International Conference on Tools with Artificial Intelligence, ICTAI 2020, Baltimore, MD, USA, November 9-11, 2020, pages 391-398. IEEE, 2020.
[13] He, Q., T. Zhou, M. Fang, et al. Figensubspace of temporal-different dynamics and how it improves value approximation in reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Datahases, vol. 14172 of Lecture Notes in Computer Science, pages 573-589. Springer, 2023.
[14] Fujimoto, S., H. Hoof, D. Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML, 2018, vol. 80 of Proceedings of Machine Learning Research, pages 1582-1591, PMLR, PMLR, 2018.
[15] Osband, I., C. Blundell, A. Pritzel, et al. Deep exploration via bootstrapped dqn. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 4033-400 4041. Curran Associates Inc., 2016.
[16] Lee, K., M. Laskin, A. Srinivas, et al. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning. pages 613
PMLR, 2021.
[17] Chen, X., C. Wang, Z. Zhou, et al. Randomized ensembled double q-learning: Learning fast without a model. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.

---

[18] Hasselt, H. v., A. Guez, D. Silver. Deep reinforcement learning with double q-leamnngg In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI 16, page 2094-2100. AAAI Press, 2016.
[19] Hasselt, H. Double q-learning. In Advances Neural Information Processing Systems, vol. 23. Curran Associates, Inc., 2010.
[20] He, Q. H. Su, J. Zhang, al. Frustratingly easy regularization on representation can boost deep reinforcement learning. In IEFF/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 20215-20225. IEEE, 2023.
[21] Zahavy, T., Y. Schroecker, F. Behbahani, et al. Discovering policies with domino: Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205 2022.
[22] Haarnoja, T., A. Zhou, P. Abbeel, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, vol. 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, PMLR, 2018.
[23] Pathak, D., P. Agrawal, A. A. Efros, et al. Muriosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning, vol. 70 of Proceedings of Machine Learning Research, pages 2778-28 2787. PMLR, 2017.
[24] Burda, Y., H. Edwards, J. Storkey, et al. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[25] Todorov, E., T.. Erez, Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages
2012.
[26] Tassa, Y., Y. Doron, A. Muldal, et al. Deepmind control suite. arXiv preprint arXiv: /01.00090.0000 2018.
[27] Wu, D., X. Dong, J. Shen, al. Reducing estimation bias via triplet-average deep deterministic policy gradient. IEEE Transactions on Neural Networks and Learning Systems, 31 (11)):433 4945, 2020.
[28] Wei, W., Y. Zhang, J. Liang, et al. Controlling underestimation bias in reinforcement learning via quasi-median operation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8621-8628, 2022.
[29] Silver, D., Lever, N. Heess, al. Deterministic policy gradient algorithms. In Proceedings of the 31 International Conference on International Conference on Machine Learning Volume 32. ICML' 14, page 1-387-1-395. JMLR.org, 2014.
[30] Faußer, S., F. Schwenker. Neural network ensembles in reinforcement learning. Neural Processing Letters, 41:55-69 69, 2015.
[31] Wu, S., J. Yao. H. Fu, et al. Quality-similas diversity via population based reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[32] Eysenbach, B., A. Gupta, J. Ibarz, al. Diversity is all you need: Learning skills without a reward function. In 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net, 2019.
[33] Laskin, M., H. Liu, X. B. Peng, et al. Unsupervised reinforcement learning with contrastive intrinsic control. Advances in Neural Information Processing Systems, vol. 35, pages 34478-34491. Curran Associates, Inc., 2022.
[34] Sharma, A., S. Gu, S. Levine, et al. Dynamics-aware unsupervised discovery of skills. In 8th International Conference Learning Representations, ICLR 2020, OpenReview.net, 2020.
12

---

[35] Jiang, Z., Gao. J. Chen. Unsupervised skill discovery via recurrent skill training. In Advances in Neural Information Processing Systems, vol. 35, pages 39034-39046. Curran Associates, Inc., 2022.
[36] Kumar, S., A. Kumar, S. Levine, et al. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198-822 -2100 2020.
[37] Zhou, Z., W. Fu, B. Zhang, et al. Continuously discovering novel strategies via rewtrd-switching policy optimization. arXiv preprint arXiv:2204.02246, 2022.
[38] Sun, H., Z. Peng. B. Dai, et al. Novel policy seeking with constrained optimization. arXiv preprint arXiv:2005. 10696, 2020.
[39] Masood, M., F. Doshi-Velez. Diversity-idducing policy gradient: Using maximum mean discrepancy find a of diverse policies. Proceedings of the Twenty-Eighng International Joint Conference Artificial Intelligence, IJCAI-19, pages 5923-5929. International Joint Conferences on Artificial Intelligence Organization, 2019.
[40] Fontaine, M., S. Nikolaidis. Differentiable quality diversity. In Advances in Neural Information Processing Systems, vol. 34, pages 10040-10052. Curran Associates, Inc., 2021.
[41] Hokker, J., A. Pacchiano, K. M. Choromanski, et al. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:18050- 18062, 2020.
[42] Li, C., C. Gong, Q- He, et al. Centralized cooperative exploration policy for continuous control tasks. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 2454-2456. 2023.
[43] Haarnoja, T., H. Tang, P. Abbeel, et al. Reinforcement learning with deep -baeeg policies. Proceedings of the 34th International Conference on Machine Learning, vol. of ICML'17, page 1352-1361. JMLR.org, 2017.
[44] Gong, C., Yang, Y. Bai, et al. Curiosity-driven and vietim-aware adversarial policies. In Proceedings of the 38th Annual Computer Security Applications Conference, ACSAC '22, page 186-200. Association for Computing Machinery, 2022.
[45] Silver, D., G. Lever, N. Heess, al. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on International Conference on Machine Learning Volume 32, ICML' 14, page 1-387-1-395 JMLR.org, 2014.
[46] Sutton, R. S. Learning to predict by the methods of temporal differences. Machine Learning. 3(1):9-44) 44, 1988.
[47] Bellman, R. Dynamic programming. Science, 153(3731):34-37, 1966.
[48] Kuznetsov, A., P. Shvechikov, A. Grishin, et al. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. the International Conference on Machine Learning (ICML). pages 5556-5566. PMLR, 2020.
[49] Liu, H., P. Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 6736-6747. PMLR, 2021.
[50] Anschel, O., N. Baram, N. Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference machine learning, pages 176-185. PMLR, 2017.
[51] Zhao, Y., Q. Ye, W. Wu, et al. Generative prompt model for weakly supervised object localiza- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6351-6361 2023.
[52] Kingma, D. P., J. Ba. Adam: A method for stochastic optimization. ICLR (Poster). 2015.
[53] Wang, Y., K. Dong, X. Chen, et al. Q-learning with ueb exploration is sample efficient for infinite-horizon mdp. In International Conference on Learning Representations.
[54] David, H. A., H. N. Nagaraja. Order statistics. John Wiley & Sons, 2004.
13


Reviews:
[{"review_id": "DUMrLdnXYy", "reviewer": "Reviewer_C6dk", "summary": "The paper presents an efficient ensemble learning strategy termed Trajectories-awarE Ensemble exploratioN (TEEN), which promotes exploration by diversifying the state-action visitation distribution of multiple sub-policies. The proposed method introduces a discrepancy measure that tries to maximize the difference in entropy of estimating the state-action visitation with and without the information of the policy (represented by the latent variable) and vice versa. Such a discrepancy is approximated by learning a discriminator that tries to differentiate between state-action pairs visited by different policies. And the log-likelihood of the discriminator is used as a regularizer for policy improvement along with maximizing the expected return of the policy. The authors employ additional strategies of updating a single policy at a time and clipping the discriminator log probabilities to ensure proper exploration. Finally, the estimation bias is controlled by calculating the target value as the minimum of the mean of the ensemble Q-values for each sub-policy. The choices are theoretically motivated and the algorithm is analyzed in simulation on Mujoco Control and DM Control Suite benchmarks.", "questions": "1. How do you decide the actions while inference? \n2. Can you clarify more about the weakness pt.2?\n3. Section C is missing in the Supplementary.\n4. I am unsure, but it seems that TEEN uses very high random exploration steps as mentioned in the supplementary. Do you use same for all the other algorithms?\n5. If the regularization is already clipped, why is there a coefficient ($\\alpha$) required? Can a comparison be shown between these two cases? Also, $\\alpha=0$ means that there is only a change in Q-value computations as compared to TD3, hence there must be a result for all the environments with $\\alpha=0$ and $\\alpha=0.2$ to show a more convincing contribution of the regularization term. Currently, it is only shown for one env (Ant) and it is particularly not clear how much the regularization term is contributing to the exploration in addition to the ensemble of sub-policies.\n6. The timesteps shown in the results are timesteps for individual sub-policy or the whole ensemble i.e. (1M/10 = 100k steps for single sub-policy)?", "limitations": "The authors have mentioned the limitation as the tuning of $\\alpha$. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TEEN provides new insights into using the measure of the state-action visitation distribution to promote exploration along with maximizing returns in policy optimization. \n\n2. Each design choice made is properly justified: namely updating one sub-policy at a time and clipping the regularizer gradients.", "weaknesses": "1. There might be states not captured by the state-action visitation of one particular sub-policy (say $\\pi_a$). In that case, the contribution of $\\rho^{\\pi_a}(s,a)$ to $\\rho$(s,a) in eq (6) can be highly noisy and misinformative (garbage probability). Giving equal weightage to state-action visitation of every policy while calculating for the ensemble might be misleading.\n\n2. It seems that the latent variable plays a key role in defining the algorithm and the discriminator. However, there is no discussion about the choice of $z_k$ and training of the discriminator in the paper or in Alg 1. How do you choose $z_k$ and how does the training take place of the discriminator? If done parallelly with the policies, the framework follows a min-max update rule, shown to be unstable in prior works like GAIL. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the your detailed feedback and insightful suggestions, please see the following for our response.\n\n> W1: There might be states not captured by the state-action visitation of one particular sub-policy(say $\\pi_{a}$). In that case, the contribution of $\\rho^{\\pi_{a}}(s,a)$ to $\\rho(s,a)$ in eq(6) can be highly noisy and misinformative (garbage probability). Giving equal weightage to state-action visitation of every policy while calculating for the ensemble might be misleading.\n\nA1: If a state is not captured by the state-action visitation of one particular sub-policy, this unexplored state gives equal large weightage to every policy and further enforces sub-policies to explore this unexplored state instead of misleading them through the reward function restricts policies to not explore aimlessly. Once the unexplored state is captured by the policy, there is no more misleading, as training proceeds.\n\n> W2: It seems that the latent variable plays a key role in defining the algorithm and the discriminator. However, there is no discussion about the choice of $z_k$ and training of the discriminator in the paper or in Alg 1. How do you choose z_k and how does the training take place of the discriminator? If done parallelly with the policies, the framework follows a min-max update rule, shown to be unstable in prior works like GAIL.\n\nA2: We sample $z_k$ from uniform distribution every 5k steps during recurrent training which can be shown in line 152 *“We fix p(z) to be uniform in this approach by randomly selecting one of the sub-policies to explore. We have $H(z) = −\\frac{1}{N} \\sum^{N}_{k=1} logp(z_k) ≈ logN$, which is a constant.”* Here, we fix $p(z)$ to be a prior distribution to guarantee that H(z) is a constant. Otherwise, there must be a more complicated form of the regular term If the distribution of z constitutes a posterior distribution through feedback. Uniform distribution means that the sub-policies are homogenous contributing equally to the ensemble exploration. As you understand, we train the discriminator parallelly with the policies, the update equation can be shown in Eq.(12) and we would add this part to our Alg.1 in the final version. The update of our algorithm is not a pure min-max update rule. TEEN updates the policy from both value function and the discriminator that you mentioned. The value function can help stabilize the gradient of policy and the exploration process further stabilizing our discriminator. \n\n\n> Q1: How to decide the actions while inference?\n \nA1: Check the response for W2.\n\n> Q2: Can you clarify more about the weakness pt.2?\n\nA2: Check the response for W2.\n\n> Q3: Section C is missing in the Supplementary.\n\nA3: Sorry about that, we upload some parts of Supplementary  C in the general rebuttal.\n\n> Q4: I am unsure, but it seems that TEEN uses very high random exploration steps as mentioned in the supplementary. Do you use same for all the other algorithms?\n\nA4: The random starting exploration time steps are set to be $2.5\\times10^{4}$ in this submission. And this is a standard hyper-parameter [1], which keeps consistent for all the tested algorithms in our submission.\n\n> Q5: If the regularization is already clipped, why is there a coefficient (α) required? Can a comparison be shown between these two cases? Also, α=0 means that there is only a change in Q-value computations as compared to TD3, hence there must be a result for all the environments with $\\alpha=0$ and $\\alpha=0.2$ to show a more convincing contribution of the regularization term. Currently, it is only shown for one env (Ant) and it is particularly not clear how much the regularization term is contributing to the exploration in addition to the ensemble of sub-policies.\n\nA5: The reason we do regular term cropping is that when p in log p is small (e.g. 0.001), it will have a large gradient which will cause instability in the optimization process, whereas the coefficient alpha is designed to do a balance between exploration and exploitation, a higher coefficient alpha means that the agent is encouraged to explore the environment. See the PDF released in the general rebuttal for the evaluation when α=0.\n\n> Q6: The timesteps shown in the results are timesteps for individual sub-policy or the whole ensemble i.e. (1M/10 = 100k steps for single sub-policy)?\n\nA6: Yes. The timesteps shown are for the whole ensemble, which is 1M/10 = 100k steps for a single sub-policy if we use 10 sub-policies.\n\n[1] https://github.com/sfujim/TD3/blob/master/main.py"}, {"review_id": "7Taur3k1HE", "reviewer": "Reviewer_aBo4", "summary": "This paper presents an algorithm called trajectories-aware ensemble exploration (TEEN) for sample efficient ensemble reinforcement learning. The authors point out that the existing ensemble methods do not effectively address the required diversity in exploration, which TEEN is designed to tackle better. It achieves this by encouraging diverse behaviors through exploration of the state-action visit distribution measure space. Theoretical results are also presented to show how the design principles of TEEN could encourage diversity in exploration, while the experiment results demonstrate its empirical success in popular benchmark tasks. \n\n", "questions": "1. Shouldn’t the equation 11 be $\\log(N) - \\mathbb{E}_{s,a,z}[-\\log(\\rho(z|s,a))]$? \n2. Also, shouldn’t the LHS and RHS of equation 11 be approximately equal and not equal?\n3. Define $\\mu$ and $\\sigma$ in theorem 1. \n4. While I did not check the proofs carefully, it appears the authors make simplification assumptions in the proof of theorem 1 that is not stated in the theorem (ex: zero-mean distribution). Please state them properly in the theorem. \n", "limitations": "Please discuss the limitations of the work, as pointed out earlier. This will help readers better appreciate the work. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This manuscript presents an interesting thesis that sample efficient ensemble reinforcement learning methods require diverse exploration in sub-policies. The authors discuss this topic in detail and systematically approach the proposed solution. I view the finding of the diversity requirement of sub-policies coupled with the provided theoretical analysis and the limited yet promising experimental results as the main strength of the work. I, therefore, believe if properly executed, this can be a valuable contribution to the community. \n\n", "weaknesses": "One of the main limitations of the work is its limited evaluations. The authors evaluate the proposed TEEN method to answer three main research questions. But I find their evaluations lacking the experimental rigor that is needed to fully appreciate the proposed method. \n\nFirst, in RQ1 in evaluating the performance against baselines, the proposed method shows visible benefits in Mujoco tasks, which is promising. But my concerns are that all four tasks evaluated may contain similar exploration challenges. Given exploration is one of the key focuses of the work, I would like to see the performance benefits across different tasks that pose different exploration challenges. Perhaps this is what the authors are attempting to do with DeepMind control experiments, but despite the mention, I did not find such experiments in the paper or in the appendix (Appendix C is empty).\n\nRegarding RQ2 and RQ3, my concerns are with the fact that it is only based on a single task (Ant-v3). For both questions, it is not significant enough to make claims based on a single task result. Furthermore, for RQ3, while I never found the results on the DeepMind control suite, the authors mention the best $\\alpha$ is 0.02. This is a very low $\\alpha$ and almost as having no trajectory-aware exploration. This further questions my concerns about how generalizable the proposed method is across different tasks. Clarification is appreciated. \n\nOverall for experiments, authors use only TD3 as the learning algorithm. I would like to see if the observed performance gains are dependent on the choice of the learning algorithm. It is also fine if there is an observable reliance, but in that case, I would like to see the contributions rephrased accordingly. \n\nI presume training the proposed model is challenging, specifically when it comes to finding the best ensemble size N and number of target values M (due to the trial and error approach in finding them, as in RQ2). It also appears that this training would require considerable computations. I would like to see a discussion of these limitations (and other limitations) of the method. \n\nIf the stated limitations are addressed, particularly related to the evaluation process, the proposed work can be of benefit to the community. Consequently, I would be inclined to reconsider my score if the authors can effectively rectify these issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our heartfelt appreciation for the invaluable feedback you provided on our submission. Your insightful comments and suggestions have been instrumental in refining our research. We clarify your concerns as follows.\n\n> W1: First, in RQ1 in evaluating the performance against baselines, the proposed method shows visible benefits in Mujoco tasks, which is promising. But my concerns are that all four tasks evaluated may contain similar exploration challenges. Given exploration is one of the key focuses of the work, I would like to see the performance benefits across different tasks that pose different exploration challenges. Perhaps this is what the authors are attempting to do with DeepMind control experiments, but despite the mention, I did not find such experiments in the paper or in the appendix (Appendix C is empty).\n\nA1: When evaluating our algorithm on MuJoCo tasks, we took that concern into account as well and that’s why we attempt to do with DeepMind control experiments as you infer. In Table 2, We report the average returns with mean and variance of evaluation roll-outs across all algorithms on DMControl Tasks with 5 other tasks. The results show that our algorithm far exceeds the baseline in a very small number of steps in most tasks. The learning curves for these tasks have been moved to Appendix.C.  Sorry for that, and check the general rebuttal for the missing part.\n\n> W2: Regarding RQ2 and RQ3, my concerns are with the fact that it is only based on a single task (Ant-v3). For both questions, it is not significant enough to make claims based on a single task result. \n\nA2: Regarding RQ2 and RQ3, the experiments for other environments have been moved to the appendix as we have mentioned in the main paper. We are sorry for the incomplete appendix, and check the general rebuttal for the missing part. And results are also consistent with our theoretical analysis.\n\n> W3: For the DeepMind control suite, the authors mention the best $\\alpha$ is 0.02. ......  Clarification is appreciated.\n\nA3: We use a small $\\alpha$ due to the small reward scale of the DeepMind control suite whereas in MuJoCo, the reward scale is 5 to 10 times of the DeepMind control suite. In the MuJoCo control suite, the accumulated reward exceeds 5k in Ant while in the DeepMind control suite, the max accumulated reward for all tasks is 1000. That’s why we use 0.2 for MuJoCo Control tasks while 0.02 for the DeepMind control suite encountering a small reward scale. And the ablation results also show that with higher alpha (0.5), our algorithm show better performance in HalfCheetah-v3 in which the reward scale is 2 times of Ant-v3.\n\n> W4: Overall for experiments, authors use only TD3 as the learning algorithm. I would like to see if the observed performance gains are dependent on the choice of the learning algorithm. It is also fine if there is an observable reliance, but in that case, I would like to see the contributions rephrased accordingly.\n\nA4: Yes, the gains are dependent on the choice of the learning algorithm. As we pointed out in Lemma 1 (Ensemble Sample Diversity Decomposition), where the ensemble diversity $\\mathcal{H}(\\rho)$ can be decomposed into two parts:\n\n$\\mathcal{H}(\\rho)=\\mathbb{E}_{z}$\n\n$[\\mathcal{D_{KL}}(\\rho(s,a|z_k)||\\rho(s,a))] + \\mathcal{H}(\\rho|z) $\n\nThe first part is the state-action visit distribution discrepancy between the sub-policies and the ensemble policy induced by the KL-divergence measure and is our optimization target. While the second part implies the diversity of state-action visited by sub-policies which depends on which algorithm is used for the sub-policy. Thanks for your valuable suggestion, we’d include this in our contribution. \n\n> W5: I presume training the proposed model is challenging, specifically when it comes to finding the best ensemble size N and number of target values M (due to the trial and error approach in finding them, as in RQ2). It also appears that this training would require considerable computations. I would like to see a discussion of these limitations (and other limitations) of the method.\n\nA5: From the experimental results (More complete in the PDF of the general rebuttal.), Increasing the target value M will lead to underestimation and increasing the number of N can mitigate this. Thus, if we want to have a more accurate value function, a smaller M (no smaller than 2.) and a larger N is recommended.  However, a larger ensemble size can lead to considerable computations, and in most cases, N=10 is enough to make the value estimate more accurate. To alleviate the computational pressure, we adopt network with multi-head structures[1] and we compare our computational cost with TD3, SAC and an ensemble method-SUNRISE. The results can be found in personal rebuttal for R4. It takes about twice as long as what SAC takes for N=10 and one fifth of what SUNRISE takes.\n\n> Q1: Shouldn’t eq. 11 be $\\log (N)−\\mathbb{E}_{s,a,z}[−log(\\rho(z|s,a))]$?\n\nA1: Sorry for this typo.\n\n> Q2: Also, shouldn’t the LHS and RHS of equation 11 be approximately equal and not equal?\n\nA2: We are trying to optimize the equation on the left, but the direct solution is not a good estimator of $p(s | z)$, so we transform it into a way to increase the lower bound of the equation while decreasing the gap between the lower bound and the true value of, see Eq. 12. \n\n> Q3: Define μ and σ in theorem 1.\n\nA3: Thanks for your advice, we consider $X_i\\sim\\mathcal{N}(\\mu, \\sigma)$, and we will add this definition in theorem 1 in the new version.\n\n> Q4: While I did not check the proofs carefully, it appears the authors make simplification assumptions in the proof of theorem 1 that is not stated in the theorem (ex: zero-mean distribution). Please state them properly in the theorem.\n\nA4: Thanks for your advice, we assume that $X_{i}\\sim \\mathcal{N}(\\mu, \\sigma)$, and we will add this assumption in theorem 1 in the new version.\n"}, {"review_id": "EyTdiXJBqT", "reviewer": "Reviewer_gTUk", "summary": "This paper presents a new ensemble RL algorithm. The main motivation of this work is to increase the variation between sub-policies, which is measured by the distance between the corresponding state-action visitation distribution. It formulates the loss function by connecting the visitation distribution distance to the mutual information theory and introduces a discriminator function to measure the policy variation. It also proposes a few tricks to improve the exploration effectiveness of the proposed formulation. The method is experimented on a few Mujoco and Deepmind control suite tasks and compared against several off-policy RL methods and one ensemble RL method.", "questions": "1. Is the statement for the overestimation problem in DDPG properly demonstrated? Eq(4) (line 113) seems to be the formulation for discrete action space while this paper and DDPG focus on continuous action space. Please refer to [[9]](https://arxiv.org/pdf/1802.09477.pdf) for details.\n\n2. It is unclear what the latent variable $z_k$ is at the beginning when it is first presented. Could be more clear if it just says $z_k$ is a categorical index or one-hot encoding of the sub-policy selection.\n\n3. Line 146 mentions using KNN to measure Eq (8) but does the proposed approach indeed use it? If so, where is it applied?\n\n4. Eq (9) is an important step to connect state-action visitation distribution distance to mutual information. Any references or derivation for Eq (9)?\n\n5. The Recurrent Optimization trick (Line 169) is adopted to prevent sub-policy from exploration degradation. If I understand correctly, it corresponds to Line 18-19 in Algorithm 1 where only the sampled policy $\\pi_k$ is updated. However, the gradient of the loss function (Eq (14)) should be independent among sub-policies. Could the authors explain how will the sub-policy $\\pi_{\\phi_k}$ change if we update all sub-policies instead of just $\\pi_{\\phi_k}$ in Algorithm 1 (Line 18-19).\n\n6. **Algorithm 1** helps understand the method while there is no reference to Algorithm 1 throughout the paper.\n\n7. How is the discriminator function $q$ trained? I could not find any details about it.\n\n8. How does the presented algorithm perform on more complex control tasks? The current evaluated tasks are relatively simple that do not require a sophisticated amount of exploration of the policy. It would be more convincing if the approach can be benchmarked on Humanoid and Dextrous hand manipulation tasks.\n\n9. The appendix is incomplete.", "limitations": "The limitations have been discussed above (i.e. it is unknown about the performance of the presented approach on complex tasks).", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is motivated by an intuitive idea that increasing the trajectory variation of the sub-policies may improve the exploration and performance of the ensemble policy. \n\n2. The results show the proposed method consistently outperforms baselines across a wide span of tasks in different benchmarks.\n\n3. The experimental analysis is helpful to understand the secret sauce behind the proposed method and the effect of different hyper-parameters.", "weaknesses": "1. The exposition of the paper need improvement (more in the next section)\n\n2. The evaluated control problems are relatively simple and the approach needs to be evaluated on more complex tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our sincere gratitude for your insightful feedback on our submission. Your valuable comments have greatly contributed to enhancing the quality of our work. In response to your suggestions, we have carefully revised the manuscript to address the points you raised.\n\n> Q1: Is the statement for the overestimation problem in DDPG properly demonstrated? Eq(4) (line 113) seems to be the formulation for discrete action space while this paper and DDPG focus on continuous action space. Please refer to [9] for details.\n\nA1:  Overestimation bias occurs for similar reasons in both discrete and continuous environments, although the types of action spaces differ. We use a more explicit way of expressing overestimation bias in continuous action space and this expression is also recognized and used (eg. TQC [2]). Please check [1] and [2] for more information. We will give a more appropriate elaboration in a subsequent version.\n\n\n\n> Q2: It is unclear what the latent variable $z_k$ is at the beginning when it is first presented. Could be more clear if it just says $z_k$ is a categorical index or one-hot encoding of the sub-policy selection.\n\nA2: In our code, $z_k$ is a categorical index to select k-th sub-policy，(eg. $z=1,[\\pi_1, \\pi_2, \\pi_3]=\\\\pi_1$). While the sub-policies can also be got by one-hot encoding as an input to the network. We don’t restrict how the sub-policy is obtained by $z_k$ because there are lots of ways to construct which is also a question worth exploring. \n\n> Q3: Line 146 mentions using KNN to measure Eq (8) but does the proposed approach indeed use it? If so, where is it applied?\n\nA3: We did not use this approach as we illustrate in lines 144-148. Estimating probability densities using KNN is computationally intensive and does not apply to continuous states.  In this paper, we avoided the problem of going to a direct solution for this difficult-to-estimate distribution by transforming Eq. using mutual information theory. The equations obtained in this way are precise and simpler to implement. \n\n> Q4: Any references or derivation for Eq (9)?\n\nA4: The Proof can be found in Appendix A. Where,\n\n$\\mathcal{H}(\\rho)=\\mathbb{E}_{(s,a)\\sim \\rho}[-\\log(\\rho(s,a))]$\n\n$=\\mathbb{E}_{(s,a,z)\\sim \\rho}\\left[\\log \\frac{\\rho(s,a|z)}{\\rho(s,a)}-\\log \\rho(s,a|z)\\right]$\n\n$= \\mathbb{E}_{z}$\n\n$[\\mathcal{D_{KL}}(\\rho(s,a|z)||\\rho(s,a))]+ \\mathcal{H}(\\rho|z)$\n\n\n\n> Q5: The Recurrent Optimization trick (Line 169) is adopted to prevent sub-policy from exploration degradation. If I understand correctly, it corresponds to Lines 18-19 in Algorithm 1 where only the sampled policy $π_k$ is updated. However, the gradient of the loss function (Eq (14)) should be independent among sub-policies. Could the authors explain how will the sub-policy $\\pi_{\\phi_{k}}$ change if we update all sub-policies instead of just π_{ϕ_{k}} in Algorithm 1 (Line 18-19).\n\nA5: if we update all sub-policies instead of just $\\pi_{\\phi_{k}}$ in Algorithm 1, We need a larger and gradually decaying coefficient alpha, and the results of this approach are much more brilliant. However, since the decay form of alpha is difficult to determine, thus, we used the Recurrent Optimization trick to avoid this problem which allows us to use a fixed coefficient $\\alpha$ to get great behavior in most of the tasks.\n\n\n> Q6: Algorithm 1 helps understand the method while there is no reference to Algorithm 1 throughout the paper.\n\nA6: Thanks for your advice, we will add this reference at the appropriate place in the paper.\n\n> Q7: How is the discriminator function q trained? I could not find any details about it.\n\nA7: we train the discriminator parallelly with the policies, the update equation can be shown in Eq.(12) and we would add this part to our Alg.1 in the final version. Thanks.\n\n> Q8: How does the presented algorithm perform on more complex control tasks? The current evaluated tasks are relatively simple that do not require a sophisticated amount of exploration of the policy. It would be more convincing if the approach can be benchmarked on Humanoid and Dextrous hand manipulation tasks.\n\nA8: In this paper, we evaluate our algorithm in different exploration challenges from the MuJoCo control suite to the DeepMind Control suite. The experimental results show our algorithm improves both the performance and sample efficiency which reveals that these tasks do need more exploration and are still promising with a sophisticated amount of exploration.  But we'd love to challenge our algorithm in more complex benchmarks. Thus, we conduct experiments in the 376-dimensional Humanoid-v3, which is exceptionally difficult to solve with off-policy algorithms. We run an experiment in 5 million (5e6) time steps in 5 seeds. The results show that our algorithm exceeds the baseline algorithm consistently. We report the average returns with mean and variance of evaluation roll-outs here and the learning curves can be found in global rebuttal.\n\n| 5M Step | TEEN | TD3 |  SAC |\n| --- | :---: | :---: | :---: | \n| Humanoid-v3 | $8259.49\\pm429.32$ | $6957.91\\pm364.05$ |$7641.03\\pm 261.63$|\n\n> Q9: The appendix is incomplete.\n\nA9: Sorry for that, we update some parts of the missing appendix in the general rebuttal.\n\n[1]Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993. \n\n[2] Kuznetsov A, Shvechikov P, Grishin A, et al. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics[C]//International Conference on Machine Learning. PMLR, 2020: 5556-5566.\n"}, {"review_id": "FxjuTjQYBk", "reviewer": "Reviewer_vN15", "summary": "This paper addresses the problem of exploration in ensemble-based reinforcement learning. It introduces a novel approach called Trajectory Aware Ensemble Exploration (TEEN) that aims to enhance exploration by increasing diversity among the sub-policies. This diversity is measured by the KL divergence between the distributions of the trajectories. The method achieves diversity by optimizing the variational lower bound, which is then incorporated into the policy gradient step with the addition of a regularizer term.\n\nTo reduce estimation bias on the Q function, the method selects the minimum value from a random set of critics based on the average Q value of the actions chosen by all sub-policies. The paper provides theoretical analysis to support this design choice, demonstrating that it reduces both estimation bias and variance.\n\nThe effectiveness of TEEN is validated through experimental results on MuJoco and Deepmind Control Suite. These results show that the proposed method successfully explores a wide range of states and surpasses the performance of existing DRL algorithms in continuous control, including those specifically designed for efficient exploration.\n", "questions": "Please refer to the weakness section.", "limitations": "- The authors have discussed one limitation regarding the sensitivity about reward scaling in Sec. 7.\n- Could the authors comment on the computational overhead of the ensemble?\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- It proposes a new method that improves the exploration in ensemble-based RL by increasing the diversity of the sub-polices in terms of their state-action visit distribution.\n- The implementation is straightforward and involves two steps on top of a DDPG-like method:\n  - add a regularizer to the policy gradient\n  - add a routine to select and update sub-policies and to compute the target Q value\n- It provides theoretical analysis to justify the choice of target Q value\n- Experimental results are presented to validate the efficacy of the method.\n", "weaknesses": "- There appears to be an inconsistency between Line 18 in Algorithm 1 and the text. Specifically, Line 18 suggests that all sub-policies are updated sequentially, while the \"recurrent optimization\" section indicates that only one sub-policy is updated in each gradient step. Could you clarify this discrepancy?\n\n- Theorem 1 demonstrates that the method reduces the expected Q value. However, the experimental results indicate that the algorithm tends to underestimate the Q value. It's unclear whether underestimation is preferable to overestimation. If possible, it would be beneficial to address this aspect in the analysis.\n\n- The main paper frequently refers to Appendix C, but it appears to be missing from the appendix section. \n\n- In Line 436 of the appendix, it seems that the last expression is missing a multiplier p(s). Could you confirm?\n\n- There's a mismatch between Lemmas 2 and 3 in the appendix and the content presented in the main paper.\n\n- The proof for (v) in Lemma 3 does not show the variance inequality: V[X_{1:{N+1}}] \\geq V[X_{1:{N}}].\n\n- Formatting Issue:\n\n  - The readability could be improved if Table 2 used the same row/column heading as those in Table 1.\n  - Appendix line 483: Table index is missing.\n\n- Grammar:\n  - Line 151: we can “increase” this equivalent optimization target. “maximize” seems more appreciate here. \n  - Line 272: “.While purely ensemble multiple models may not certainly improve the performance.“ \n  - Line 273: “it does not take effect“, “evenly degrading”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed feedback and insightful suggestions, please see the following for our response.\n\n> Q1: There appears to be an inconsistency between Line 18 in Algorithm 1 and the text. Specifically, Line 18 suggests that all sub-policies are updated sequentially, while the \"recurrent optimization\" section indicates that only one sub-policy is updated in each gradient step. Could you clarify this discrepancy?\n\nA1: Thanks for pointing this out. This line is incorrectly stated, we do update only one sub-policy in each gradient step with recurrent optimization. We will modify this part of the statement in the algorithm in the next version.\n\n> Q2: Theorem 1 demonstrates that the method reduces the expected Q value. However, the experimental results indicate that the algorithm tends to underestimate the Q value. It's unclear whether underestimation is preferable to overestimation. If possible, it would be beneficial to address this aspect in the analysis. \n\nA2: This is an open question. Underestimation may not always be preferable to overestimation for all tasks. However, in general, overestimation bias is usually considered more harmful than underestimation bias [1,2,3]. This is because usually overestimation bias directly destroys the learning of an intelligent body in severe cases [3], while underestimation bias can be mitigated by visiting the corresponding state-action pairs multiple times [1]. When dealing with underestimation bias, it is easy to introduce overestimation bias [1,2]. We follow the general way of dealing with overestimation bias. Our concern is how to make the value function estimate closer to the true estimate in this submission.\n\n> Q3: The main paper frequently refers to Appendix C, but it appears to be missing from the appendix section.\n\nA3: Sorry for that, and check the general rebuttal for the missing part.\n\n> Q4: In Line 436 of the appendix, it seems that the last expression is missing a multiplier $p(s)$. Could you confirm?\n\nA4: Sorry for this typo. We present the full proof of Lemma 2 here.\n\nLemma 2\n\nProof.\n\nBy definition,\n\n$\\mathcal{I}(\\rho;z) = \\mathcal{H}(\\rho) - \\mathcal{H}(\\rho|z) = \\mathcal{H}(z) - \\mathcal{H}(z|\\rho)$\n\n\nBy randomly selecting the latent variable $z$, we consider that $\\mathcal{H}(z)$ is a constant depending on the number of $z$. Thus, we have,\n\n$\\mathcal{H}(z|\\rho) = \\mathcal{H}(z) + \\mathcal{H}(\\rho|z) - \\mathcal{H}(\\rho)$\n\n$\\\\propto \\\\mathbb{E}_{(s,a,z)\\\\sim \\\\rho(s,a,z)}[-\\\\log{\\\\rho(s,a|z)}]$\n\n$- \\\\mathbb{E}_{(s,a)\\\\sim \\\\rho(s,a)}[-\\\\log \\\\rho(s,a)]$\n\n$=\\mathbb{E}_{(s,a,z)\\sim \\rho(s,a,z)}[-\\log \\rho(s,a|z)] - \\int -\\rho(s,a)\\log \\rho(s,a) dsda$\n\n$=\\mathbb{E}_{(s,a,z)\\sim \\rho(s,a,z)}[-\\log \\rho(s,a|z)] - \\int -\\rho(s,a,z)\\log \\rho(s,a) dsdadz$\n\n$=\\mathbb{E}_{(s,a,z)\\sim \\rho(s,a,z)}[\\log \\rho(s,a)-\\log \\rho(s,a|z)]$\n\nWhere, $\\rho(s,a)=\\int \\rho(s,a|z)p(z) dz = \\frac{1}{n}\\sum_{k=1}^{n} p(s,a|z_k)$.\n\nThen, \n\n$\\mathcal{H}(z|\\rho) = \\mathbb{E}_{(s,a,z)\\sim \\rho(s,a,z)}[-\\log \\rho(z|s,a)]$\n\n$= \\mathcal{H}(z) + \\mathcal{H}(\\rho|z) - \\mathcal{H}(\\rho)$\n\n$\\\\propto \\\\mathbb{E}_{(s,a,z) \\\\sim \\\\rho(s,a,z)}$\n\n$\\[\\\\frac{1}{n}\\\\sum_{k=1}^{n}\\\\rho(s,a|z_k)-\\\\log \\\\rho(s,a|z)\\]$\n\n$\\propto - \\mathcal{D_{KL}}\\left[\\rho(s,a|z)||\\frac{1}{n}\\sum_{k=1}^{n} \\rho(s,a|z_k)\\right].$\n\n> Q5: There's a mismatch between Lemmas 2 and 3 in the appendix and the content presented in the main paper.\n\nA5: It's true that some of the symbols don't match the main paper, we'll fix that part, hopefully, this won't affect your understanding of this submission. For Lemma 3, we put some of the key formulas in the main paper which leads to an inconsistency.\n\n> Q6: The proof for (v) in Lemma 3 does not show the variance inequality: $V[X_{1:{N+1}}] \\geq V[X_{1:{N}}]$.\n\nA6: Sorry about that. For some unknown reasons, our latest version was not uploaded correctly to the submission system. Our main body of this submission does not involve this result and we will remove this part in a subsequent version. Hopefully, this will not affect your understanding of our article.\n\n> Q7: Formatting Issue and Grammar.\n\nA7: Thanks for your advice. We will update these issues in the subsequent version.\n\n>L2: The computational overhead of the ensemble.\n\nA2: On the issue of computational cost, we use network with multi-head structure [4]. And with the use of recurrent training, our computational costs are highly reduced. To convinced that we calculate the training time over one epoch in HalfCheetah-v3 and compared with TD3, SAC and an ensemble method-SUNRISE. We list the results here. All the experiment are done in TITAN RTX 3060 with one single task mode.\n\n|Method | Epoch 1(s/Epoch)| Epoch 2 (s/Epoch)|Epoch 3 (s/Epoch)|Epoch 4 (s/Epoch)|Epoch 5 (s/Epoch)|\n|---|---|---|---|---|---|\n|TEEN (N=5) | 19.37| 18.35| 19.32| 17.49| 16.81|\n|TEEN (N=10) |18.35| 16.67|19.14 |19.09 |18.25 |\n|TEEN (N=15) |19.32 |18.89 |18.96 |19.84 |19.31 |\n|TD3|5.24 |4.68 |5.4 |5.63 |4.88 |\n|SAC| 11.74|8.51 |8.43 |8.44 |9.23 |\n|SUNRISE (N=5)|106.06 |105.9 |106.24 |102.29 |109.94 |\n\n \n\n[1] Kuznetsov A, Shvechikov P, Grishin A, et al. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics[C]//International Conference on Machine Learning. PMLR, 2020: 5556-5566.\n\n[2] Fujimoto, Scott, Herke Hoof, and David Meger. \"Addressing function approximation error in actor-critic methods.\" International conference on machine learning. PMLR, 2018.\n\n[3] Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning[C]//Proceedings of the AAAI conference on artificial intelligence. 2016, 30(1).\n\n[4] Osband I, Blundell C, Pritzel A, et al. Deep exploration via bootstrapped DQN[J]. Advances in neural information processing systems, 2016, 29."}]

Author rebuttal:
We greatly appreciate the reviewer's thorough reading and detailed comments about our submission. Your constructive reviews strengthen our draft. And some major concerns might be due to potential misinterpretations of the paper and we would like to clarify them in the responses.
We updated Appendix C in this place. 

**1. Performance evaluation on Deepmind control suite.**
In Figure 1, we present the performance curves on DMControl tasks. Our proposed algorithm TEEN also demonstrates superior performance on these tasks, which is consistent with our main paper. TEEN outperforms other baselines by a large margin on the fish-swim task. 

**2. Ablation study.**
To understand the impact of hyper-parameters, we conducted experiments in various environments to show how the ensemble size N, the number of value functions M, and the weight $\alpha$ take effect. The results of the experiment are also in line with our expectations. The results can be found in Figure 2. 

**3. Performance on a challenging task humanoid.**
To convince our evaluation, we conduct our algorithm in a challenging environment Humanoid-v3 in the MuJoCo suite, which is shown in Figure 3. The state dimension of Humaniod is 376 [^1], which is exceptionally difficult to solve. Our algorithm TEEN shows good performance in this task. We compared our algorithm  TEEN with sample-efficient algorithms TD3 and SAC. We follow the standard evaluation settings, carrying out experiments over five million (5e6) steps and running all baselines with 5 random seeds. 

[^1] https://www.gymlibrary.dev/environments/mujoco/humanoid/





Meta-review: This work present an approach that encourages exploration through ensemble policies. This results in more uniform state-action exploration, and ultimately to better performance.

All the reviewers agree that the paper present interesting insight into this important problem of RL.

Please address the feedback of the reviewers in your manuscript.
Decision: Accept (poster)