Title: Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Ensemble Exploration

Control Tasks
Abstract: The combination of deep reinforcement learning (DRL) with ensemble methods has been proved to be highly effective in addressing complex sequential decision-making problems. This success can be primarily attributed to the utilization of multiple models, which enhances both the robustness of the policy and the accuracy of value function estimation. However, there has been limited analysis of the empirical success of current ensemble RL methods thus far. Our new analysis reveals that the sample efficiency of previous ensemble DRL algorithms may be limited by sub-policies that are not as diverse as they could be. Motivated by these findings, our study introduces a new ensemble RL algorithm, termed \textbf{T}rajectories-awar\textbf{E} \textbf{E}nsemble exploratio\textbf{N} (TEEN). The primary goal of TEEN is to  maximize the expected return while promoting more diverse trajectories. Through extensive experiments, we demonstrate that TEEN not only enhances the sample diversity of the ensemble policy compared to using sub-policies alone but also improves the performance over ensemble RL algorithms. On average, TEEN outperforms the baseline ensemble DRL algorithms by 41\% in performance on the tested representative environments.

Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control

Abstract

The combination of deep reinforcement learning (DRL) with ensemble methods has been proved to be highly effective in addressing complex sequential decisionmaking problems. This success can be primarily attributed to the utilization of multiple models, which enhances both the robustness of the policy and the accuracy of value function estimation. However, there has been limited analysis of the empirical success of current ensemble RL methods thus far. Our new analysis reveals that the sample efficiency of previous ensemble DRL algorithms may be limited by sub-policies that are not as diverse as they could be. Motivated by these findings, our study introduces a new ensemble RL algorithm, termed T rajectories-awar E E nsemble exploratio N (TEEN). The primary goal of TEEN is to maximize the expected return while promoting more diverse trajectories. Through extensive experiments, we demonstrate that TEEN not only enhances the sample diversity of the ensemble policy compared to using sub-policies alone but also improves the performance over ensemble RL algorithms. On average, TEEN outperforms the baseline ensemble DRL algorithms by 41% in performance on the tested representative environments.

1 Introduction

Deep Reinforcement Learning (DRL) [1] has demonstrated significant potential in addressing a range of complex sequential decision-making problems, such as video games [2; 3; 4], autonomous driving [5; 6], robotic control tasks [7; 8], board games [9; 10], etc. Although the combination of high-capacity function approximators, such as deep neural networks (DNNs), enables DRL to solve more complex tasks, two notorious problems impede the widespread use of these DRL in real-world domains. i) function approximation error: Q-learning algorithm converges to sub-optimal solutions due to the error propagation [11; 12; 13], e.g., the maximization of random function approximation errors accumulate into consistent overestimation bias [14]. ii) sample inefficiency , model-free algorithm is notorious for requiring sample diversity-training of neural networks requires a large number of samples which is hard to acquire in real-world scenarios.

Ensemble reinforcement learning shows great potential in solving the issues mentioned above by combining multiple models of the value function and (or) policy [14; 15; 16; 17; 18; 19; 20] to improve the accuracy and diversity. For instance, Max-Min Q-learning [11] reduces estimation bias by utilizing an ensemble of estimates and selecting the minimum value as the target estimate. Bootstrapped DQN [15] and SUNRISE [16] train an ensemble of value functions and policies,

leveraging the uncertainty estimates of value functions to enrich the diversity of learning experiences. However, we claim that existing ensemble methods do not effectively address the crucial issue of diverse exploration, which is essential for enhancing sample efficiency. Although randomly initializing sub-policies shares some similarities to adding random noise with the ensemble policy, it is not sufficient for significantly improving the sample diversity of the ensemble policy. Thus, how to enhance the exploration diversity of ensemble policy remains an open question.

This paper presents a novel approach called T rajectories-awar E E nsemble exploratio N (TEEN) that encourages diverse exploration of ensemble policy by encouraging diverse behaviors through exploration of the state-action visit distribution measure space. The state-action visit distribution measure quantifies the frequency with which a particular state-action pair is visited when using a certain policy. Thus, policies with diverse state-action visit distributions induce different trajectories [21]. Moreover, value functions are learned from the trajectories generated by the policy, rather than the actions the policy may take. While previous research [22; 23; 24] has emphasized promoting diverse actions, this approach does not always lead to diverse trajectories. This paper highlights the importance of diverse trajectories in improving sample efficiency, which TEEN achieves.

Our contributions can be summarized in three aspects. (1) We propose T rajectories-awar E E nsemble exploratio N (TEEN) algorithm, a highly sample-efficient ensemble reinforcement learning algorithm. TEEN trains sub-policies to exhibit a greater diversity of behaviors, accomplished by considering the distribution of trajectories. (2) Theoretical analysis confirms that our algorithm facilitates more diverse exploration by enhancing the varied behaviors of the sub-policies. (3) Extensive experimental results show that our method outperforms or achieves a similar level of performance as the current state-of-the-art across multiple environments, which include both MuJoCo control [25] tasks and DMControl tasks [26]. Specifically, on average across the tasks we investigated, TEEN surpasses the baseline ensemble DRL algorithm by 41% and the state-of-the-art off-policy DRL algorithms by 7.3% in terms of performance. Furthermore, additional experiments demonstrate that our algorithm samples from a diverse array of trajectories.

2 Related Work

Ensemble Methods. Ensemble methods have been widely utilized in DRL to serve unique purposes. Overestimation bias in value function severely undermines the performance of Q-learning algorithms, and a variety of research endeavors on the ensemble method have been undertaken to alleviate this issue [14; 18; 19; 27; 11; 28]. TD3 [14] clips Double Q-learning to control the overestimation bias, which significantly improves the performance of DDPG [29]. Maxmin Q-learning [11] directly mitigates the overestimation bias by using a minimization over multiple action-value estimates. Ensemble methods also play a role in encouraging sample efficient exploration [30]. For example, Bootstrapped DQN [15] leverages uncertainty estimates for efficient (and deep) exploration. On top of that, REDQ [17] enhances the sample efficiency by using a update-to-data ratio ≫ 1. SUNRISE [16] turns to an ensemble policy to consider for compelling exploration.

Diverse Policy. A range of works have been proposed for diverse policy, and these can generally be classified into three categories [31]. The first category involves directly optimizing policy diversity during pre-training without extrinsic rewards. Fine-tuning with rewards is then applied to complete the downstream tasks. These methods [32; 33; 34; 35] train the policy by maximizing the mutual information between the latent variable and the trajectories, resulting in multiple policies with diverse trajectories. The second category addresses constraint optimization problems, where either diversity is optimized subject to quality constraints, or the reverse is applied - quality is optimized subject to diversity constraints [36; 37; 38; 39]. The third category comprises quality diversity methods [31; 40; 41], which prioritize task-specific diversity instead of task-agnostic diversity. These methods utilize scalar functions related to trajectories and simultaneously optimize both quality and diversity.

Efficient Exploration. Efficient exploration methods share similar methods to diverse policy methods in training the agent, wherein exploration by diverse policies can enhance the diversity of experiences, ultimately promoting more efficient exploration. [42]. Other prior works focusing on efficient exploration consider diverse explorations within a single policy, with entropy-based methods being the representative approach [22; 43]. These methods strive to simultaneously maximize the expected cumulative reward and the entropy of the policy, thus facilitating diverse exploration. Additionally,

another representative category of research is the curiosity-driven method [24; 23; 44]. These methods encourage the agent to curiously explore the environment by quantifying the state with "novelty", such as state visit counts [23], model prediction error [24], etc.

3 Preliminaries

3.1 Markov Decision Process

Markov Decision Process (MDP) can be described as a tuple ⟨S , A , P, r, γ ⟩ , where S and A represent the state and action spaces, respectively. The function P maps a state-action pair to the next state with a transition probability in the range [0 , 1] , i.e., P : S × A × S → [0 , 1] . The reward function r : S × A → ❘ denotes the reward obtained by performing an action a in a particular state s . Additionally, γ ∈ (0 , 1) is the discount factor. The goal of reinforcement learning algorithms is to find a policy π ∗ that maximizes the discounted accumulated reward J ( π ) , defined as:

3.2 Overestimation Bias in Deep Deterministic Policy Gradient

The Deep Deterministic Policy Gradient (DDPG) algorithm [45; 29] forms the foundation of various continuous control tasks such as TD3 [14] and SAC [22]. By enabling policy output to produce deterministic actions, DDPG offers a powerful algorithmic solution to these tasks. DDPG utilizes two neural networks: the actor and critic networks. The actor network is responsible for generating the policy, while the critic network evaluates the value function. In deep reinforcement learning (DRL), the policy π ( a | s ) and the value function Q ( s, a ) are expressed by deep neural networks parameterized with ϕ and θ respectively. DDPG suggests updating the policy through a deterministic policy gradient, as follows:

In order to approximate the discounted cumulative reward (as specified in Eq.(1)), the function Q ( s, a ) is updated by minimizing the temporal difference (TD) errors [46] between the estimated value of the next state s ′ and the current state s .

In policy evaluation, Q function is used to approximate the value by taking action a in state s . The Q function is updated by Bellman equation [47] with bootstrapping, where we minimize the expected Temporal-Difference (TD) [46] error δ ( s, a ) between value and the target estimate in a mini-batch of samples B ,

This process can lead to a consistent overestimation bias as the holding of Jensen's inequality [48]. Specifically, the overestimation of value evaluation arises due to the following process:

4 Methodology

This section describes T rajectories-awar E E nsemble exploratio N (TEEN) method that achieves diverse exploration for agents. TEEN aims to enforce efficient exploration by encouraging diverse behaviors of an ensemble of N sub-policies parameterized by { ϕ i } N i =1 . TEEN initially formulates a discrepancy measure and enforces the discrepancy among the sub-policies while simultaneously maximizing the expected return. Next, we present how to solve this optimization problem indirectly by utilizing the mutual information theory. Finally, we conduct a theoretical analysis to ensure whether the diversity of the samples collected by the ensemble policy is enhanced by solving this optimization problem. We summarize TENN in Algorithm 1.

4.1 Discrepancy Measure

Weassess the diverse behaviors of policies by examining the diversity of trajectories generated by their interactions with the environments. The state-action visit distribution, indicating the probability of encountering a specific state-action pair when initiating the policy from a starting state, encapsulates the diversity of these trajectories directly. Thus, TEEN measures the diverse behaviors of policies by directly assessing their state-action visit distribution. Formally, given an ensemble of N sub-policies { π 0 , π 1 , ..., π k , ..., π N } , we use ρ π k to induce the state-action visit distribution deduced by policy π k and use ρ for the ensemble policy. We have,

For convenience, we use the conditioned state-action visit probability, denoted as ρ ( s, a | z ) , to represent the respective state-action visit distribution of each sub-policy. This can be expressed as follows,

Where { z k } N is the latent variable representing each policy. Our discrepancy measure KL -divergence is an asymmetric measure of the difference between two probability distributions. Given this discrepancy measure, we define the difference of policies as the KL -divergence of the conditioned state-action visit distribution with the state-action visit distribution.

Definition 1 (Difference between Policies) Given an ensemble of N policies { π 0 , π 1 , ..., π k , ..., π N } , the difference between policy π k and other policies can be defined by the conditioned state-action occupation discrepancy:

Consequently, we improve the difference between the conditioned state-action visit distribution to enforce diverse behaviors of sub-polices. While directly optimizing the Eq. (8) can be hard: Obtaining the state-action visit distribution can be quite challenging. In response to this, Liu, H., et al., as referenced in citation [49], have suggested the use of K-Nearest Neighbors (KNN) as an approximate measure for this distribution. We notice that it can be expressed by the form of entropy discrepancy:

In which the H [ · ] is the Shannon Entropy with base e , H ( ρ ) is the entropy of the state-action visit distribution. With the mutual information theory, we can transform this optimization target as follows:

Thus, we turn to this equivalent optimization target. The first term encourages a high entropy of p ( z ) . We fix p ( z ) to be uniform in this approach by randomly selecting one of the sub-policies to explore. We have H ( z ) = -1 N Σ N k =1 log p ( z k ) ≈ log N , which is a constant. The second term emphasises the sub-policy is easy to infer given a state-action pair. We approximate this posterior with a learned discriminator q ζ ( z | s, a ) and optimize the variation lower bound.

Where we use non-negativity of KL-divergence, that is D KL ≥ 0 . We minimize D KL [ ρ ( z | s, a ) || q ζ ( z | s, a )] with respect to parameter ζ to tighten the variation lower bound:

Consequently, we have the equivalent policy gradient with regularizer:

4.2 Trajectories-aware Ensemble Exploration

We select TD3 [14] as the algorithm of sub-policies which is not a powerful exploration algorithm. We then show how to ensemble algorithm with poor exploration performance and encourage efficient ensemble exploration by solving this optimization problem. We consider an ensemble of N TD3 agents i.e., { Q θ k , π ϕ k } N k =1 , where θ k and ϕ k denote the parameters of k -th Q-function and sub-policy. Combined with deterministic policy gradient method [29], we update the each sub-policy by policy gradient with regularizer.

Recurrent Optimization. Updating all sub-policies every time step in parallel may suffer from the exploration degradation problem [35]. Consider a state-action pair ( s, a 1 ) , which is frequently visited by sub-policy π k and another state-action pair ( s, a 2 ) which is visited by multiple sub-policies such that we have q ζ ( z k | s, a 1 ) > q ζ ( z k | s, a 2 ) . Under this circumstance, the constraint encourage sub-policy π k to execute action a 1 while preventing from executing action a 2 which leads to the problem that some explored state-actions are prevented from being visited by the sub-policies. To tackle this challenge, we employ a recurrent training method for our sub-policies. In particular, given an ensemble of N sub-policies, we randomly select a single sub-policy, denoted as π k , every T episodes. We then concentrate our efforts on regulating the behavior of the selected sub-policy, rather than all sub-policies simultaneously..

Gradient Clip. TEEN uses α E ( s,a,z ) ∼ ρ [log q ζ ( z | s, a ))] as a constraint to optimize the policy, while the gradient of this term is extremely large when the probability q ζ ( z | s, a ) is small. And a state-action with small probability to infer the corresponding sub-policy z k implies that the state-action is rarely visited by sub-policy z k but frequently visited by other sub-policies. The target of the constraint is to increase the discrepancy of the sub-policies, while making the sub-policy visit this state-action will instead reduce the discrepancy. Further, for a state-action with large probability q ζ ( z | s, a ) , continuing to increase this probability will prevent the sub-policy from exploring other possible state-actions. Thus, we use the clipped constraint and the main objective we propose is the following:

where ϵ is a hyper-parameter, and we set ϵ = 0 . 1 for all of our experiments.

4.3 Controlling the Estimation Bias

In Q-value estimation, we share some similarities with Max-Min DQN [11] and Averaged DQN [50]. While Max-Min DQN takes the minimum over an ensemble of value functions as the target encouraging underestimation, Averaged DQN reduces the variance of estimates by using the mean of the previously learned Q-values. We combine these two techniques and apply both mean and min operators to reduce overestimation bias and variance. Instead of using the mean of the previously learned Q-values, we use the mean value on N actions of N sub-policies. TEEN then uses the minimum over a random subset M of M Q-functions as the target estimate.

We give some analytical evidences from the perspective of order statistics to show how we control the estimation bias by adjusting M and N .

Theorem 1. Let X 1 , X 2 , ..., X N be an infinite sequence of i.i.d. random variables with a probability density function (PDF) of f ( x ) and a cumulative distribution function (CDF) of F ( x ) . Denote µ = E [ X i ] and σ 2 = V [ X i ] . Let X 1: N ≤ X 2: N ≤ X 3: N ... ≤ X N : N be the order statistics corresponding to { X i } N . Denote PDF and CDF of the k -th order statistic X k : N as f k : N and F k : N respectively. The following statements hold.

Algorithm 1 Trajectories-Aware Ensemble Exploration algorithm (TEEN)

1: Initialize N policy parameters ϕ k , N Q-function parameters θ k , k = 1 , 2 , ..., N

2: Initialize target Q-function parameters θ ′ k ← θ k , k = 1 , 2 , ..., N .

3: Initialize discriminator q ζ

4: Initialize empty replay buffer D

5: Initialize recurrent interval d

6: for each iterations do

7: Randomly sample a policy π k

8: for each time step t do

9: Choose one action with noise a t = π ϕ k ( s ) + ϵ, ϵ ∼ N (0 , σ )

10:

Collect state

s

t

+1

and reward

r

t

from the environment by taking action

11: Store transition τ t = ( s t , a t , s t +1 , r t ) in replay buffer B ← B ∪ τ t

12: end for

13: if t mod d then

14: Randomly select one sub-policy π ϕ i

15: end if

16:

for each gradient step do

17:

18: Randomly select a set M of M distinct indices from { 1 , 2 , ...N } .

19: Compute the Q target y by Eq. (16)

20: For all of the value functions, update θ k using gradient decent

21:

22: For sub-policy π ϕ i , update ϕ i by gradient decent

23:

24: Update target Q-function by θ ′ k ← ρθ k +(1 -ρ ) θ ′ k

25: Update discriminator q ζ by Eq. (12)

26: end for

27: end for

Points 1 of the Theorem indicates that the minimum over an ensemble of values reduces the expected value, which implies that we can control the estimation from over estimates to under or proper estimates. The second point indicates that the mean over an ensemble of values reduces the variance. Thus, by combining these two operators, we control the estimation bias and the variance.

4.4 Theoretical Analysis

We analyze the diversity of state-actions gathered by ensemble policy on the basis of Shannon entropy H [ · ] , a commonly used diversity measure [32; 34; 51], which is a measure of the dispersion of a distribution. A high entropy of the state-action visit distribution implies that the trajectory distribution is more dispersed, which means more diverse trajectories. Thus, to achieve diverse trajectories, we maximize the expected return while maximizing the entropy of the state-action visit distribution:

Where ρ π deduce the state-action visit distribution induced by the ensemble policy π and α is the weighting factor. The diversity of trajectories gathered by the ensemble policy comes from two

a

t

0.0

0.5

1.0

Time Steps (1e6)

0

5000

10000

Average Return

HalfCheetah-v3

0.0

0.5

1.0

Time Steps (1e6)

0

2000

4000

0.0

0.5

1.0

Time Steps (1e6)

0

2000

4000

0.0

0.5

1.0

Time Steps (1e6)

0

2000

4000

6000

TEEN

TD3

SAC

RND

SUNRISE

0.0

0.5

1.0

Time Steps (1e6)

0

500

Average Return

cheetah-run

0.0

0.5

1.0

Time Steps (1e6)

0

500

1000

finger-spin

0.0

0.5

1.0

Time Steps (1e6)

0

200

400

fish-swim

0.0

0.5

1.0

Time Steps (1e6)

0

500

1000

walker-walk

0.0

0.5

1.0

Time Steps (1e6)

0

250

500

750

walker-run

TEEN

TD3

SAC

RND

SUNRISE

components: the discrepancy of sub-policies and the diversity of trajectories gathered by the subpolicies. To illustrate that,

Lemma 1 (Ensemble Sample Diversity Decomposition) Given the state-action visit distribution of the ensemble policy ρ . The entropy of this distribution is H ( ρ ) . Notice that this term can be decomposed into two parts:

Where the first term is the state-action visit distribution discrepancy between the sub-policies and the ensemble policy induced by KL -divergence measure. The second term implies the diversity of state-action visited by sub-policies which depends on which algorithm is used for the sub-policy, such as TD3 [14], SAC [22], RND [24], etc.

As shown in this inequality, H ( ρ ) is irrelevant of ensemble size N . Therefore, the ensemble size may improve the diversity of the ensemble policy by influencing the discrepancy of sub-policies indirectly, which implies that the diversity is not guaranteed with the increased ensemble size. Our method increases the discrepancy of sub-policies which theoretically improve the sample diversity of the ensemble policy.

5 Experimental Setup

This section presents the experimental setup used to evaluate our proposed TEEN and assesses its performance in the MuJoCo environments.

Environments. We evaluate our algorithm on several continuous control tasks from MuJoCo control suite [25] and DeepMind Control Suite [26]. We conduct experiments on 4 control tasks in MuJoCo control suite namely HalfCheetah-v3, Hopper-v3, Walker2d-v3, Ant-v3. In Halfcheetah-v3, the task is to control a cheetah with only one forefoot and one hind foot to run forward. In Hopper-v3, we control a single-legged robot to hop forward and keep balance when hopping. Walker2d-v3 is a bipedal robot environment, where we train the agent to walk or run forward. The target of Ant-v3 is to train a quadrupedal agent to stay balance and move forward. These four environments are all challenging continuous control tasks in MuJoCo. On DeepMind Control Suite, we evaluate our algorithm in 5 environments: cheetah-run, finger-spin, fish-swim, walker-walk, walker-run.

(1) TEEN

(2) TD3

(3) SAC

(4) RND

(5) SUNRISE

0.00

0.25

0.50

0.75

1.00

Time Steps (1e6)

0

1000

2000

3000

4000

5000

6000

Average Return

TEEN(M=2,N=2)

TEEN(M=2,N=5)

TEEN(M=2,N=10)

TD3

(a) Performance

0.00

0.25

0.50

0.75

1.00

Time Steps (1e6)

0.8

0.6

0.4

0.2

Average Normalized Bias

TEEN(M=2,N=2)

TEEN(M=2,N=5)

TEEN(M=2,N=10)

TD3

(b) Average Bias

0.00

0.25

0.50

0.75

1.00

Time Steps (1e6)

0

1000

2000

3000

4000

5000

6000

Average Return

TEEN(M=2,N=10)

TEEN(M=3,N=10)

TEEN(M=4,N=10)

TD3

(c) Performance

0.00

0.25

0.50

0.75

1.00

Time Steps (1e6)

0.8

0.6

0.4

0.2

Average Normalized Bias

TEEN(M=2,N=10)

TEEN(M=3,N=10)

TEEN(M=4,N=10)

TD3

(d) Average Bias

0.00

0.25

0.50

0.75

1.00

Time Steps (1e6)

0

1000

2000

3000

4000

5000

6000

Average Return

TEEN,

=0.0

TEEN,

=0.2

TEEN,

=0.5

TEEN,

=1.0

TD3

(e) Performance

0.00

0.25

0.50

0.75

1.00

Time Steps (1e6)

0.4

0.6

0.8

1.0

Average

H

(

)

TEEN,

=0.0

TEEN,

=0.2

TEEN,

=0.5

TEEN,

=1.0

TD3

(f) Average

H

[

ρ

]

Baselines. Beyond simply comparing our approach with the foundational algorithm TD3 [14], a leading-edge deterministic policy gradient reinforcement learning algorithm, we also contrast our method with other efficient exploration algorithms across three distinct categories. We examine curiosity-based exploration baselines, such as RND [24], which promotes efficient exploration through intrinsic curiosity rewards. Regarding maximum entropy-based exploration, we consider SAC [22], an off-policy deep reinforcement learning algorithm that offers sample-efficient exploration by maximizing entropy during training. For ensemble exploration, we look at SUNRISE [16], a unified framework designed for ensemble-based deep reinforcement learning.

Evaluation. We follow the standard evaluation settings, carrying out experiments over one million (1e6) steps and running all baselines with ten random seeds per task to produce the main results. We maintain consistent learning rates and update ratios across all baselines. Further details regarding these parameters can be found in the Appendx B.2. For replication of TD3, SAC and SUNRISE, we use the code the author released for each baseline without any modifications to ensure the performance. For the reproduction of RND, we follow to the repository provided in the original code. For a fair

comparison, we use fully connected network with 3 layers as the architecture of the critic networks and actor networks. The learning rate for each network is set to be 3 e -4 with Adam [52] as the optimizer. The implementation details for each baseline can be found in Appendx B.2.

6 Experimental Results

We conducted experiments to evaluate our algorithm's performance in addressing three key research questions (RQ). Specifically, we sought to determine whether TEEN outperforms the current stateof-the-art across multiple environments and achieves a boarder range of exploration region, as well as how the ensemble size and discrepancy of sub-policies (defined in Eq. 9) impact TEEN's performance.

RQ1: Whether TEEN outperforms the current state-of-the-art across multiple environments? Table 1 and Table 2 report the average returns with mean and variance of evaluation roll-outs across all algorithms. The learning curves during training on MuJoCo control suite are shown in Figure 1. Learning curves for DeepMind Control suite can be found in Figure 2. On MuJoCo continuous control suite, TEEN shows superior performance across all environments which implies that by ensemble sub-policies with weak exploration performance, such as TD3, the exploration performance of the ensemble policy enhances evenly exceeding algorithms with strong exploration performance, SAC, RND, etc. While purely ensembling multiple models may not certainly improve the performance. Although SUNRISE improves the exploration performance by UCB exploration [53], it does not take effect when the uncertainty shows little relevance to the environment, and even degrades the performance (as shown in HalfCheetah-v3). TEEN can be seen as an immediate solution for diverse exploration within ensemble reinforcement learning by directly enforcing an ensemble policy to discover a broader range of trajectories. We further conduct experiments to validate that TEEN achieves a boarder exploration region (Figure 3). For a fair comparison, all the algorithms are trained in Ant-v3 with the same seed at half of the training process. In order to get reliable results, the states explored are gathered in 10 episodes with different seeds. We apply the same t-SNE transformation to the states explored by all of the algorithms for better visualization.

RQ2: How the ensemble size N and number of target values M impact TEEN's performance? We make experiments in Ant-v3 environment from OpenAI gym. As shown in Figure 3, when using M = 2 as the number of target values, the estimate of the target value is underestimated and the estimation bias increases with the increase of M . This underestimation is mitigated with an increase in ensemble size N resulting in better performance. Thus, we choose 10 to be the ensemble size for all the environments. To enhance reliability, we conducted ablation studies in various MuJoCo

environments, obtaining consistent results. The results for other environments can be found in the Appendix C.

RQ3: How the discrepancy of sub-policies impact TEEN's performance? In order to validate the benefits of trajectory-aware exploration, we conduct experiments in the Ant-v3 environment, progressively increasing the weight parameter denoted as α . We adjust the value of the weight α from [0 , 0 . 2 , 0 . 5 , 1] . With α = 0 , we show the performance without trajectory-aware exploration. As the results presented in Figure 3, with adequate reward scale, TEEN improves the performance with the use of trajectory-aware exploration. The effectiveness of the approach fluctuates with changes in reward scale. For a large α value, TEEN aims to enhance the diversity of trajectories. However, as α increases, TEEN fails to effectively utilize reward signals, resulting in sub-optimal performance.In the experiment, we used α = 0 . 2 for the MuJoCo control suites. However, the reward scales in the DeepMind control suites are significantly smaller than those in the MuJoCo control suites. In the MuJoCo control suite, the accumulated reward exceeds 5,000 in the Ant environment, while in the DeepMind control suite, the maximum accumulated reward for all tasks is 1,000. As a result, we utilized a smaller α = 0 . 02 for the DeepMind control suite, considering the smaller reward scale. We also conducted ablation studies with varying reward scales across different environments, please refer to Appendix C for more details.

7 Conclusion

This paper presents a theoretical analysis of the ensemble policy's diversity exploration problem, based on trajectory distribution. We innovatively discover that the diversity of the ensemble policy can only be enhanced by increasing the Kullback-Leibler (KL) divergence of the sub-policies when entropy is used as the measure of diversity. Guided by this analysis, we introduce the Trajectoriesaware Ensemble Exploration (TEEN) method - a novel approach that encourages diverse exploration by improving the diversity of trajectories. Our experiments demonstrate that TEEN consistently elevates the performance of existing off-policy RL algorithms such as TD3, SAC, RND, and ensemble RL algorithms like SUNRISE. There are still important issues to be resolved, e.g., TEEN is sensitive to the reward scale resulting in poor performance without a proper parameter setting. We hope these will be addressed by the community in the future.

Acknowledgments

This work was funded by the National Key R & D Program of China (2021YFC2800500).

References

Sutton, R. S., A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

Mnih, V., K. Kavukcuoglu, D. Silver, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Mnih, V., A. P. Badia, M. Mirza, et al. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , vol. 48 of JMLR Workshop and Conference Proceedings , pages 1928-1937. PMLR, 2016.

Gong, C., Y. Bai, X. Hou, et al. Stable training of bellman error in reinforcement learning. In Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18-22, 2020, Proceedings, Part V 27 , pages 439-448. Springer, 2020.

Ye, Y., X. Zhang, J. Sun. Automated vehicle's behavior decision making using deep reinforcement learning and high-fidelity simulation environment. Transportation Research Part C: Emerging Technologies , 107:155-170, 2019.

Gong, C., Z. Yang, Y. Bai, et al. Mind your data! hiding backdoors in offline reinforcement learning datasets. arXiv preprint arXiv:2210.04688 , 2022.

Andrychowicz, M., F. Wolski, A. Ray, et al. Hindsight experience replay. In Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017.

Lillicrap, T. P., J. J. Hunt, A. Pritzel, et al. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings . 2016.

Silver, D., T. Hubert, J. Schrittwieser, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

Silver, D., J. Schrittwieser, K. Simonyan, et al. Mastering the game of go without human knowledge. Nature , 550(7676):354-359, 2017.

Lan, Q., Y. Pan, A. Fyshe, et al. Maxmin q-learning: Controlling the estimation bias of qlearning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

He, Q., X. Hou. WD3: taming the estimation bias in deep reinforcement learning. In 32nd IEEE International Conference on Tools with Artificial Intelligence, ICTAI 2020, Baltimore, MD, USA, November 9-11, 2020 , pages 391-398. IEEE, 2020.

He, Q., T. Zhou, M. Fang, et al. Eigensubspace of temporal-difference dynamics and how it improves value approximation in reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , vol. 14172 of Lecture Notes in Computer Science , pages 573-589. Springer, 2023.

Fujimoto, S., H. Hoof, D. Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018 , vol. 80 of Proceedings of Machine Learning Research , pages 1582-1591. PMLR, PMLR, 2018.

Osband, I., C. Blundell, A. Pritzel, et al. Deep exploration via bootstrapped dqn. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, page 4033-4041. Curran Associates Inc., 2016.

Lee, K., M. Laskin, A. Srinivas, et al. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning , pages 6131-6141. PMLR, 2021.

Chen, X., C. Wang, Z. Zhou, et al. Randomized ensembled double q-learning: Learning fast without a model. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Hasselt, H. v., A. Guez, D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , AAAI'16, page 2094-2100. AAAI Press, 2016.

Hasselt, H. Double q-learning. In Advances in Neural Information Processing Systems , vol. 23. Curran Associates, Inc., 2010.

He, Q., H. Su, J. Zhang, et al. Frustratingly easy regularization on representation can boost deep reinforcement learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 20215-20225. IEEE, 2023.

Zahavy, T., Y. Schroecker, F. Behbahani, et al. Discovering policies with domino: Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205.13521 , 2022.

Haarnoja, T., A. Zhou, P. Abbeel, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018 , vol. 80 of Proceedings of Machine Learning Research , pages 1856-1865. PMLR, PMLR, 2018.

Pathak, D., P. Agrawal, A. A. Efros, et al. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning , vol. 70 of Proceedings of Machine Learning Research , pages 2778-2787. PMLR, 2017.

Burda, Y., H. Edwards, A. J. Storkey, et al. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Todorov, E., T. Erez, Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. 2012.

Tassa, Y., Y . Doron, A. Muldal, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Wu, D., X. Dong, J. Shen, et al. Reducing estimation bias via triplet-average deep deterministic policy gradient. IEEE Transactions on Neural Networks and Learning Systems , 31(11):49334945, 2020.

Wei, W., Y. Zhang, J. Liang, et al. Controlling underestimation bias in reinforcement learning via quasi-median operation. Proceedings of the AAAI Conference on Artificial Intelligence , 36(8):8621-8628, 2022.

Silver, D., G. Lever, N. Heess, et al. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32 , ICML'14, page I-387-I-395. JMLR.org, 2014.

Faußer, S., F. Schwenker. Neural network ensembles in reinforcement learning. Neural Processing Letters , 41:55-69, 2015.

Wu, S., J. Yao, H. Fu, et al. Quality-similar diversity via population based reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Eysenbach, B., A. Gupta, J. Ibarz, et al. Diversity is all you need: Learning skills without a reward function. In 7th International Conference on Learning Representations, ICLR 2019 . OpenReview.net, 2019.

Laskin, M., H. Liu, X. B. Peng, et al. Unsupervised reinforcement learning with contrastive intrinsic control. In Advances in Neural Information Processing Systems , vol. 35, pages 34478-34491. Curran Associates, Inc., 2022.

Sharma, A., S. Gu, S. Levine, et al. Dynamics-aware unsupervised discovery of skills. In 8th International Conference on Learning Representations, ICLR 2020 . OpenReview.net, 2020.

Jiang, Z., J. Gao, J. Chen. Unsupervised skill discovery via recurrent skill training. In Advances in Neural Information Processing Systems , vol. 35, pages 39034-39046. Curran Associates, Inc., 2022.

Kumar, S., A. Kumar, S. Levine, et al. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems , 33:8198-8210, 2020.

Zhou, Z., W. Fu, B. Zhang, et al. Continuously discovering novel strategies via reward-switching policy optimization. arXiv preprint arXiv:2204.02246 , 2022.

Sun, H., Z. Peng, B. Dai, et al. Novel policy seeking with constrained optimization. arXiv preprint arXiv:2005.10696 , 2020.

Masood, M., F. Doshi-Velez. Diversity-inducing policy gradient: Using maximum mean discrepancy to find a set of diverse policies. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pages 5923-5929. International Joint Conferences on Artificial Intelligence Organization, 2019.

Fontaine, M., S. Nikolaidis. Differentiable quality diversity. In Advances in Neural Information Processing Systems , vol. 34, pages 10040-10052. Curran Associates, Inc., 2021.

Parker-Holder, J., A. Pacchiano, K. M. Choromanski, et al. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems , 33:1805018062, 2020.

Li, C., C. Gong, Q. He, et al. Centralized cooperative exploration policy for continuous control tasks. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages 2454-2456. 2023.

Haarnoja, T., H. Tang, P. Abbeel, et al. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning , vol. 70 of ICML'17 , page 1352-1361. JMLR.org, 2017.

Gong, C., Z. Yang, Y. Bai, et al. Curiosity-driven and victim-aware adversarial policies. In Proceedings of the 38th Annual Computer Security Applications Conference , ACSAC '22, page 186-200. Association for Computing Machinery, 2022.

Silver, D., G. Lever, N. Heess, et al. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32 , ICML'14, page I-387-I-395. JMLR.org, 2014.

Sutton, R. S. Learning to predict by the methods of temporal differences. Machine Learning , 3(1):9-44, 1988.

Bellman, R. Dynamic programming. Science , 153(3731):34-37, 1966.

Kuznetsov, A., P. Shvechikov, A. Grishin, et al. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In the International Conference on Machine Learning (ICML) , pages 5556-5566. PMLR, 2020.

Liu, H., P. Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning , pages 6736-6747. PMLR, 2021.

Anschel, O., N. Baram, N. Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning , pages 176-185. PMLR, 2017.

Zhao, Y., Q. Ye, W. Wu, et al. Generative prompt model for weakly supervised object localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6351-6361. 2023.

Kingma, D. P., J. Ba. Adam: A method for stochastic optimization. In ICLR (Poster) . 2015.

Wang, Y., K. Dong, X. Chen, et al. Q-learning with ucb exploration is sample efficient for infinite-horizon mdp. In International Conference on Learning Representations .

David, H. A., H. N. Nagaraja. Order statistics . John Wiley & Sons, 2004.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of exploration in ensemble-based reinforcement learning. It introduces a novel approach called Trajectory Aware Ensemble Exploration (TEEN) that aims to enhance exploration by increasing diversity among the sub-policies. This diversity is measured by the KL divergence between the distributions of the trajectories. The method achieves diversity by optimizing the variational lower bound, which is then incorporated into the policy gradient step with the addition of a regularizer term.\n\nTo reduce estimation bias on the Q function, the method selects the minimum value from a random set of critics based on the average Q value of the actions chosen by all sub-policies. The paper provides theoretical analysis to support this design choice, demonstrating that it reduces both estimation bias and variance.\n\nThe effectiveness of TEEN is validated through experimental results on MuJoco and Deepmind Control Suite. These results show that the proposed method successfully explores a wide range of states and surpasses the performance of existing DRL algorithms in continuous control, including those specifically designed for efficient exploration.\n\nSTRENGTHS:\n- It proposes a new method that improves the exploration in ensemble-based RL by increasing the diversity of the sub-polices in terms of their state-action visit distribution.\n- The implementation is straightforward and involves two steps on top of a DDPG-like method:\n  - add a regularizer to the policy gradient\n  - add a routine to select and update sub-policies and to compute the target Q value\n- It provides theoretical analysis to justify the choice of target Q value\n- Experimental results are presented to validate the efficacy of the method.\n\nWEAKNESSES:\n- There appears to be an inconsistency between Line 18 in Algorithm 1 and the text. Specifically, Line 18 suggests that all sub-policies are updated sequentially, while the \"recurrent optimization\" section indicates that only one sub-policy is updated in each gradient step. Could you clarify this discrepancy?\n\n- Theorem 1 demonstrates that the method reduces the expected Q value. However, the experimental results indicate that the algorithm tends to underestimate the Q value. It's unclear whether underestimation is preferable to overestimation. If possible, it would be beneficial to address this aspect in the analysis.\n\n- The main paper frequently refers to Appendix C, but it appears to be missing from the appendix section. \n\n- In Line 436 of the appendix, it seems that the last expression is missing a multiplier p(s). Could you confirm?\n\n- There's a mismatch between Lemmas 2 and 3 in the appendix and the content presented in the main paper.\n\n- The proof for (v) in Lemma 3 does not show the variance inequality: V[X_{1:{N+1}}] \\geq V[X_{1:{N}}].\n\n- Formatting Issue:\n\n  - The readability could be improved if Table 2 used the same row/column heading as those in Table 1.\n  - Appendix line 483: Table index is missing.\n\n- Grammar:\n  - Line 151: we can “increase” this equivalent optimization target. “maximize” seems more appreciate here. \n  - Line 272: “.While purely ensemble multiple models may not certainly improve the performance.“ \n  - Line 273: “it does not take effect“, “evenly degrading”.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\n- The authors have discussed one limitation regarding the sensitivity about reward scaling in Sec. 7.\n- Could the authors comment on the computational overhead of the ensemble?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new ensemble RL algorithm. The main motivation of this work is to increase the variation between sub-policies, which is measured by the distance between the corresponding state-action visitation distribution. It formulates the loss function by connecting the visitation distribution distance to the mutual information theory and introduces a discriminator function to measure the policy variation. It also proposes a few tricks to improve the exploration effectiveness of the proposed formulation. The method is experimented on a few Mujoco and Deepmind control suite tasks and compared against several off-policy RL methods and one ensemble RL method.\n\nSTRENGTHS:\n1. The proposed method is motivated by an intuitive idea that increasing the trajectory variation of the sub-policies may improve the exploration and performance of the ensemble policy. \n\n2. The results show the proposed method consistently outperforms baselines across a wide span of tasks in different benchmarks.\n\n3. The experimental analysis is helpful to understand the secret sauce behind the proposed method and the effect of different hyper-parameters.\n\nWEAKNESSES:\n1. The exposition of the paper need improvement (more in the next section)\n\n2. The evaluated control problems are relatively simple and the approach needs to be evaluated on more complex tasks.\n\nQUESTIONS:\n1. Is the statement for the overestimation problem in DDPG properly demonstrated? Eq(4) (line 113) seems to be the formulation for discrete action space while this paper and DDPG focus on continuous action space. Please refer to [[9]](https://arxiv.org/pdf/1802.09477.pdf) for details.\n\n2. It is unclear what the latent variable $z_k$ is at the beginning when it is first presented. Could be more clear if it just says $z_k$ is a categorical index or one-hot encoding of the sub-policy selection.\n\n3. Line 146 mentions using KNN to measure Eq (8) but does the proposed approach indeed use it? If so, where is it applied?\n\n4. Eq (9) is an important step to connect state-action visitation distribution distance to mutual information. Any references or derivation for Eq (9)?\n\n5. The Recurrent Optimization trick (Line 169) is adopted to prevent sub-policy from exploration degradation. If I understand correctly, it corresponds to Line 18-19 in Algorithm 1 where only the sampled policy $\\pi_k$ is updated. However, the gradient of the loss function (Eq (14)) should be independent among sub-policies. Could the authors explain how will the sub-policy $\\pi_{\\phi_k}$ change if we update all sub-policies instead of just $\\pi_{\\phi_k}$ in Algorithm 1 (Line 18-19).\n\n6. **Algorithm 1** helps understand the method while there is no reference to Algorithm 1 throughout the paper.\n\n7. How is the discriminator function $q$ trained? I could not find any details about it.\n\n8. How does the presented algorithm perform on more complex control tasks? The current evaluated tasks are relatively simple that do not require a sophisticated amount of exploration of the policy. It would be more convincing if the approach can be benchmarked on Humanoid and Dextrous hand manipulation tasks.\n\n9. The appendix is incomplete.\n\nLIMITATIONS:\nThe limitations have been discussed above (i.e. it is unknown about the performance of the presented approach on complex tasks).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an algorithm called trajectories-aware ensemble exploration (TEEN) for sample efficient ensemble reinforcement learning. The authors point out that the existing ensemble methods do not effectively address the required diversity in exploration, which TEEN is designed to tackle better. It achieves this by encouraging diverse behaviors through exploration of the state-action visit distribution measure space. Theoretical results are also presented to show how the design principles of TEEN could encourage diversity in exploration, while the experiment results demonstrate its empirical success in popular benchmark tasks.\n\nSTRENGTHS:\nThis manuscript presents an interesting thesis that sample efficient ensemble reinforcement learning methods require diverse exploration in sub-policies. The authors discuss this topic in detail and systematically approach the proposed solution. I view the finding of the diversity requirement of sub-policies coupled with the provided theoretical analysis and the limited yet promising experimental results as the main strength of the work. I, therefore, believe if properly executed, this can be a valuable contribution to the community.\n\nWEAKNESSES:\nOne of the main limitations of the work is its limited evaluations. The authors evaluate the proposed TEEN method to answer three main research questions. But I find their evaluations lacking the experimental rigor that is needed to fully appreciate the proposed method. \n\nFirst, in RQ1 in evaluating the performance against baselines, the proposed method shows visible benefits in Mujoco tasks, which is promising. But my concerns are that all four tasks evaluated may contain similar exploration challenges. Given exploration is one of the key focuses of the work, I would like to see the performance benefits across different tasks that pose different exploration challenges. Perhaps this is what the authors are attempting to do with DeepMind control experiments, but despite the mention, I did not find such experiments in the paper or in the appendix (Appendix C is empty).\n\nRegarding RQ2 and RQ3, my concerns are with the fact that it is only based on a single task (Ant-v3). For both questions, it is not significant enough to make claims based on a single task result. Furthermore, for RQ3, while I never found the results on the DeepMind control suite, the authors mention the best $\\alpha$ is 0.02. This is a very low $\\alpha$ and almost as having no trajectory-aware exploration. This further questions my concerns about how generalizable the proposed method is across different tasks. Clarification is appreciated. \n\nOverall for experiments, authors use only TD3 as the learning algorithm. I would like to see if the observed performance gains are dependent on the choice of the learning algorithm. It is also fine if there is an observable reliance, but in that case, I would like to see the contributions rephrased accordingly. \n\nI presume training the proposed model is challenging, specifically when it comes to finding the best ensemble size N and number of target values M (due to the trial and error approach in finding them, as in RQ2). It also appears that this training would require considerable computations. I would like to see a discussion of these limitations (and other limitations) of the method. \n\nIf the stated limitations are addressed, particularly related to the evaluation process, the proposed work can be of benefit to the community. Consequently, I would be inclined to reconsider my score if the authors can effectively rectify these issues.\n\nQUESTIONS:\n1. Shouldn’t the equation 11 be $\\log(N) - \\mathbb{E}_{s,a,z}[-\\log(\\rho(z|s,a))]$? \n2. Also, shouldn’t the LHS and RHS of equation 11 be approximately equal and not equal?\n3. Define $\\mu$ and $\\sigma$ in theorem 1. \n4. While I did not check the proofs carefully, it appears the authors make simplification assumptions in the proof of theorem 1 that is not stated in the theorem (ex: zero-mean distribution). Please state them properly in the theorem.\n\nLIMITATIONS:\nPlease discuss the limitations of the work, as pointed out earlier. This will help readers better appreciate the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an efficient ensemble learning strategy termed Trajectories-awarE Ensemble exploratioN (TEEN), which promotes exploration by diversifying the state-action visitation distribution of multiple sub-policies. The proposed method introduces a discrepancy measure that tries to maximize the difference in entropy of estimating the state-action visitation with and without the information of the policy (represented by the latent variable) and vice versa. Such a discrepancy is approximated by learning a discriminator that tries to differentiate between state-action pairs visited by different policies. And the log-likelihood of the discriminator is used as a regularizer for policy improvement along with maximizing the expected return of the policy. The authors employ additional strategies of updating a single policy at a time and clipping the discriminator log probabilities to ensure proper exploration. Finally, the estimation bias is controlled by calculating the target value as the minimum of the mean of the ensemble Q-values for each sub-policy. The choices are theoretically motivated and the algorithm is analyzed in simulation on Mujoco Control and DM Control Suite benchmarks.\n\nSTRENGTHS:\n1. TEEN provides new insights into using the measure of the state-action visitation distribution to promote exploration along with maximizing returns in policy optimization. \n\n2. Each design choice made is properly justified: namely updating one sub-policy at a time and clipping the regularizer gradients.\n\nWEAKNESSES:\n1. There might be states not captured by the state-action visitation of one particular sub-policy (say $\\pi_a$). In that case, the contribution of $\\rho^{\\pi_a}(s,a)$ to $\\rho$(s,a) in eq (6) can be highly noisy and misinformative (garbage probability). Giving equal weightage to state-action visitation of every policy while calculating for the ensemble might be misleading.\n\n2. It seems that the latent variable plays a key role in defining the algorithm and the discriminator. However, there is no discussion about the choice of $z_k$ and training of the discriminator in the paper or in Alg 1. How do you choose $z_k$ and how does the training take place of the discriminator? If done parallelly with the policies, the framework follows a min-max update rule, shown to be unstable in prior works like GAIL.\n\nQUESTIONS:\n1. How do you decide the actions while inference? \n2. Can you clarify more about the weakness pt.2?\n3. Section C is missing in the Supplementary.\n4. I am unsure, but it seems that TEEN uses very high random exploration steps as mentioned in the supplementary. Do you use same for all the other algorithms?\n5. If the regularization is already clipped, why is there a coefficient ($\\alpha$) required? Can a comparison be shown between these two cases? Also, $\\alpha=0$ means that there is only a change in Q-value computations as compared to TD3, hence there must be a result for all the environments with $\\alpha=0$ and $\\alpha=0.2$ to show a more convincing contribution of the regularization term. Currently, it is only shown for one env (Ant) and it is particularly not clear how much the regularization term is contributing to the exploration in addition to the ensemble of sub-policies.\n6. The timesteps shown in the results are timesteps for individual sub-policy or the whole ensemble i.e. (1M/10 = 100k steps for single sub-policy)?\n\nLIMITATIONS:\nThe authors have mentioned the limitation as the tuning of $\\alpha$.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}