SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chao Li $^{1,4}$  Chen Gong $^{2}$  Qiang He $^{3}$  Xinwen Hou $^{1\dagger}$
$^{1}$ Institute of Automation, Chinese Academy of Sciences, China
$^{2}$ University of Virginia, USA
$^{3}$ Ruhr University Bochum, Germany
<sup>4</sup>University of Chinese Academy of Sciences, China
{lichao2021，xinwen.hou}@ia.ac.cn
fzv6en@virginia.edu, qianghe97@gmail.com
The combination of deep reinforcement learning (DRL) with ensemble methods has been proved to be highly effective in addressing complex sequential decision-making problems. This success can be primarily attributed to the utilization of multiple models, which enhances both the robustness of the policy and the accuracy of value function estimation. However, there has been limited analysis of the empirical success of current ensemble RL methods thus far. Our new analysis reveals that the sample efficiency of previous ensemble DRL algorithms may be limited by sub-policies that are not as diverse as they could be. Motivated by these findings, our study introduces a new ensemble RL algorithm, termed Trajectories-awarE Ensemble explorationN (TEEN). The primary goal of TEEN is to maximize the expected return while promoting more diverse trajectories. Through extensive experiments, we demonstrate that TEEN not only enhances the sample diversity of the ensemble policy compared to using sub-policies alone but also improves the performance over ensemble RL algorithms. On average, TEEN outperforms the baseline ensemble DRL algorithms by  $41\%$  in performance on the tested representative environments.
Deep Reinforcement Learning (DRL) [1] has demonstrated significant potential in addressing a range of complex sequential decision-making problems, such as video games [2; 3; 4], autonomous driving [5; 6], robotic control tasks [7; 8], board games [9; 10], etc. Although the combination of high-capacity function approximators, such as deep neural networks (DNNs), enables DRL to solve more complex tasks, two notorious problems impede the widespread use of these DRL in real-world domains. i) function approximation error: Q-learning algorithm converges to sub-optimal solutions due to the error propagation [11; 12; 13], e.g., the maximization of random function approximation errors accumulate into consistent overestimation bias [14]. ii) sample inefficiency, model-free algorithm is notorious for requiring sample diversity—training of neural networks requires a large number of samples which is hard to acquire in real-world scenarios.
Ensemble reinforcement learning shows great potential in solving the issues mentioned above by combining multiple models of the value function and (or) policy [14; 15; 16; 17; 18; 19; 20] to improve the accuracy and diversity. For instance, Max-Min Q-learning [11] reduces estimation bias by utilizing an ensemble of estimates and selecting the minimum value as the target estimate. Bootstrapped DQN [15] and SUNRISE [16] train an ensemble of value functions and policies,
leveraging the uncertainty estimates of value functions to enrich the diversity of learning experiences. However, we claim that existing ensemble methods do not effectively address the crucial issue of diverse exploration, which is essential for enhancing sample efficiency. Although randomly initializing sub-policies shares some similarities to adding random noise with the ensemble policy, it is not sufficient for significantly improving the sample diversity of the ensemble policy. Thus, how to enhance the exploration diversity of ensemble policy remains an open question.
This paper presents a novel approach called Trajectories-aware E Ensemble exploration (TEEN) that encourages diverse exploration of ensemble policy by encouraging diverse behaviors through exploration of the state-action visit distribution measure space. The state-action visit distribution measure quantifies the frequency with which a particular state-action pair is visited when using a certain policy. Thus, policies with diverse state-action visit distributions induce different trajectories [21]. Moreover, value functions are learned from the trajectories generated by the policy, rather than the actions the policy may take. While previous research [22; 23; 24] has emphasized promoting diverse actions, this approach does not always lead to diverse trajectories. This paper highlights the importance of diverse trajectories in improving sample efficiency, which TEEN achieves.
Our contributions can be summarized in three aspects. (1) We propose Trajectories-awarE Ensemble explorationN (TEEN) algorithm, a highly sample-efficient ensemble reinforcement learning algorithm. TEEN trains sub-policies to exhibit a greater diversity of behaviors, accomplished by considering the distribution of trajectories. (2) Theoretical analysis confirms that our algorithm facilitates more diverse exploration by enhancing the varied behaviors of the sub-policies. (3) Extensive experimental results show that our method outperforms or achieves a similar level of performance as the current state-of-the-art across multiple environments, which include both MuJoCo control [25] tasks and DMControl tasks [26]. Specifically, on average across the tasks we investigated, TEEN surpasses the baseline ensemble DRL algorithm by  $41\%$  and the state-of-the-art off-policy DRL algorithms by  $7.3\%$  in terms of performance. Furthermore, additional experiments demonstrate that our algorithm samples from a diverse array of trajectories.
Ensemble Methods. Ensemble methods have been widely utilized in DRL to serve unique purposes. Overestimation bias in value function severely undermines the performance of Q-learning algorithms, and a variety of research endeavors on the ensemble method have been undertaken to alleviate this issue [14; 18; 19; 27; 11; 28]. TD3 [14] clips Double Q-learning to control the overestimation bias, which significantly improves the performance of DDPG [29]. Maxmin Q-learning [11] directly mitigates the overestimation bias by using a minimization over multiple action-value estimates. Ensemble methods also play a role in encouraging sample efficient exploration [30]. For example, Bootstrapped DQN [15] leverages uncertainty estimates for efficient (and deep) exploration. On top of that, REDQ [17] enhances the sample efficiency by using a update-to-data ratio  $\gg 1$ . SUNRISE [16] turns to an ensemble policy to consider for compelling exploration.
Diverse Policy. A range of works have been proposed for diverse policy, and these can generally be classified into three categories [31]. The first category involves directly optimizing policy diversity during pre-training without extrinsic rewards. Fine-tuning with rewards is then applied to complete the downstream tasks. These methods [32; 33; 34; 35] train the policy by maximizing the mutual information between the latent variable and the trajectories, resulting in multiple policies with diverse trajectories. The second category addresses constraint optimization problems, where either diversity is optimized subject to quality constraints, or the reverse is applied - quality is optimized subject to diversity constraints [36; 37; 38; 39]. The third category comprises quality diversity methods [31; 40; 41], which prioritize task-specific diversity instead of task-agnostic diversity. These methods utilize scalar functions related to trajectories and simultaneously optimize both quality and diversity.
Efficient Exploration. Efficient exploration methods share similar methods to diverse policy methods in training the agent, wherein exploration by diverse policies can enhance the diversity of experiences, ultimately promoting more efficient exploration. [42]. Other prior works focusing on efficient exploration consider diverse explorations within a single policy, with entropy-based methods being the representative approach [22; 43]. These methods strive to simultaneously maximize the expected cumulative reward and the entropy of the policy, thus facilitating diverse exploration. Additionally,
another representative category of research is the curiosity-driven method [24; 23; 44]. These methods encourage the agent to curiously explore the environment by quantifying the state with "novelty", such as state visit counts [23], model prediction error [24], etc.
Markov Decision Process (MDP) can be described as a tuple  $\langle S, \mathcal{A}, P, r, \gamma \rangle$ , where  $S$  and  $\mathcal{A}$  represent the state and action spaces, respectively. The function  $P$  maps a state-action pair to the next state with a transition probability in the range  $[0,1]$ , i.e.,  $P: S \times \mathcal{A} \times S \to [0,1]$ . The reward function  $r: S \times \mathcal{A} \to \mathbb{R}$  denotes the reward obtained by performing an action  $a$  in a particular state  $s$ . Additionally,  $\gamma \in (0,1)$  is the discount factor. The goal of reinforcement learning algorithms is to find a policy  $\pi^{*}$  that maximizes the discounted accumulated reward  $J(\pi)$ , defined as:
$$
J (\pi) = \mathbb {E} _ {s, a \sim \pi} [ \Sigma_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, a _ {t}) ] \tag {1}
$$
3.2 Overestimation Bias in Deep Deterministic Policy Gradient
The Deep Deterministic Policy Gradient (DDPG) algorithm [45; 29] forms the foundation of various continuous control tasks such as TD3 [14] and SAC [22]. By enabling policy output to produce deterministic actions, DDPG offers a powerful algorithmic solution to these tasks. DDPG utilizes two neural networks: the actor and critic networks. The actor network is responsible for generating the policy, while the critic network evaluates the value function. In deep reinforcement learning (DRL), the policy  $\pi(a|s)$  and the value function  $Q(s,a)$  are expressed by deep neural networks parameterized with  $\phi$  and  $\theta$  respectively. DDPG suggests updating the policy through a deterministic policy gradient, as follows:
$$
\nabla_ {\phi} J (\phi) = \mathbb {E} _ {s \sim P _ {\pi}} \left[ \left. \nabla_ {a} Q ^ {\pi} (s, a) \right| _ {a = \pi (s)} \nabla_ {\phi} \pi_ {\phi} (s) \right] \tag {2}
$$
In order to approximate the discounted cumulative reward (as specified in Eq.(1)), the function  $Q(s, a)$  is updated by minimizing the temporal difference (TD) errors [46] between the estimated value of the next state  $s'$  and the current state  $s$ .
$$
\theta^ {*} = \arg \min  _ {\theta} \mathbb {E} \left[ r (s, a) + \gamma Q _ {\theta} ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}\right) - Q _ {\theta} ^ {\pi} (s, a) \right] ^ {2} \tag {3}
$$
In policy evaluation,  $Q$  function is used to approximate the value by taking action  $a$  in state  $s$ . The  $Q$  function is updated by Bellman equation [47] with bootstrapping, where we minimize the expected Temporal-Difference (TD) [46] error  $\delta(s, a)$  between value and the target estimate in a mini-batch of samples  $\mathcal{B}$ ,
$$
\delta (s, a) = \mathbb {E} _ {\mathcal {B}} \left[ r (s, a) + \max  _ {a ^ {\prime}} Q \left(s ^ {\prime}, a ^ {\prime}\right) - Q (s, a) \right] \tag {4}
$$
This process can lead to a consistent overestimation bias as the holding of Jensen's inequality [48]. Specifically, the overestimation of value evaluation arises due to the following process:
$$
\mathbb {E} _ {\mathcal {B}} \left[ \max  _ {a ^ {\prime} \in \mathcal {A}} Q \left(s ^ {\prime}, a ^ {\prime}\right) \right] \geq \max  _ {a ^ {\prime} \in \mathcal {A}} \mathbb {E} _ {\mathcal {B}} \left[ Q \left(s ^ {\prime}, a ^ {\prime}\right) \right] \tag {5}
$$
This section describes Trajectories-awarE Ensemble exploration (TEEN) method that achieves diverse exploration for agents. TEEN aims to enforce efficient exploration by encouraging diverse behaviors of an ensemble of  $N$  sub-policies parameterized by  $\{\phi_i\}_{i=1}^N$ . TEEN initially formulates a discrepancy measure and enforces the discrepancy among the sub-policies while simultaneously maximizing the expected return. Next, we present how to solve this optimization problem indirectly by utilizing the mutual information theory. Finally, we conduct a theoretical analysis to ensure whether the diversity of the samples collected by the ensemble policy is enhanced by solving this optimization problem. We summarize TENN in Algorithm 1.
4.1 Discrepancy Measure
We assess the diverse behaviors of policies by examining the diversity of trajectories generated by their interactions with the environments. The state-action visit distribution, indicating the probability of encountering a specific state-action pair when initiating the policy from a starting state, encapsulates the diversity of these trajectories directly. Thus, TEEN measures the diverse behaviors of policies by directly assessing their state-action visit distribution. Formally, given an ensemble of  $N$  sub-policies  $\{\pi_0,\pi_1,\dots ,\pi_k,\dots ,\pi_N\}$ , we use  $\rho^{\pi_k}$  to induce the state-action visit distribution deduced by policy  $\pi_{k}$  and use  $\rho$  for the ensemble policy. We have,
$$
\rho = \frac {1}{N} \sum_ {k = 1} ^ {N} \rho^ {\pi_ {k}} \tag {6}
$$
For convenience, we use the conditioned state-action visit probability, denoted as  $\rho(s, a|z)$ , to represent the respective state-action visit distribution of each sub-policy. This can be expressed as follows,
$$
\rho^ {\pi_ {k}} (s, a) = \rho (s, a | z _ {k}) \tag {7}
$$
Where  $\{z_k\}^N$  is the latent variable representing each policy. Our discrepancy measure  $\mathcal{KL}$ -divergence is an asymmetric measure of the difference between two probability distributions. Given this discrepancy measure, we define the difference of policies as the  $\mathcal{KL}$ -divergence of the conditioned state-action visit distribution with the state-action visit distribution.
Definition 1 (Difference between Policies) Given an ensemble of  $N$  policies  $\{\pi_0,\pi_1,\dots,\pi_k,\dots,\pi_N\}$ , the difference between policy  $\pi_{k}$  and other policies can be defined by the conditioned state-action occupation discrepancy:
$$
\mathcal {D} _ {\mathcal {K L}} \left[ \rho^ {\pi_ {k}} | | \rho \right] := \mathcal {D} _ {\mathcal {K L}} \left[ \rho (s, a | z _ {k}) | | \rho (s, a) \right] \tag {8}
$$
Consequently, we improve the difference between the conditioned state-action visit distribution to enforce diverse behaviors of sub-polices. While directly optimizing the Eq. (8) can be hard: Obtaining the state-action visit distribution can be quite challenging. In response to this, Liu, H., et al., as referenced in citation [49], have suggested the use of K-Nearest Neighbors (KNN) as an approximate measure for this distribution. We notice that it can be expressed by the form of entropy discrepancy:
$$
\mathbb {E} _ {z} \left[ \mathcal {D} _ {\mathcal {K L}} (\rho (s, a | z) | | \rho (s, a)) \right] = \mathcal {H} (\rho) - \mathcal {H} (\rho | z) \tag {9}
$$
In which the  $\mathcal{H}[\cdot ]$  is the Shannon Entropy with base  $e$ ,  $\mathcal{H}(\rho)$  is the entropy of the state-action visit distribution. With the mutual information theory, we can transform this optimization target as follows:
$$
\mathcal {H} (\rho) - \mathcal {H} (\rho | z) = \mathcal {H} (z) - \mathcal {H} (z | \rho) \tag {10}
$$
Thus, we turn to this equivalent optimization target. The first term encourages a high entropy of  $p(z)$ . We fix  $p(z)$  to be uniform in this approach by randomly selecting one of the sub-policies to explore. We have  $\mathcal{H}(z) = -\frac{1}{N}\sum_{k=1}^{N}\log p(z_k) \approx \log N$ , which is a constant. The second term emphasises the sub-policy is easy to infer given a state-action pair. We approximate this posterior with a learned discriminator  $q_{\zeta}(z|s,a)$  and optimize the variation lower bound.
$$
\begin{array}{l} \mathcal {H} (z) - \mathcal {H} (z | \rho) = \log N + \mathbb {E} _ {s, a, z} [ \log \rho (z | s, a) ] \\ = \log N + \mathbb {E} _ {s, a} \left[ \mathcal {D} _ {\mathcal {K L}} \left(\rho (z | s, a) \mid \mid q _ {\zeta} (z | s, a)) \right] + \mathbb {E} _ {s, a, z} \left[ \log q _ {\zeta} (z | s, a)) \right] \right. \tag {11} \\ \geq \log N + \mathbb {E} _ {s, a, z} [ \log q _ {\zeta} (z | s, a) ]) \\ \end{array}
$$
Where we use non-negativity of KL-divergence, that is  $\mathcal{D}_{\mathcal{KL}} \geq 0$ . We minimize  $\mathcal{D}_{\mathcal{KL}}[\rho(z|s,a)||q_{\zeta}(z|s,a)]$  with respect to parameter  $\zeta$  to tighten the variation lower bound:
$$
\nabla_ {\zeta} \mathbb {E} _ {s, a} [ \mathcal {D} _ {\mathcal {K L}} (\rho (z | s, a) | | q _ {\zeta} (z | s, a)) ] = - \mathbb {E} _ {s, a} [ \nabla_ {\zeta} \log q _ {\zeta} (z | s, a) ] \tag {12}
$$
Consequently, we have the equivalent policy gradient with regularizer:
$$
\pi^ {*} = \arg \max  _ {\pi \in \Pi} J (\pi) + \alpha \mathbb {E} _ {(s, a, z) \sim \rho} [ \log q _ {\zeta} (z | s, a)) ] \tag {13}
$$
4.2 Trajectories-aware Ensemble Exploration
We select TD3 [14] as the algorithm of sub-policies which is not a powerful exploration algorithm. We then show how to ensemble algorithm with poor exploration performance and encourage efficient ensemble exploration by solving this optimization problem. We consider an ensemble of  $N$  TD3 agents i.e.,  $\{Q_{\theta_k},\pi_{\phi_k}\}_{k = 1}^N$ , where  $\theta_{k}$  and  $\phi_{k}$  denote the parameters of  $k$ -th Q-function and sub-policy. Combined with deterministic policy gradient method [29], we update the each sub-policy by policy gradient with regularizer.
$$
\nabla J _ {t o t a l} \left(\phi_ {k}\right) = \mathbb {E} _ {s \sim \rho} \left[ \nabla_ {a} \left(Q ^ {\pi} (s, a) + \alpha \log q _ {\zeta} \left(z _ {k} | s, a\right)\right) \mid_ {a = \pi_ {\phi_ {k}} (s)} \nabla_ {\phi_ {k}} \pi_ {\phi_ {k}} (s) \right] \tag {14}
$$
Recurrent Optimization. Updating all sub-policies every time step in parallel may suffer from the exploration degradation problem [35]. Consider a state-action pair  $(s, a_1)$ , which is frequently visited by sub-policy  $\pi_k$  and another state-action pair  $(s, a_2)$  which is visited by multiple sub-policies such that we have  $q_{\zeta}(z_k|s, a_1) > q_{\zeta}(z_k|s, a_2)$ . Under this circumstance, the constraint encourages sub-policy  $\pi_k$  to execute action  $a_1$  while preventing from executing action  $a_2$  which leads to the problem that some explored state-actions are prevented from being visited by the sub-policies. To tackle this challenge, we employ a recurrent training method for our sub-policies. In particular, given an ensemble of  $N$  sub-policies, we randomly select a single sub-policy, denoted as  $\pi_k$ , every  $T$  episodes. We then concentrate our efforts on regulating the behavior of the selected sub-policy, rather than all sub-policies simultaneously.
Gradient Clip. TEEN uses  $\alpha \mathbb{E}_{(s,a,z)\sim \rho}[\log q_{\zeta}(z|s,a)]$  as a constraint to optimize the policy, while the gradient of this term is extremely large when the probability  $q_{\zeta}(z|s,a)$  is small. And a state-action with small probability to infer the corresponding sub-policy  $z_{k}$  implies that the state-action is rarely visited by sub-policy  $z_{k}$  but frequently visited by other sub-policies. The target of the constraint is to increase the discrepancy of the sub-policies, while making the sub-policy visit this state-action will instead reduce the discrepancy. Further, for a state-action with large probability  $q_{\zeta}(z|s,a)$ , continuing to increase this probability will prevent the sub-policy from exploring other possible state-actions. Thus, we use the clipped constraint and the main objective we propose is the following:
$$
\pi^ {*} = \arg \max  _ {\pi \in \Pi} J (\pi) + \alpha \mathbb {E} _ {s, a, z} [ \log \operatorname {c l i p} \left(q _ {\zeta} (z | s, a), \epsilon , 1 - \epsilon)\right) ] \tag {15}
$$
where  $\epsilon$  is a hyper-parameter, and we set  $\epsilon = 0.1$  for all of our experiments.
4.3 Controlling the Estimation Bias
In Q-value estimation, we share some similarities with Max-Min DQN [11] and Averaged DQN [50]. While Max-Min DQN takes the minimum over an ensemble of value functions as the target encouraging underestimation, Averaged DQN reduces the variance of estimates by using the mean of the previously learned Q-values. We combine these two techniques and apply both mean and min operators to reduce overestimation bias and variance. Instead of using the mean of the previously learned Q-values, we use the mean value on  $N$  actions of  $N$  sub-policies. TEEN then uses the minimum over a random subset  $\mathcal{M}$  of  $M$  Q-functions as the target estimate.
$$
Q ^ {\text {t a r g e t}} = r (s, a) + \gamma \min  _ {i = 1, 2, \dots , M} \frac {1}{N} \sum_ {j = 1} ^ {N} Q _ {i} (s, \pi_ {j} (s)) \tag {16}
$$
We give some analytical evidences from the perspective of order statistics to show how we control the estimation bias by adjusting  $M$  and  $N$ .
Theorem 1. Let  $X_{1}, X_{2}, \ldots, X_{N}$  be an infinite sequence of i.i.d. random variables with a probability density function (PDF) of  $f(x)$  and a cumulative distribution function (CDF) of  $F(x)$ . Denote  $\mu = \mathbb{E}[X_i]$  and  $\sigma^2 = \mathbb{V}[X_i]$ . Let  $X_{1:N} \leq X_{2:N} \leq X_{3:N} \dots \leq X_{N:N}$  be the order statistics corresponding to  $\{X_i\}_N$ . Denote PDF and CDF of the  $k$ -th order statistic  $X_{k:N}$  as  $f_{k:N}$  and  $F_{k:N}$  respectively. The following statements hold.
$$
\mathrm {(i)} \mu - \frac {(N - 1) \sigma}{\sqrt {2 N - 1}} \leq \mathbb {E} [ X _ {1: N} ] \leq \mu , N > 1. \mathbb {E} [ X _ {1: N + 1} ] \leq \mathbb {E} [ X _ {1: N} ]
$$
Algorithm 1 Trajectories-Aware Ensemble Exploration algorithm (TEEN)
1: Initialize  $N$  policy parameters  $\phi_{k}$ ,  $N$  Q-function parameters  $\theta_{k}$ ,  $k = 1,2,\ldots,N$   
2: Initialize target Q-function parameters  $\theta_{k}^{\prime} \gets \theta_{k}$ ,  $k = 1,2,\ldots,N$ .  
3: Initialize discriminator  $q_{\zeta}$   
4: Initialize empty replay buffer  $\mathcal{D}$   
5: Initialize recurrent interval  $d$   
6: for each iterations do  
7: Randomly sample a policy  $\pi_{k}$   
8: for each time step  $t$  do  
9: Choose one action with noise  $a_{t} = \pi_{\phi_{k}}(s) + \epsilon, \epsilon \sim \mathcal{N}(0, \sigma)$   
10: Collect state  $s_{t+1}$  and reward  $r_{t}$  from the environment by taking action  $a_{t}$   
11: Store transition  $\tau_{t} = (s_{t}, a_{t}, s_{t+1}, r_{t})$  in replay buffer  $\mathcal{B} \gets \mathcal{B} \cup \tau_{t}$   
12: end for  
13: if  $t$  mod  $d$  then  
14: Randomly select one sub-policy  $\pi_{\phi_{i}}$   
15: end if  
16: for each gradient step do  
17: Sample random minibatch  $\mathcal{B} = \{\tau_{t}\}^{B} \sim \mathcal{D}$   
18: Randomly select a set  $\mathcal{M}$  of M distinct indices from  $\{1,2,\ldots,N\}$ .  
19: Compute the  $Q$  target  $y$  by Eq. (16)  
20: For all of the value functions, update  $\theta_{k}$  using gradient decent  
21:  $\nabla_{\theta_{k}} \frac{1}{|B|} \sum_{(s,a,r,s') \in B} (y - Q_{\theta_{k}}(s,a))^{2}$   
22: For sub-policy  $\pi_{\phi_{i}}$ , update  $\phi_{i}$  by gradient decent  
23:  $\nabla_{\phi_{i}} \frac{1}{|B|} \sum_{(s,a,r,s') \in B} (\alpha \log q_{\zeta}(z_{i}|s,a) - Q_{\theta_{i}}(s,a)), a = \pi_{\phi_{i}}(s)$   
24: Update target Q-function by  $\theta_{k}' \gets \rho \theta_{k} + (1 - \rho) \theta_{k}'$   
25: Update discriminator  $q_{\zeta}$  by Eq. (12)  
26: end for  
27: end for
(ii) Let  $\bar{X} = \frac{1}{N}\Sigma_{i=1}^{N}X_{i}$ , then,  $\mathbb{E}[\bar{X}] = \mu$ ,  $Var[\bar{X}] = \frac{1}{N}\sigma^2$
Points 1 of the Theorem indicates that the minimum over an ensemble of values reduces the expected value, which implies that we can control the estimation from over estimates to under or proper estimates. The second point indicates that the mean over an ensemble of values reduces the variance. Thus, by combining these two operators, we control the estimation bias and the variance.
4.4 Theoretical Analysis