SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Woojun  $\mathbf{Kim}^{1,*}$ , Yongjae  $\mathbf{Shin}^{2,*}$ , Jongeui Park $^{2}$ , Youngchul Sung $^{2}$
1Carnegie Mellon University 2KAIST
woojunk@andrew.cmu.edu {yongjae.shin, jongeui.park, ycsung}@kaist.ac.kr
Deep reinforcement learning (RL) has achieved remarkable success in solving complex tasks through its integration with deep neural networks (DNNs) as function approximators. However, the reliance on DNNs has introduced a new challenge called primacy bias, whereby these function approximators tend to prioritize early experiences, leading to overfitting. To mitigate this primacy bias, a reset method has been proposed, which performs periodic resets of a portion or the entirety of a deep RL agent while preserving the replay buffer. However, the use of the reset method can result in performance collapses after executing the reset, which can be detrimental from the perspective of safe RL and regret minimization. In this paper, we propose a new reset-based method that leverages deep ensemble learning to address the limitations of the vanilla reset method and enhance sample efficiency. The proposed method is evaluated through various experiments including those in the domain of safe RL. Numerical results show its effectiveness in high sample efficiency and safety considerations.
With the remarkable success of deep learning in diverse fields such as image classification [10], the integration of reinforcement learning (RL) with deep learning, called deep RL, has been considered a promising approach to solving complex control tasks. In particular, using deep neural networks (DNNs) as function approximators enables the representation of state-action value function over a huge state-action space without requiring large storage as in tabular methods, which is often infeasible in complex tasks [12]. Despite its effectiveness, however, DNN-based function approximators in deep RL can lead to an overfitting problem called primacy bias, which was first investigated in Nikishin et al. [13]. Primacy bias means that DNN-based function approximators in RL may overfit to early experiences, impeding their ability as function approximators for subsequent later experiences. This phenomenon basically results from the use of an replay buffer in deep RL serving as a dynamic dataset to which experience samples are added during the training process so that experiences collected early in the training process are sampled more in training than those collected later [6]. Due to this primacy bias, it has been shown that performance tends to deteriorate as the replay ratio, defined as the number of updates per environment interaction, increases [6, 13].
To mitigate the primacy bias, Nikishin et al. [13] recently proposed a simple but effective method based on resetting. The reset method periodically resets either a portion or the entirety of a deep RL agent while preserving the replay buffer. This method enhances both the steady-state performance and sample efficiency by allowing the deep RL agent to increase the replay ratio, whereas the performance of the vanilla deep RL agent deteriorates with the increasing replay ratio. Despite its potential as a means to solve primacy bias, the reset method has a problem. That is, it causes periodic
* Equal contribution. Correspondence to: Youngchul Sung <ycsung@kaist.ac.kr>
collapses in performance immediately after the resets. The occurrence of periodic collapses is an undesirable side effect, which is an issue from the perspective of safe RL and regret minimization [13]. That is, performance collapse can lead to the violation of safety constraints, thereby limiting the applicability of the reset method to real-world RL scenarios such as autonomous vehicles. In addition, the reset method can be ineffective when the replay ratio is low, or the environment requires extensive exploration with balance with exploitation, for example, MiniGrid, as we will see in Sec. 4.
In this paper, we propose a new reset-based method that is safe—avoiding performance collapses—and achieves enhanced sample efficiency, by leveraging deep ensemble learning. In the proposed method, we first construct  $N$ -ensemble agents that adaptively integrate into a single agent based on the action value function, and then reset each agent in the ensemble sequentially. The adaptive integration of the ensemble agents prevents performance collapses after the reset by minimizing the probability of selecting the action from the recently-reset agent, also improves the sample efficiency by leveraging the diversity gain of the ensemble agents. We evaluate the proposed method combined with several deep RL algorithms such as deep Q-network (DQN) [12] and soft-actor-critic (SAC) [7], on various environments including Minigrid [5], Atari-100k [4], and DeepMind Control Suite (DMC) [17]. We show that the proposed method significantly improves the performance of the baseline algorithm in the considered environments. In addition, we apply the proposed method to a safe RL algorithm called Worst-Case SAC (WCSAC). The result demonstrates that the proposed method improves return performance while simultaneously ensuring the safety constraint. The main contributions of this work are summarized as follows:
- To the best of our knowledge, this is the first work that addresses the issue of primacy bias while simultaneously avoiding performance collapses, which can be detrimental in safety-critical real-world scenarios.
- We present a novel method that adaptively combines  $N$ -ensemble agents into a single agent, incorporating sequential resets for each ensemble agent, to effectively harness the diversity gain of the ensemble agents and prevent performance collapses.
- We empirically demonstrate that the proposed method yields superior performance compared with the base algorithm and the vanilla reset method on various environments. We also provide a comprehensive analysis of the underlying operations, including how the proposed method effectively prevents performance collapses.
- We provide a further experiment in the domain of safe RL, demonstrating that the proposed method surpasses the baseline approaches in terms of reward performance and effectively reduces the occurrence of safety constraint violations of the reset method during the training process.
We consider a Markov decision process (MDP). At each time step  $t$ , the agent selects an action  $a_{t}$  based on the current state  $s_{t}$  according to its policy  $\pi(a_{t}|s_{t})$ . The environment makes a transition to a next state  $s_{t+1}$  and yields a reward  $r_{t}$  to the agent according to the transition probability  $p(s_{t+1}|s_{t},a_{t})$  and the reward function  $r(s_{t},a_{t})$ , respectively. Through this iterative process, the policy  $\pi$  is optimized to maximize the discounted return  $R_{t} = \sum_{\tau=t}^{\infty}\gamma^{\tau}r_{\tau}$ , where  $\gamma \in [0,1)$  is the discounted factor.
Deep Ensemble Learning Deep ensemble learning has emerged as a promising approach, showing effectiveness in domains such as image classification and RL [3, 14]. It involves aggregating an ensemble of multiple DNNs to exploit their diversity and improve performance. RL with deep ensemble learning combines policies from multiple agents with the same architecture but with different initial parameters, leveraging their diversity to enhance robustness and efficiency. For example, Anschel et al. [3] proposed training and averaging multiple Q-networks for stabilization and improved performance. Lee et al. [11] employed an ensemble-based weighted Bellman backup technique that re-weights the target Q-value based on an uncertainty estimate from the Q-ensemble.
Off-Policy RL Off-policy RL algorithms aim to optimize a target policy by using experiences generated by a behavior policy to increase sample efficiency compared with on-policy RL algorithms [13]. Off-policy learning typically uses a replay buffer to store experiences and use them to train the policy. One representative off-policy RL algorithm is DQN [12], which employs a DNN as a function approximator for the Q-function, defined as  $Q_{DQN}^{\pi}(s_t,a_t)\coloneqq \mathbb{E}_{\{a_t\} \sim \pi}\left[\sum_{l = t}^{\infty}\gamma^{l - t}r_l|s_t,a_t\right]$ . The
DNN parameters of the Q-function are trained to minimize the temporal difference (TD) error. Another example is SAC [7], which aims to maximize the cumulative sum of reward and entropy to improve exploration. SAC is built on the actor-critic architecture, consisting of a policy  $\pi (\cdot |s_t)$  and soft-Q function, defined as  $Q_{SAC}^{\pi}(s_t,a_t)\coloneqq r_t + \mathbb{E}_{\tau_{t + 1}\sim \pi}\left[\sum_{l = t + 1}^{\infty}\gamma^{l - t}(r_l + \alpha \mathcal{H}(\pi (\cdot |s_l)))|s_t,a_t\right]$ . The policy is updated through soft policy iteration [7].
Safe RL Ensuring safety in RL is a crucial issue when applying RL to real-world applications such as autonomous vehicles and robotics. To address this issue, safe RL has emerged as an area of research that aims to learn policies satisfying safety constraints under a Constrained Markov decision Process (CMDP), which incorporates a cost function  $c: S \times \mathcal{A} \to \mathbb{R}$  to the conventional MDP framework. Recent safe RL research has focused on optimizing the return-based objective function while imposing constraints on the safety-related cost function [1, 9, 18]. For example, Yang et al. [18] introduced a SAC-based safety-constrained RL algorithm named WCSAC that aims to maintain the Conditional Value-at-Risk (CVaR) below a certain threshold. CVaR quantifies the level of risk aversion with respect to safety and CVaR for risk level  $\alpha$  is defined as
$$
\operatorname {C V a R} _ {\alpha_ {r i s k}} (s, a) = Q _ {\pi} ^ {c} (s, a) + \alpha_ {r i s k} ^ {- 1} \phi (\Phi^ {- 1} (\alpha_ {r i s k})) \sqrt {\operatorname {V a r} _ {\pi} ^ {c} (s , a)}, \tag {1}
$$
where  $\phi$  and  $\Phi$  denote the PDF and the CDF of the standard normal distribution, respectively, and  $Q_{\pi}^{c}(s,a)$  and  $\mathrm{Var}_{\pi}^{c}(s,a)$  respectively denote the conditional mean and the conditional standard deviation of the discounted sum of costs  $C_{\pi}(s,a) = \sum_{t=0}^{\infty}\gamma^{t}c(s_{t},a_{t})$  given  $s_0 = s$  and  $a_0 = a$ , which are called safety critic. WCSAC uses DNNs to parameterize the safety critic as well as the reward critic. Based on both the safety critic and the reward critic, the actor is trained to maximize the reward while not violating the risk level  $\alpha_{risk}$ .
Primacy Bias and Resetting Deep RL Agent. Enhancing sample efficiency in terms of environment interactions is a fundamental challenge in RL. One simple approach to improve sample efficiency is to increase the replay ratio, which is defined as the number of policy or value parameter updates per environment time step. In deep RL, however, the reliance on DNNs for function approximators can degrade performance as the replay ratio increases. This is because the deep RL agent is susceptible to overfitting on early experiences, which deteriorates the function approximation ability of DNNs for subsequent experiences. This phenomenon is referred to as primacy bias, first investigated in the context of RL by Nikishin et al. [13]. To overcome the primacy bias, Nikishin et al. [13] proposed a simple method based on increasing the replay ratio and periodically resetting either a portion or the entirety of the RL agent while preserving the replay buffer. It was shown that the deep RL algorithms employing both periodic resets and high replay ratios achieved impressive performance across several environments. In this line of research, D'Oro et al. [6] recently proposed fixing the number of updates for resetting, rather than the number of environment time steps, to maximally increase the replay ratio. This approach leads to more frequent parameter resets as the replay ratio increases, but results in improved sample efficiency in terms of interaction with the environment.
Although the existing reset methods mentioned above were shown to be effective in many tasks, certain limitations of the current reset methods exist. First, performance collapses are unavoidable immediately after executing the reset, which is not desired in the context of safe RL and regret minimization [13]. Second, the current reset method is ineffective for some off-policy RL algorithms such as DQN as well as on some environments requiring extensive exploration such as MiniGrid. Lastly, a high replay ratio is necessary to achieve the desired performance enhancement, which may not be supported in computing environments with limited resources.
To overcome the aforementioned limitations and increase sample efficiency further, we propose a simple Reset-based algorithm by leveraging Deep Ensemble learning (RDE). The proposed method involves a sequential resetting process of ensemble agents that are adaptively integrated into a single agent. The algorithm is comprised of three main components: (1) the construction of ensemble agents, (2) the sequential resetting mechanism, and (3) the adaptive integration of the ensemble agents into a single agent. With RDE, we aim to achieve superior performance in various environments while simultaneously addressing safety concerns. The overall operation of RDE is shown in Fig. 1.
Figure 1: Overall diagram of RDE: We generate N ensemble agents with unique sets of initialized parameters. During the training phase, these ensemble agents are adaptively composited into a single agent that interacts with the environment. At every  $T_{\text{reset}}$  time-step, a resetting mechanism operates by selecting a single agent  $k$  and resetting all of its parameters  $\theta_k$ . Further details regarding the adaptive composition and sequential resetting mechanism can be found in Sections 3.2 and 3.1, respectively.
3.1 Sequential Resetting Ensemble Agents
We first construct  $N$  ensemble agents, each with an identical neural network architecture but initialized with distinct parameters. In this paper, we denote the parameters of the  $k$ -th agent by  $\theta_{k}$ , where  $k \in \{1, 2, \dots, N\}$ . For instance, in the case of SAC,  $\theta_{k}$  includes all parameters for both the actor and critic of the  $k$ -th agent. Note that these ensemble agents are combined into a single agent that interacts with the environment. The integration of the ensemble into a single agent will be presented in Section 3.2.
Next, we perform the sequential and periodic resetting of the ensemble agents, while preserving the replay buffer. Specifically, every  $T_{\text{reset}}$  time-step, we reset each parameter  $\theta_1, \theta_2, \dots, \theta_N$  in sequential order with time-step gap  $T_{\text{reset}} / N$ , returning back to  $\theta_1$  after we reset  $\theta_N$ . Consequently, the existence of  $N - 1$  non-reset agents at the time of each reset operation serves to mitigate performance collapses, whereas the vanilla reset method with a single agent inevitably encounters performance collapse after each reset. As the reset operation is applied to all ensemble agents, our method can still tackle the issue of primacy bias effectively.
3.2 Adaptive Ensemble Agents Composition
Despite the potential effectiveness of utilizing an ensemble of  $N$  agents to mitigate performance collapses, the use of an ensemble alone is not sufficient. Suppose that we adopt a naive approach that selects one agent uniformly from the ensemble of the  $N$  agents and follows the action that the agent generates. Then, performance collapses may still occur since the most recently initialized agent can be selected to generate untrained random actions or biased actions. Therefore, we need a method to judiciously integrate the ensemble into a single agent, considering that a lower probability should be assigned to the selection of the recently reset agent.
To achieve this, we propose the adaptive integration of the  $N$ -ensemble agents into a single agent that interacts with the environment, inspired by Zhang et al. [19]. The proposed integration method first generates a set of actions  $\hat{a} = (a_{1}, a_{2}, \dots, a_{N})$  given a state  $s$  from the policies  $(\pi_{\theta_{1}}, \pi_{\theta_{2}}, \dots, \pi_{\theta_{N}})$  of the ensemble agents. Subsequently, we select a single action from the set of  $N$  actions with a probability distribution that is based on the action-value function. We assign a higher selection probability to the action with a higher action value, thereby reducing the chance that the most recently reset policy will be selected. Thus, the most recently reset policy is seldom selected immediately
after the reset operation, and the probability of selecting this element policy gradually increases as it is trained. The designed selection mechanism effectively mitigates performance collapses, as demonstrated in Sec. 4.3.
Let us explain the adaptive integration mechanism in detail. We design the probability distribution  $p_{\text{select}}$  of selecting an action from the set of  $N$  actions as a categorical distribution whose probability values are determined as
$$
p _ {\text {s e l e c t}} = \left[ p _ {1}, p _ {2}, \dots , p _ {N} \right] = \operatorname {s o f t m a x} \left[ \hat {Q} (s, a _ {1}) / \alpha , \hat {Q} (s, a _ {2}) / \alpha , \dots , \hat {Q} (s, a _ {N}) / \alpha \right], \tag {2}
$$
where  $p_k$  denotes the probability of selecting the action from Agent  $k$  and  $\alpha$  is the temperature parameter. To deal with the scale of the Q-function, we set the temperature parameter as  $\alpha = \max(\hat{Q}(s, a_1), \hat{Q}(s, a_2), \dots, \hat{Q}(s, a_N)) / \beta$  and adjust the coefficient  $\beta$ . Here, we choose  $\hat{Q}$  to be the estimated action-value function of the agent that underwent a reset operation at the earliest point with respect to the current time step among the ensemble agents. For example, after the  $k$ -th agent is reset, the earliest reset agent is the  $(k + 1)$ -th agent (or the first agent if  $k = N$ ). Then,  $\hat{Q} = Q_{\theta_{k+1}}$  (or  $Q_{\theta_1}$  if  $k = N$ ). The underlying rationale for adopting this approach is that the estimated Q-value function of a recently reset network is inaccurate due to lack of training time. By leveraging the oldest Q-function, which provides a more precise estimation of the true action value function, we can effectively decrease the probability of selecting action  $a_k$  that has a low return. We initialize  $p_{select}$  before the first reset operation as  $p_i = 1/N$  for all  $i$ .
In addition to its role in preventing performance collapse, our use of deep ensemble learning provides the potential to enhance performance through the diversity gained from ensemble agents. With ensemble agents initialized differently, we can effectively avoid getting trapped in local optima. Moreover, the adaptive integration allows the RL agent to balance the exploration-exploitation trade-off, in contrast to the vanilla reset method, which disrupts this balance by restarting from scratch and consequently losing its exploitation ability after a reset operation. The proposed method leverages the presence of  $N - 1$  non-reset agents, ensuring that the exploitation ability is preserved even after a reset. Thus, the proposed reset method is a more balanced approach in terms of exploitation-exploitation trade-off, and still preserves some level of exploitation while exploring the environment.
In summary, the advantages of RDE are threefold: (1) it effectively prevents performance collapses, (2) it balances the exploration-exploitation trade-off through the adaptive ensemble integration, and (3) it improves sample efficiency by addressing the primacy bias and leveraging the benefits of deep ensemble learning. The pseudo-code of the overall algorithm is provided in Appendix A.
**Environments** We consider both continuous and discrete tasks including DeepMind Control Suite (DMC) [17], Minigrid[5], and Atari-100k[4] environments. Here, we consider 9 tasks in DMC, 26 Atari games, and 5 tasks in Minigrid. The details are provided in Appendix B.
Baselines We built the vanilla reset method [13] and the proposed method, both of which were implemented on top of the base algorithms. We used SAC [8] and DQN [12] as the base algorithms in DMC and Minigrid/Atari-100K environments, respectively. For each environment, we compared three algorithms: the base algorithm (denoted as X), the vanilla reset method (denoted as  $\mathrm{SR + X}$ ), and the proposed method (denoted as  $\mathrm{RDE + X}$ ).
DNN Architectures SAC employed an actor network and two critic networks, each consisting of three multi-layer perceptions (MLPs). For Atari-100k, DQN used a Convolutional Neural Network (CNN) for the first 3 layers and followed by 2 MLPs. For Minigrid, DQN employed 5 layers MLPs.
Reset Depth Both the vanilla reset and our proposed method involve resetting parts or the entirety of DNNs. The degree of reset, referred to as the reset depth, is a hyperparameter that varies depending on the specific environment. We follow [13] in resetting the entire layers of DNNs in the SAC for the DMC environments. For the Minigrid environment, we observed that a higher reset depth is more advantageous for tasks requiring extensive exploration, which we will see in Sec. 4.3. The used reset depth in Minigrid and Atari-100k are provided in Appendix B.
(b) Atari-100k
Figure 2: IQM results of the considered algorithms on (a) DMC, (b) Atari-100k, and (c) Minigrid. The y-axis in (a)-(c) represents IQM metric, based on the test return normalized by SAC with the replay ratio of 1, the test return normalized by DQN with the replay ratio of 1, and the test return, respectively. The x-axis denotes the replay ratio.
(c) MiniGrid
Reset Frequency The reset frequency  $T_{\text{reset}}$  is a hyperparameter that is fixed with respect to the number of updates, as proposed in D'Oro et al. [6]. Accordingly, for each environment, the reset frequency in terms of timesteps is a linearly decreasing function of the replay ratio. For example, let us assume that the reset frequency is  $4 \times 10^{5}$  timesteps when the replay ratio is 1. Then, a replay ratio of 2 and 4 correspond to reset frequencies of  $2 \times 10^{5}$  and  $10^{5}$  timesteps, respectively. Note that the replay ratio can be smaller than 1. For instance, a replay ratio of 0.5 implies training DNNs every 2 timesteps. To ensure a fair evaluation, the reset frequency of each ensemble agent in RDE is equivalent to the reset frequency in the vanilla reset method. In summary, when the reset frequency with the replay ratio 1 in the vanilla reset is  $T_{\text{reset}}^{v,rr=1}$ , and the reset frequency with the replay ratio of rr with  $N$ -ensemble agents is  $T_{\text{reset}}^{v,rr=1} / (N \times rr)$ . The reset frequency values for each environment is provided in Appendix B.
We first evaluated the performance of the proposed method in comparison with the base algorithm and the vanilla reset method by varying the replay ratio. The results on DMC, Atari-100k, and MiniGrid are shown in Fig. 2, using the interquartile mean (IQM) [2] performance metric. Note that the primacy bias is observed in the base algorithm, where its performance deteriorates as the replay ratio increases, as reported by Nikishin et al. [13] and D'Oro et al. [6]. While the vanilla reset method demonstrates better performance than the base algorithm, particularly with high replay ratios, our proposed method outperforms both baselines even with low replay ratios, as shown in Fig. 2.