Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agents
Authors: Woojun Kim
Yongjae Shin
Jongeui Park
Youngchul Sung
Venue: NeurIPS 2023 poster (2023)
Keywords: deep reinforcement learning
primacy bais
reset
deep ensemble learning
TL;DR: Deep RL suffers from primacy bias. We propose a reset method that leverages deep ensemble learning to improve efficiency and safety in RL.

Abstract: Deep reinforcement learning (RL) has achieved remarkable success in solving complex tasks through its integration with deep neural networks (DNNs) as function approximators. However, the reliance on DNNs has introduced a new challenge called primacy bias, whereby these function approximators tend to prioritize early experiences, leading to overfitting. To alleviate this bias, a reset method has been proposed, which involves periodic resets of a portion or the entirety of a deep RL agent while preserving the replay buffer. However, the use of this method can result in performance collapses after executing the reset, raising concerns from the perspective of safe RL and regret minimization. In this paper, we propose a novel reset-based method that leverages deep ensemble learning to address the limitations of the vanilla reset method and enhance sample efficiency. The effectiveness of the proposed method is validated through various experiments including those in the domain of safe RL. Numerical results demonstrate its potential for real-world applications requiring high sample efficiency and safety considerations.

Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agents
Woojun Kim
Yongjae Shin? Jongeui Park², Youngchul Sung?
Carnegiee Mellon University "KAIST
woo woojunkeandrew.commm cmu. edu fyongjae. shin, jongeui.pazz ycsung) ac kr
Abstract
Deep reinforcement learning (RL) has achieved remarkable success in solving complex tasks through its integration with deep neural networks (DNNs) function approximators. However, the reliance on DNNs has introduced new challenge called primacy bias, whereby these function approximators tend to prioritize early experiences. leading to overfitting. To mitigate this primacy bias, reset method has been proposed, which performs periodic resets of portion the entirety of a deep RL agent while preserving the replay buffer. However, the use of the reset method can result in performance collapses after executing the reset, which can be detrimental from the perspective of safe RL and regret minimization. In this paper, we propose new reset-based method that leverages deep ensemble learning to address the limitations of the vanilla reset method and enhance sample efficiency.
The proposed method is evaluated through various experiments including those the domain of safe RL. Numerical results show its effectiveness in high sample efficiency and safety considerations.
Introduction
With the remarkable success deep learning in diverse fields such as image classification 10, the integration of reinforcement learning (RL) with deep learning, called deep RI., has been considered a promising approach to solving complex control tasks. In particular, using deep neural networks (DNNs) as function approximators enables the representation of state-action value function over huge state-action space without requiring large storage as in tabular methods, which is often infeasible in complex tasks [1]. Despite its effectiveness, however, DNN-based function approximators in deep RL can lead to an overfitting problem called primacy bias, which was first investigated in Nikishin et al. [13. Primacy bias means that DNN-based function approximators in RL may overfit to early experiences, impeding their ability as function approximators for subsequent later experiences. This phenomenon basically results from the use of replay buffer in deep RL serving as dynamic dataset to which experience samples are added during the training process so that experiences collected early in the training process sampled more in training than those collected later [6]. Due this primacy bias, has been shown that performance tends to deteriorate as the replay ratio, defined as the number of updates per environment interaction, increases 6 [3]
To mitigate the primacy bias, Nikishin et al. [3] recently proposed a simple but effective method based on resetting. The reset method periodically resets either portion the entirety of a deep RL agent while preserving the replay buffer. This method enhances both the stcady-state performance and sample efficiency by allowing the deep RL agent to increase the replay ratio, whereas the performance of the vanilla deep RL agent deteriorates with the increasing replay ratio. Despite its potential means to solve primacy bias, the reset method has problem. That is, causes periodic
Equal contribution. Correspondence to: Youngchul Sung <ycsung @ @ak
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

collapses in performance immediately after the resets. The occurrence of periodic collapses is an undesirable side effect, which an issue from the perspective of safe RL and regret minimization 13]. That is, performance collapse can lead to the violation of safety constraints, thereby limiting the applicability of the reset method to real-world RI. scenarios such as autonomous vehicles. In addition, the reset method can be ineffective when the replay ratio is low, or the environment requires extensive exploration with balance with exploitation, for example, MiniGrid, as we will see in Sec. T
In this paper, we propose a new reset-based method that is safe- performance collapses- and achieves enhanced sample efficiency, by leveraging deep ensemble learning. In the proposed method, we first construct N N-ensemble agents that adaptively integrate into single agent based on the action value function, and then reset each agent in the ensemble sequentially. The adaptive integration of the ensemble agents prevents performance collapses after the reset by minimizing the probability of selecting the action from the recently-reset agent, also improves the sample efficiency by leveraging the diversity gain of the ensemble agents. We evaluate the proposed method combined with several deep RI. algorithms such as deep Q-network (DQN) [2 and soft-actor-critic (SAC) (7]. various environments including Minigrid [5]. Atari- 100k 44. and DeepMind Control Suite (DMC) [77. We show that the proposed method significantly improves the performance of the baseline algorithm in the considered environments. In addition, we apply the proposed method to a safe RL algorithm called Worst-Case SAC (WCSAC). The result demonstrates that the proposed method improves return performance while simultaneously ensuring the safety constraint. The main contributions of this work are summarized as follows:
To the best of our knowledge, this is the first work that addresses the issue of primacy bias while simultaneously avoiding performance collapses, which can be detrimental in safety-critical real-world scenarios.
We present a novel method that adaptively combines N N-ensemble agents into a single agent, incorporating sequential resets for each ensemble agent, to effectively harness the diversity gain of the ensemble agents and prevent performance collapses.
We empirically demonstrate that the proposed method yields superior performance compared with the base algorithm and the vanilla reset method on various environments. We also provide comprehensive analysis of the underlying operations, including how the proposed method effectively prevents performance collapses.
We provide further experiment in the domain of safe RL, demonstrating that the proposed method surpasses the baseline approaches in terms of reward performance and effectively reduces the occurrence of safety constraint violations of the reset method during the training process.
2 Preliminaries and Related Works
We consider a Markov decision process (MDP). At each time step t, the agent selects action ag based on the current state s; according to its policy π [o)])s)). The environment makes a transition to a next state St+1 and yields a reward Te to the agent according to the transition probability p($ (8,, az) and the reward function r(st, az), respectively. Through this iterative process, the policy is optimized to maximize the discounted return R. Σ rr, where y E [0,1) is the discounted factor.
Deep Ensemble Learning Deep ensemble learning has emerged promising approach, showing effectiveness in domains such as image classification and RL 3 [4] It involves aggregating an ensemble of multiple DNNs to exploit their diversity and improve performance. RL. with deep ensemble learning combines policies from multiple agents with the same architecture but with different initial parameters, leveraging their diversity to enhance robustness and efficiency. For example, Anschel et al. 3 proposed training and averaging multiple Q-networks for stabilization and improved performance. Lee et al. |III employed an ensemble-based weighted Bellman backup technique that re- weights the target Q Q-value based on an uncertainty estimate from the Q-ensemble.
Off-Policy RL Off-policy RL algorithms aim to optimize target policy by using experiences generated by a behavior policy to increase sample efficiency compared with on-policy RL algorithms [13 Off-policy learning typically uses a replay buffer to store experiences and use them to train the policy. One representative off off-policy RI. algorithm DQN [12, which employs a DNN as a function
-0 1 St, ar). The approximator for the Q-function, defined as QDQ (St, as) == Eaasoo //

---

DNN parameters of the Q-function are trained to minimize the temporal difference (TD) error. Another example is SAC [7]. which aims to maximize the cumulative sum of reward and entropy to improve exploration. SAC is built on the actor-critic architecture, consisting of policy x (t-1ss) and soft-Q function, defined as Ac 0z) == Tt + Er Die0+ of 1- (Ti (-(((((():)) at]. The policy is updated through soft policy iteration 77.
Safe RL Ensuring safety in RL is a crucial issue when applying RL to real-world applications such autonomous vehicles and robotics. To address this issue, safe RL has emerged an area of research that aims to learn policies satisfying safety constraints under a Constrained Markov decision Process (CMDP), which incorporates a cost function : S x A framework. Recent safe RI. research has focused on optimizing the return-based objective function while imposing constraints on the safety-related cost function I 9 [8] For example, Yang et al. [88 introduced a SAC-based safety-constrained RL algorithm named WCSAC that aims to maintain the Conditional Value-at-Risk (CVaR) below certain threshold. CVaR quantifies the level of risk aversion with respect to safety and CVaR for risk level a is defined as
IR to the conventional MDP
CVa (8, = 27 a) + orisk (($ ((rrssk) )) Var (8,a);
where & and & denote the PDF and the CDF of the standard normal distribution, respectively, and (s, a) and Var (s. a) respectively denote the conditional mean and the conditional standard deviation of the discounted sum of costs C2 (s, a) Et= y'c(st, az) given so and dg which are called safety critic. WCSAC uses DNNs to parameterize the safety critic as well as the reward critic. Based on both the safety critic and the reward critic, the actor is trained to maximize the reward while not violating the risk level Orisk-
Primacy Bias and Resetting Deep RL Agent. Enhancing sample efficiency in terms of environment interactions is a fundamental challenge in RL. One simple approach to improve sample efficiency is to increase the replay ratio, which is defined as the number of policy value parameter updates per environment time step. In deep RL, however, the reliance on DNNs for function approximators can degrade performance as the replay ratio increases. This is because the deep RL agent is susceptible to overfitting on early experiences, which deteriorates the function approximation ability of DNNs for subsequent experiences. This phenomenon is referred to as primacy bias, first investigated in the context of RL by Nikishin et al. [13]. To overcome the primacy bias, Nikishin et al. [33 proposed a simple method based on increasing the replay ratio and periodically resetting either portion the entirety of the RL agent while preserving the replay buffer. It was shown that the deep RL algorithms employing both periodic resets and high replay ratios achieved impressive performance across several environments. In this line of research, D'Oro et al. [6] recently proposed fixing the number of updates for resetting, rather than the number of environment time steps, maximally increase the replay ratio. This approach leads to more frequent parameter resets as the replay ratio increases, but results in improved sample efficiency in terms of interaction with the environment.
Although the existing reset methods mentioned above were shown to be effective in many tasks, certain limitations of the current reset methods exist. First, performance collapses are unavoidable immediately after executing the reset, which is not desired in the context of safe RL. and regret minimization [3). Second, the current reset method is ineffective for some ofllpy RL algorithms such as DQN as well as on some environments requiring extensive exploration such as MiniGrid. Lastly, high replay ratio is necessary achieve the desired performance enhancement, which may not be supported in computing environments with limited resources.
3 Methodology
To overcome the aforementioned limitations and increase sample efficiency further, we propose simple Reset-based algorithm by leveraging Deep Ensemble learning (RDE). The proposed method involves sequential resetting process of ensemble agents that are adaptively integrated into a single agent. The algorithm is comprised of three main components: (1) the construction of ensemble agents, (2) the sequential resetting mechanism, and (3) the adaptive integration of the ensemble agents into a single agent. With RDE, we aim to achieve superior performance in various environments while simultaneously addressing safety concerns. The overall operation of RDE is shown in Fig. [
3

---

N-Ensemble Agents
Select a single agent
M
Adaptive Composition
a2
Composited Agent
Environment
Reset 6
8co
Store transition
Train RR times
Replay Buffer
8c4-1
if (timestep) % (reset frequency) == 0: then reset
We generate ensemble agents with unique sets of initialized parameters. During the training phase, these ensemble agents are adaptively composited into single agent that interacts with the environment. At every Treset time-step, a resetting mechanism operates by selecting single agent k and resetting all of its parameters 0ş. Further details regarding the adaptive composition and sequential resetting mechanism can be found in Sections 3.2 and respectively.
Figure 1: Overall diagram of RDE:
3.1 Sequential Resetting Ensemble Agents
We first construct N ensemble agents, each with identical neural network architecture but initialized with distinct parameters. In this paper, we denote the parameters of the k-th agent by Os, where k E 21, N]. For instance, in the case of SAC, 0s includes all parameters for both the actor and critic of the k-th agent. Note that these ensemble agents are combined into a single agent that interacts with the environment. The integration of the ensemble into a single agent will be presented in Section 3.2
Next, we perform the sequential and periodic resetting of the ensemble agents, while preserving the replay buffer. Specifically, every Treset time-step, we reset each parameter 01, 62. ..., 0 in sequential order with time-step gap Treset/N, returning back to 01 after we reset dx. Consequently, existence
non-reset agents at the time of each reset operation serves to mitigate performance collapses, whereas the vanilla reset method with single agent inevitably encounters performance collapse after each reset. As the reset operation is applied to ensemble agents, our method can still tackle the issue of primacy bias effectively.
3.2 Adaptive Ensemble Agents Composition
Despite the potential effectiveness of utilizing an ensemble of N agents to mitigate performance collapses, the use of an ensemble alone is not sufficient. Suppose that we adopt naive approach that selects one agent uniformly from the ensemble the N agents and follows the action that the agent generates. Then, performance collapses may still occur since the most recently initialized agent can be selected to generate untrained random actions or biased actions. Therefore, we need a method to judiciously integrate the ensemble into single agent, considering that lower probability should be assigned to the selection of the recently reset agent.
To achieve this, we propose the adaptive integration of the N-ensemble agents into single agent that interacts with the environment, inspired by Zhang al. [19]. The proposed integration method first generates a set of actions à = (a1, 022 .aN) given a state from the policies (To, Te., Tox) of the ensemble agents. Subsequently, we select single action from the set of N actions with probability distribution that is based on the action-value function. We assign higher selection probability to the action with higher action value, thereby reducing the chance that the most recently reset policy will be selected. Thus, the most recently reset policy is seldom selected immediately

---

after the reset operation, and the probability of selecting this element policy gradually increases as it is trained. The designed selection mechanism effectively mitigates performance collapses, as demonstrated in Sec. 4.3
Let us explain the adaptive integration mechanism in detail. We design the probability distribution Paelect of selecting an action from the set of N actions as categorical distribution whose probability values are determined
Psotect [pt: P2, .pN] softmax [ Q(+, (2)/,a))a..... (,....,(ssss,, Q(s,aa((/a)
(2)
where Pa denotes the probability of selecting the action from Agent l and a is the temperature parameter. To deal with the scale of the Q-function, we set the temperature parameter as a max(Q(s, a1), ((s, az),... (s, ax) / and adjust the coefficient B. Here, we choose Q to be the estimated action-value function of the agent that underwent a reset operation at the earliest point with respect to the current time step among the ensemble agents. For example, after the k-th agent is reset, the earliest reset agent is the (A + 1) agent (or the first agent if k = N). Then, Q = Q0 (or Qo, if A N). The underlying rationale for adopting this approach is that the estimated Q-value function of a recently reset network is inaccurate due to lack of training time. By leveraging the oldest Q-function, which provides a more precise estimation of the true action value function, can effectively decrease the probability of selecting action ag that has a low return. We initialize Pretect before the first reset operation as pt 1/N for all i.
In addition to its role in preventing performance collapse, our use of deep ensemble learning provides the potential to enhance performance through the diversity gained from ensemble agents. With ensemble agents initialized differently, we can effectively avoid getting trapped in local optima. Moreover, the adaptive integration allows the RL agent to balance the exploration-exploitiiii off, in contrast to the vanilla reset method, which disrupts this balance by restarting from scratch and consequently losing its exploitation ability after a reset operation. The proposed method leverages the presence of N --1 non-reset agents, ensuring that the exploitation ability is preserved even after a reset. Thus, the proposed reset method is more balanced approach in terms of xxpliitaiion-exlloaaiio trade-off, and still preserves some level of exploitation while exploring the environment.
trade-
In summary, the advantages of RDE are threefold: (1) it effectively prevents performance collapses, (2) it balances the eexpltotinn trade-off through the adaptive ensemble integration, and (3) it improves sample efficiency by addressing the primacy bias and leveraging the benefits of deep ensemble learning. The pseudo-code of the overall algorithm is provided in Appendix A
Experiments
4.1 Experimental Setup
Environments
We consider both continuous and discrete tasks including DeepMind Control Suite (DMC) [17], MS) and Atari- 4 environments. Here, we consider 9 tasks in DMC, 26 Atari games, and 5 tasks in Minigrid. The details are provided Appendix B
Baselines We built the vanilla reset method [3 and the proposed method, both of which were implemented on top of the base algorithms. We used SAC [8] and DQN [12) as the base algorithms in DMC and Minigrid/ Minigrid/Attari-0009 Ator environments, respectively. For each environment, we compared three algorithms: the base algorithm (denoted as X), the vanilla reset method (denoted as SR+X), and the proposed method (denoted as RDE+X).
DNN Architectures SAC employed an actor network and two critic networks, each consisting of three multi-layer perceptions (MLPs). For 100k, DQN used a Convolutional Neural Network (CNN) for the first 3 layers and followed by 2 MLPs. For Minigrid, DQN employed 5 layers MLPs.
Reset Depth Both the vanilla reset and our proposed method involve resetting parts or the entirety of DNNs. The degree of reset, referred to as the reset depth, is hyperparameter that varies depending on the specific environment. We follow [13 in resetting the entire layers of DNNs the SAC for the DMC environments. For the Minigrid environment, we observed that a higher reset depth is more advantageous for tasks requiring extensive exploration, which we will see in Sec. 43 The used reset depth in Minigrid and Atari- 100k are provided in Appendix B
5

---

(ours)
1.4
1.1
1.2
IOM
1.0
0.9
2
2
0.5
Replay Ratio
Replay Ratio
Replay Ratio (c) MiniGrid
(a) DMC
(b) Atari-100k
Figure 2: IQM results the considered algorithms on (a) DMC, (b) Atari-100k,, and (c) Minigrid. The y-axis in (a)-(c) represents IQM metric, based on the test return normalized by SAC with the replay ratio of 1, the test return normalized by DQN with the replay ratio of 1, and the test return, respectively. The x-axis denotes the replay ratio.
Reset Frequency The reset frequency Treset is a hyperparameter that fixed with respect to number of updates, as proposed in D'Oro et al. [6]. Accordingly, for each environment, the reset frequency in terms timesteps a linearly decreasing function the replay ratio. For example, let us assume that the reset frequency is 4 x 10y timesteps when the replay ratio is 1. Then, replay ratio of 2 and 4 correspond to reset frequencies
2 x 10 and 10 timesteps, respectively. Note that the replay ratio be smaller than For instance, replay ratio of 0.5 implies training DNNs every 2 timesteps. To ensure fair evaluation, the reset frequency of each ensemble agent in RDE is equivalent to the reset frequency in the vanilla reset method. In summary, when the reset frequency with the replay ratio in the vanilla reset is Turrz= and the reset frequency with the replay ratio of rr with N-ensemble agents is Terr== /(N X rr). The reset frequency values for each environment is provided in Appendix B
4.2 Performance Comparison
We first evaluated the performance of the proposed method comparison with the base algorithm and the vanilla reset method by varying the replay ratio. The results on DMC, Atari-100k, and MiniGrid are shown in Fig. 2 using the interquartile mean (IQM) 22 performance metric. Note that the primacy bias is observed in the base algorithm, where its performance deteriorates the replay ratio increases, as reported by Nikishin et al. [33 and D' D'Oro et al. (6]. While the vanilla reset method demonstrates better performance than the base algorithm, particularly with high replay ratios, our proposed method outperforms both baselines even with low replay ratios, as shown Fig. 2
The performance results of the proposed method and the baselines on the considered environments are shown Fig. 3 where the considered tasks are hopper- hhop and humanoid run in DMC and Fourroom and GotoDoor in MiniGrid. It is seen that the proposed method not only yields superior also prevents performance collapses. In the Fourroom environment, the proposed method solves the game whereas both the base algorithm and the vanilla reset method fail to learn. The inherent loss of exploitation ability following a reset operation in the vanilla reset method prevents effective learning of this hard task. In other environments, the vanilla reset method suffers from performance collapses. Especially humanoid-rue there is clear difference between our method and the vanilla reset method: immediately after reset operation, our method sustains performance without significant drop, whereas the vanilla reset method collapses to the initial performance.
performance the baselines
expected,
More results on Atari-100k are provided Appendix C
4.3 Analysis
Performance Collapse The results in Sec. 4.2 demonstrate the effectiveness of the proposed method eliminating performance collapses. As aforementioned, the probability selecting an action from the ensemble of N actions is determined based Eq. 2 which assigns a lower selection

---

X (ours)
5.0
500
200
Tell fulur
Tesl Rehum
Teat Raturn
302
300
202
200
Tee:
100
100
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.
1.0
Time
Time
10")
Time or
(a) hopper-hop (1)
(b) hopper-hop (4)
(c) humanoid-run (1)
(d) humanoid-run (4)
0.25
0.25
1.00
1.00
0.20
0.20
Tea Retam
0.15
Rea.
Text Ratam
0.75
0.50
0.50
0.10
0.10
0.05
0.25
0.00
0.00
0.00
0.6
0.8
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
3.5
2.0
Time step e 109)
Time 5000 10')
Time stop 10f1
Timo stop Dr 10')
(e) FourRoom (0.5)
(f) FourRoom (2)
(g) Goto (0.5)
(h) Goto (2)
Figure 3: Performance comparison on hopper-hop and humanoid- in DMC and Fourroom and GotoDoor in MiniGrid. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is 10 Performances are averaged over 5 seeds.
probability to an action of a recently reset agent. This effectively prevents performance collapses, and the degree of this effect can be controlled by adjusting the parameter B. To investigate the impact of the parameter B, we conducted experiments on the humanoid-run task using RDE+SAC with
N = 2. We varied B across {-10, O, 50, 300} and measured the corresponding empirical prob- ability Pt. which denote the prob- ability of selecting the action gen- erated by 0j. In Fig. (b), we ob- serve that the empirical P1 experi- ences a sudden drop immediately after resetting 0,. On the other hand, when 02 is reset, P1 jumps up, indicating the sudden drop of Pa. Note that the occurrence of performance collapses was nearly climinated when 3 was set to 300. This demonstrates the effective- ness of our adaptive integration in mitigating performance collapses by appropriately reducing the probability of selecting actions from recently-initialized agents. Fur- thermore, increasing the value of B3 quickly adapt the values of Pselects leading to effective reduction of performance collapses.
8+0
$
300
Reset O
1.0
150
0.8
Test Return
Empircal p.
100
0.6
0.4
50
0.2
0
0,6
0.00
0.25
0.50
1.00
10.25- -10.501
1.00
Time stop (x 10")
Reset 0
Time step (* 10%
(a) Performance
(b) Empirical P1
Figure 4: Performance and empirical probability of action selection from 01 for RDE+SAC with varying 3 values on humanoid-rum.
Reset Depth The choice of reset depth is a crucial hyperparameter that significantly impacts the final performance of reset-based methods. We observed that higher reset depth can provide an advantage in more challenging tasks that require hard exploration, whereas may present slight disadvantage in relatively easy tasks. We provide the performance of RDE+DQN with two different reset depths: reset-part, which resets the last two layers, and reset-all, which completely reset all layers. These evaluations were conducted on three MiniGrid environments, comprising two challenging environments, FourRoom and SimpleCrossing,, and one comparatively easier environment, GotoDoor. As shown in Fig- 55] reset-part exhibits inferior performance and instability compared with reset-all two relatively challenging tasks. However, in two relatively easy tasks, the reset-pt performs marginally better than reset-all.
7

---

RDE (reset all)
RDE (reset part)
RDE (wlo reset)
RDE (N
0.25
0.20
Test Rotur
0.15
0.10
Teel
0.10
0.50
0.25
0.05
0.00
0.00
0.2
0.4
0.8
1.0
1.0
Time stop (s 10P)
step (x 10%)
Time (K 10t)
(a) FourRoom (I)
(b) SimpleCrossing) (1)
(c) GotoDoor (2)
Figure 5: Ablation studies regarding reset depth, ensemble learning, and reset mechanism. Perfor- mances are averaged over 5 seeds.
Ensemble and Reset Effect The effectiveness of our proposed method is due key factors: ensemble learning and reset. To assess the contributions of these factors, we conducted two ablation studies by evaluating the performance of RDE by (1) varying the number of ensemble agents N, and (2) eliminating the reset mechanism. In the first ablation study, we compared the performance of RDE+DQN with N 4 and N on the MiniGrid environments. The results, shown in Fig. 5 demonstrate that increasing the number of ensemble agents improves the performance of the proposed method. In the second ablation study, we examined the performance of RDE+DQN with and without reset. Here, the latter approach solely relies on deep ensemble learning, As shown in Fig. 5] the reset mechanism significantly contributes to the enhanced performance of the proposed method, especially in the Roor environment.
Experiment: Safe Reinforcement Learning
As previously mentioned, Safe RL aims to find high-return policy while avoiding unsafe states and actions during training much as possible. Ray et al. [16) formulated the problem as a constrained RL problem, where the agent incurs a cost when visits hazardous state. The goal maximize the expected return while ensuring that the cumulative cost remains below predefined threshold. the other hand, Jung et al. (9] considered maximizing the expected return while satisfying the probability of outage events. When applying the reset mechanism in such setting, challenge arises because the deep RL agent loses all its knowledge about states that should be avoided after operating reset. Consequently, until the safety critic is sufficiently retrained, the agent may continue to visit these undesirable states, thereby increasing the overall training cost. We expect that RDE with slight modification taking into account both reward and cost can maintain the training cost low while benefiting from the performance gains associated with the reset operation still.
For safe RL, we adopted WCSAC, which maximizes the expected return while constraining the CVaR defined as Eq. (T). and modified the adaptive integration method proposed Sec. 3.2, Specifically, we modified the probability of action selection in Eq. (2) by incorporating the cost function. The
modified probability of adaptive action selection is now defined as paafe KPsetect + (1 Papettt where & is the mixing coefficient and Pretectt is given by
Pretect pff PT.
pxx softmax [-
01) (s)/aa ((s
"., (3)
where a max ( a1).. [Č(s, az))...., (C(s, ax)1//3. Here, we choose Ò to be the estimated CVaR value function that underwent a reset operation at the earliest point with respect to the current time step among the ensemble agent. Note that the sign of Č(s, a) in Eq. ( is inverted to prioritize
actions with low CVaR values, as we aim minimize the cost function.
5.1 Result
We compared our algorithm with WCSAC and SR-WCSAC, which incorporates the reset method into WCSAC, on the 3 environments in Safety-Gym, The details are provided in Appendix B

---

(ours)
30
20
10
20
Test Retuun
10
Test Retum
Test Relum
-10
-20
-20
-30
-30
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Epoch
Epach
Epoch
600
400
Train Cost
400
Train Coet
600
Train Cost
200
200
40
100
40
100
40
80
100
Epoch
Epoch
Epoch
(a) PointGoal
(b) PointButton
(c) CarGoal
Figure 6: Performance Comparison in Safety Gym Environments. The first and second rows show the test returns and training costs, respectively. The performance metrics are averaged over five different seeds.
Fig. shows the test return and the training cost. As expected, although the application of the simple reset method leads slight performance improvement compared with the baseline WCSAC agent, also incurred a higher training cost. In particular, the soaring cost makes the naive reset method impractical in real-world scenarios concerning safety. On the other hand, our algorithm achieved superior test performance compared with the simple reset method, while significantly reducing the training cost. Note that the presence of non-reset agents and adaptive composition prevent cost from soaring after the reset operation.
6
Conclusion
We have proposed a novel RDE (Reset with Deep Ensemble) framework that ensures safety and enhances sample efficiency by leveraging deep ensemble learning and the reset mechanism. The proposed method effectively combines N ensemble agents into a single agent to mitigate performance collapse that follows reset. The experimental results demonstrate the effectiveness of the proposed method, which significantly improves sample efficiency and final performance, while avoiding performance collapse in various reinforcement learning tasks. In addition, the context of safe reinforcement learning, our method outperforms the vanilla reset approach without incurring high costs, whereas the latter suffers from prohibitive training cost.
Acknowledgments
This work was supported in part by Institute of Information Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-0044) Devel- opment of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones) and in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2021R1A2222009 Information Theory-Based Reinforcement Learning for Generalized Environments).

---

References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017. Constrained policy opti- mization. In International conference on machine learning. PMLR, 22-31.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. 2021. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems 34 (2021), 19304-293200
[3] Oron Anschel, Nir Baram, and Nahum Shimkin. 2017. Averaged-dqu- Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning. PMLR, 176-185.
[4] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research 47 (2013), 253-279.
[5] Maxime Chevalier-Boisee environment for openai gym. (2018).
Lucas Willems, and Suman Pal. 2018. Minimalistic gridworld
[6] Pierluca D' D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. 2023, Cample-Efficieee Reinforcement Learning by Breaking the Replay Ratio Barrier. In The Eleventh International Conference on Learning Representations.
[7] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. PMLR, 1861-1870.
[8] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. 2018. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.0550 (2018).
[9] Whiyoung Jung, Myungsik Cho, Jongeui Park, and Youngchul Sung. 2022. Quantile Con- strained Reinforcement Learning: Reinforcement Learning Framework Constraining Outage Probability. Advances in Neural Information Processing Systems 35 (2022), 6437-6449.
[10] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84-90.
[11] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. 2021. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, PMLR, 6131-6141.
[12] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529-533.
[13] Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. 2022. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning. PMLR, 16828-16847.
[14] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. 2016. Deep explo- ration via bootstrapped DQN. Advances in neural information processing systems 29 (2016).
[15] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. Stable-Baselines;; Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22, 268 (2021), 1-8. http://jmlr org/ /
v22/20- 1364. html
[16] Alex Ray, Joshua Achiam, and Dario Amodei. 2019. Benchmarking Safe Exploration in Deep Reinforcement Learning. (2019).
[17] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. 2018. Deepmind control suite. arXiv preprint 1rXiv:1801.00660 (2018).
10

---

[18] Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. 2021. WCSAC: Worst-case soft actor critic for canety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 10639-10646.
[19] Haichao Zhang, Wei Xu, and Haonan Yu. 2023. Policy Expansion for Bridging Offline-to- Online Reinforcement Learning. In International Conference on Learning Representations (ICLR).

---

A Algorithm Pseudo-code
Algorithm 1 RDE+X
Input: Algorithm X, Ensemble size N, Reset frequency Treset, Replay Ratio RR, Coefficient B, Initialize -aagents parameters, (01, 02, 0v), and the replay buffer D. Set index l for each episode do for each timestep do for i = 1 to N do
Collect action of ~ (ajls))
end for
Calculate Pselect according to Eq. 2 and sample a, based on Pselect-
Execute and get and 8141. Store transition to D
for jorj= RR do
Sample random minibatch from D Update (01, 02 ... (2) based on X end for
if % (reset/N) == then Reset 02 and +- (A: 1)%N end if end for
end for
B Experimental Details and Hyperparameter
1. DeepMind Control Suite. DeepMind Control Suite (DMC) is collection of continuous control tasks that involve the manipulation of high-dimensional systems [17]. We considered nine distinct tasks of DMC, listed in Table 2 Our implementation is based Stable Baseline3 (15) and the
hyperparameters are provided in Table
Hyperparameters
Value
# of ensemble agents Training steps Discount factor Initial collection steps Minibatch size
2
1 x 106
0.99
5000
Environment Task
1024
Optimizer (all)
Adam
acrobot swingup
Optimizer (all): learning rate
0.0001
humanoid-run
cheetah run finger turn_hard
0.0003 otherwise
Networks (all) activation
ReLU
fish swim
Networks (all): n. hidden layers
2 1024
hopper hop
Networks (all) hidden units
humanoid run
Initial Temperature
1
quadruped
Replay Buffer Size
1 x 106
swimmer swimmer15
Updates per step (Replay Ratio)
(1,
walker run
Target network update period
0.005
Table 2: The considered tasks of
DMC
Reset Interval (gradient steps)
4 x 105 50
(action select coefficient)
Table 1: Hyperparameters for RDE+SAC on DMC.
12

---

2. Atari 100k. Atari 100k is a benchmark that tests an agent's abilities by allowing it to interact with 100k environment steps (equivalent 400k frames with frameskip 4) in 26 Atari games []. Each game has different mechanics, providing diverse evaluation of the agent's capabilities. The benchmark imposes restriction of 100k actions per environment, which roughly corresponds to 2 hours of human gameplay. The hyperparameters are provided in Table 3 and Table 4 Our implementation based on Stable Baseline3 [5].
Reset Interval
Environments
4 x 10*
Assault, Asterix, Battle Zone, Boxing, Crazy CLimber, Freeway, Froatbite, Krull, Ma Pacman, Qbert, Seaquest, Up N Down
8 x 10
Alien, Amidar, Bank Heist, Breakout, Chopper Command, Demon Attack, Copher, Hero, Jameabond, Kangaroo, Kung Fu Master, Pong, Private Eye, Road Runner
Table 3: Reset Interval in terms of the gradient step for each environment Atari-100k
Hyperparameters
Value
# of ensemble agents
2
Gray-scaling
True
Observation down-sampling
84 x 84
Frames stacked
4
Action repetitions
4
Reward clipping
[-1, 1]
Terminal on loss of life
True
Max gradient norm
10
Replay periode every
1 step
Training steps
1 x 10
Discount factor
0.99
Initial collection steps
x 104 32 Adam
Minibatch size
Optimizer
Optimizer: learning rate
0,0001
Q network: channels
32, 64, 64 8 8, 4 3
Q network: filter size
Q network: : stride
4,
Q network: activation
ReLU
Q network: : hidden units
512
Replay Buffer Size
1x 105
Updates step (Replay Ratio)
21,
Target network update period Exploration e-decay
e-greedy
1 x 10
0.005
B (action select coefficient)
50
Reset depth
last 2 layers: Amidar, Asterix Bank Heist, Freeway, Frostbite Gopher, Hero, Kangaroo, Pong last layer: otherwise
Table 4: Hyperparameters for RDE+DQN on Atari 100k.
13

---

3. MiniGrid.
Minigrid is collection of goal orientr tasks in 2D grid- The agent receives sparse reward R 10 if the goal is achieved. The spare reward penalized based on the number of interaction steps. In this paper, we considered 5 tasks of Minigrid including FourRooms, SupleeCoossing9N
SovaCrossingSNN,,, 3mmpleCrossingSMM, and GoToDoor-8zz We now provide the brief introduction of each environment.
FourRooms This environment designed with four rooms and comprises single agent and a green goal. At the start of each episode, both the agent and the green goal are randomly placed within the four rooms. The goal of the agent is to navigate through the environment and ultimately reach the green goal.
LavaCrossingS9Ni This environment is designed with two rooms that blocked by obstacles, such as lava (for LavaCrossing) and walls (for SimpleCrossing). The objective successfully reach goal while avoiding these obstacles. In LavaCrossing, the episode comes to an end if the agent collides with the obstacle, whereas in SimpleCrossing, the episode continues despite the collision.
SimpleCrosiigSSNLL
GoToDoor-8x8 This environment designed with a single room, four doors, and single mission text string. The string provides instructions which door the agent should reach.
LavaGapS7 This environment is designed with a single room, strip of lava, and a green goal. The objective is to successfully reach the goal while avoiding the lava.
We provide the hyperparameters used in MiniGrid follows.
Hyperparameters
Value
0.9 + 0.05
dedc time step target update period
105
103
Replay buffer size
x 10
Mini-batch size
256
Optimizer
RMSProp
Learning rate
0.0001
The maxmimum number of steps
100
Reset Interval (gradient steps)
2 x 10sh (GoToDoor, LavaCrossing, LavaGap)
x 10s (FourRooms, SimpleCrossing)
3 (action select coefficient)
50
Table 5: Hyperparameters MiniGrid.
14

---

4. Safety-Gym. Safety Gym presents a collection of environments having safety constraints. Within each of these environments, a robotic entity operates within intricate setting with the objective of fulfilling a task while maintaining observance to predefined constraints governing its interactions with nearby objects and spatial regions. These environments incorporate separate reward and cost functions, which respectively describe task-specifie objectives and safety prerequisites. We considered 3 environments including PointGoal, PointBut and CarGoal. We now provide the brief introduction of each environment.
PointGoal singular point robot tasked with traversing toward a specific goal while effectively evading hazards. The point robot's mobility is confined to 2-dimensional plane, facilitated by separate actuators enabling rotational motion and moving worward/backward.
PointButton This environment requires the point robot to sequentially press a series of goal buttons. These buttons, immobile in nature, are dispersed throughout the environment, compelling the agent to navigate and press towards the currently highlighed button, serving as the immediate goal. CarGoal This environment introduces a robot of increased complexity, equipped with two inde- pendently driven parallel wheels and an additional freely rotating rear wheel. Effective coordination of both turning and forward/backward locomotion necessitates concurrent manipulation of both actuators. The objective of this environment coincides with that of the PointGoal environment.
Hyperparameters
Value
# of ensemble agents Epochs
2
100
Steps per epcoh
20000
Discount factor
0.99
Learning start steps
500
Minibatch size
256
Optimizer (all)
Adam
Optimizer (all) learning rate
0.001
Networks (all) activation
ReLU
Networks (all) n. hidden layers
Networks (all) hidden units
256
Initial Temperature
Entropy constraint
-1
Replay Buffer Size
x 106
Target network update period
T
0.005
Safety constraint
25
Risk level
0.5
Reset Interval (gradient steps)
4 x 10
B (action select coefficient)
50
Table 6: Hyperparameters in Safety Gym.
15

---

C Experimental Results
We provide the entire results of DQN, SR+DQN, and RDE+DQN on Atari 100k and Minigird in Table and Table respectively. We report the per-environment learning curves of Minigird in Fig. The learning curves of DMC are provided in Fig. 7 and Fig.
RR
2
Game
DQN
SR+DQN
RDE+DON
DQQ
SR+DQN
RDE+DON
DQN
SR+DQN
RDE+DQN
Alicn
423.2
512.4
414.4
596.6
506.4
502.4
414.0
639.6
610.0
Amidar
46.8
43.2
47.8
54.6
58.2
68.2
31.6
66.4
55.2
Assault
438.8
354.5
409.1
409.9
369.2
431.8
372.1
455.3
462.0
Asterix
418.0
426.0
418.0
394.0
474.0
562.0
306.0
620.0
678.0
Bank Hoist
16.8
23.2
27.2
21.2
14.4
26.8
33.6
Battle Zone
4040.0
3360.0
4040.0
2120.0
4520.0
7880.0
3840.0
7000.0
8240.0
Boxing
-2.6
0.8
-0.6
4.2
Breakout
16.1
6.7
16.1
19.6
15.0
21.2
23.8
20.8
19.5
Chapper Command
Crazy Climber
12472.0
16240.0
22556.0
22100.0
22216.0
25784.0
16028.0
25072.0
$6324.0
Demon
324.6
Freeway
15.1
7.2
4.0
10.9
6,4
16.8
14.8
7.9
21.2
Frusthite
Gopher
225.6
390.4
470.4
460.8
720.6
911.2
434.0
1169.6
1045.6
Hero
621.6
738.4
1068.8
2725.6
2819.4
754.0
3073.2
3564.2
Jameshond
68.0
70.0
126.0
178.0
50.0
138.0
140.0
92.0
78.0
Kangaroo
104.0
72.0
160.0
160.0
176.0
48.0
232.0
208.0
Kruil
1905,2
2262.4
5325.6
2637.5
2460.4
1854.0
2533.6
3144.0
3374.0
Kung Master
8264.0
5908.0
7256.0
6244.0
8216.0
7524.0
6008
7996.0
8284.0
Ms Pacmann
790.0
769.6
609.6
9U7.2
832.0
831.6
868.4
954.8
1223.2
Pong
-20.7
-20.6
-20.0
-19.1
-18.8
-17.1
-14,4
-16.0
-16.8
Private Eye
20.0
2.1
44.0
44.0
-69.1
64.0
0.0
40,0
83,5
Qhert
457.0
388.0
415.0
497.0
489.0
436.0
615.0
467.0
941.0
Road Runner
2288.0
576.0
1976.0
2288.0
2488.0
3180.0
1680.0
1924.0
3036.0
Seaquest
292.0
222.4
243.2
207.2
240.0
337.6
216.0
341.6
372.0
Up N Down
1396.8
1068.8
1734.4
1756.0
1769.2
2258.0
1662.4
1472.4
1503.6
IQM
1.000
0.852
0.987
1.203
1.063
1.377
1,073
1,351
1.422
Table Results Atari-100k
Gume
DQN
SR+DQN
RDE+DQN
DQN
SR+DQN
RDE+DQN
DQN
SR+DON
RDE+DQN
GoToDoar-8e 8x8
0.709
0.71
0.911
0.544
0.659
0.944
0.159
0.684
0.929
LavaCrossing
0.035
0.248
0.016
0.02
0.215
0.029
0
0.237
SimpleCrrosingSNNi
0.012
0.186
0
0.013
0.014
0.137
FourRooms
0.002
0.03
0.148
0
0.072
0.159
d
0.034
0.155
LavaGapS7
0.747
0.655
0.793
0.761
0.729
0.793
0.674
0.706
0.791
Table Results on MiniGrid
16

---

(ours)
(a) acrobot-swingup(]
(b) acrobot-swingup(2)
(c) acrobot-swingup(4)
1000
1000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(d) cheotah-rum()
(c) cheetah-run(2)
(f) cheetah-run(4)
1000
1000
1000
1.0
0.8
0.6
(g) finger-turn (arrr____)))))_
(h) Langer-turn_arrddee
(i) geaneertum_hhaad(
800
8.00
200
200
2:00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.6
0.8
1.0
(j) fish-swim(1 (1)
(k) fish-swim(2)
(1) fish-swim(4)
500
5.0
400
400
4.0
100
1.0
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.G
0.8
1.0
(m) hopper-hop(1)
(n) hopper-hop(2)
(0) hopper-hop(4)
Figure 7: Per-environment performance in DMC with varying replay ratio values. Note that the number parentheses indicates the replay ratio. The scale of the x-axis is 105, Performances are averaged over 5 seeds.
17

---

(ours)
150
150
150
1
50
50
50
humanoid-run(1)
(b) humanoid-run((2
humanoid-run(4)
1000
600
600
6.00
400
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
08
1.0
0.0
0.2
0.6
0.8
1.0
(d) quadruped-rund
(c) quadruped-rum(2
(f) quadruped-rume
600
600
6:0
400
400
4.00
200
200
11 PM
200
3.0
(g) swimmerrswiimmer15(
(h) swimmer-swimmerr (5)
swimmer-swinmer
1000
1000
1000
800
400
400
400
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.6
0.8
1.0
(j) walker-run(I)
(k) walker-run(2)
(l) walker-run(4)
Figure 8: Per-environment performance DMC with varying replay ratio values. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is 10 Performances are averaged over 5 seeds.
18

---

(ours)
0.23
0.20
0.15
0.15
0.10
0.10
0.10
0.05
0.05
0.05
0.00
0.00
0.00
0.0
0.2
0.0
0.2
0.4
0.6
0.9
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) FourRoom((.55
(b) FourRoom(1)
FourRoom(2)
1.00
1.00
1.00
My Weite Wwah
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
3.0
1.5
2.0
(d) Gotodoor(U.S)
(e) Gotodoor(1)
(f) Gotodoor(2)
0.4
0.4
0.4
M
W
0.3
WM
0.3
0.2
0.2
0.2
0.1
0.1
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
(g) (0)))
(h) Lavacrossing9(1)
(i) Lavacrossing9(2)
1.00
0.75
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.0
0.2
0.4
0.6
0.8
1.0
(j) (avagap7((.5))
(k) Lavagap7(1)
(1) Lavagap7(2)
0.25
0.25
0.20
0.20
0.15
0.10
0.10
0.05
0.05
0.05
0.00
0.00
0.0
1.0
1.5
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
2.0
(m) Simplecrossing(055)
(n) Simplccrossing()
(0) Simplccrossing(2)
Figure 9: Per-environment performance in minigird with varying replay ratio values. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is 10 Performances are averaged over 5 seeds.
19

---

D Reset depth for Continuous Environments
investigated the impact of reset depth in Minigird environment. In order to investigate the effect of reset depth in continuous tasks, we conduct the additional experiments of RDE with two different depths: reser-1, which only resets the last layer, and reset-all, which entails complete reset of all layers in the DMC environments. As shown in Fig. [0] it observed that reset-1 exhibits comparable performance to reset-all in the cheetah-run, finger-turr__e hopper-hop, fiimmer-swimmett reset-/ in acrobot-swingup fish-swim, humanoid-run, quadruped tasks. Fur- thermore, in the cases of humanoid-rum the performance of reset- deteriorates with increasing replay ratio, suggesting that shallower levels of resetting render it more susceptible to primacy bias.
In Section
walker-run tasks. However, resei-all performs better than
reset
reset-1 (RR
reset (RR 4) (RR 4)
reset
reset (RR 2)
400
1000
300
8.00
200
6:0
400
100
200
2.00
0.0
0.2
0.8
0.8
1.0
0.0
0.2
0.4
0.8
1.0
0.2
0.6
0.8
(a) acrobot-swingup
(b) chectah-run
(c) finger-turn hard
500
600
400
150
400
100
0
1.0
(d) fish-swim
(e) hopper-hop
(f) humanoid-run
1000
600
4.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.0
0.2
0.6
0.8
1.0
(g) guadruped-run
(h) swimmer-s 5wimmerr-
(i) walker-run
Figure 10: Per-environment performance in DMC with varying replay ratio values. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is 10S. Performances are averaged over 5 seeds.
20

---

E Further Experiments Regarding Reset Interval
The reset interval is an important hyperparameter reset-based algorithms. We conduct performance comparisons by varying the reset interval, and the corresponding results are in Table 9 RDE outperforms the vamilla reset method. To ensure fair comparison, we set the reset interval of one ensemble agent to match that of an SR agent. This is important because excessively frequent resetting can have negative impact on the learning process, causing reset operations to occur before DNNs have fully recovered their performance. To illustrate this, we include additional experiments using the reset interval reset/ /NN rT) SR (vanilla reset) in DMC and Minigrid environments. The corresponding results shown in Fig. IT (indicated by the green line). Notably, the highly frequent vanilla reset (with the same number reset operations as in RDE) performs worse, even more poorly than the base algorithm in the Minigrid environment.
It is seen that
RR
2
4
Treset
1 x 10.
x 10y
4 x 10
x 10.
2 x 10
4 x 10a
x 10
2
10a
x
10
SR+SAC
1.00
1.08
1.02
1.00
1.15
1.15
1.02
1.13
1.21
RDE+SAC
0.99
1.10
1.20
1.02
1.10
1.17
1.16
1.25
Table 9: Results on DMC with varying reset interval
1.00
1.2
Test Return
0.75
BR-
DDN
Test Return
0.75
SR+DON (HAF)
An MASi
0.50
1.0
0.25
0.25
0.9
0.00
0.0
0.3
1.0
1.5
0.0
0.2
Time step (x 10f)
Time step (x 10%)
Reploy Reñio
(a) DMC
(b) GotoDoor (0.5)
(c) GotoDoor (2)
Figure 11: The green line represents SR+X with an equal number reset operations as RDE+X. (a) IQM metric normalized by SAC with replay ratio of I DMC. (b) & (c): Test performances GotoDoor. High reset frequency (HRF) refers to the number reset operations being the same as in RDE.
21

---

Further Experiments Regrading # of Ensemble Agents (N)
We include additional experiments confirm the effectiveness of ensemble learning. Increasing the number of ensemble agents, denoted as N, can lead to greater diversity gain. Additionally, the presence of N non-reset agents can aid effectively mitigating performance collapses after resetting. Illustrated in Fig. [2] RDE with N 4 demonstrates improved sample efficiency in the Minigrid environment. also observed in the Minigrid environment that the proposed method more effectively prevents performance collapse in comparison to RDE with N 2.
1.00
1.00
Test Return
Test Return
0.75
0.75
0.50
0.2
0.50
Test
0.25
(N-4) RDE (1=21
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.4
0.8
Time step (X 10")
Time step (x 10%)
Time step (x 10")
(a) GotoDoor (0.5)
(b) LavaCrossing9 (0.5)
(c) Lavagap7 (0.5)
Figure 12: Ablation study in MiniGrid environments with varying the number of ensemble agents, N. The red line and brown line represent RDE with N 4 and RDE with respectively.
G Limitations
Computational costs increase linearly raise the number of ensemble agents the replay ratio. limitation of our approach is the escalated computational cost due to ensemble agents. Nonetheless, is important to highlight that, typically, the challenge in reinforcement learning lies more in sample efficiency, owing to the substantial costs tied to environment interaction, rather than computational intricacies. We convinced that method substantially enhances sample efficiency and safety, especially in environments with ample computational resources.
22


Reviews:
[{"review_id": "SCAPglAJbm", "reviewer": "Reviewer_6LEC", "summary": "This paper proposes a novel reset-based method that leverages deep ensemble learning to address the primacy bias issue in deep reinforcement learning. The authors construct N-ensemble agents and reset each ensemble agent sequentially to prevent performance collapses and improve sample efficiency. The proposed method is evaluated through experiments on various environments, including safe RL scenarios, and the results demonstrate its effectiveness and potential for real-world applications.", "questions": "1. Can you show more experimental results on safe RL benchmark?", "limitations": "The paper stated that RDE was conducted for tasks with safety requirements, but it only did one experiment on safe RL benchmark. The paper has no negative social impacts.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an important issue in deep reinforcement learning, namely primacy bias, and proposes a novel method to mitigate its effects. This is valuable as primacy bias can lead to overfitting and performance deterioration, which affects the applicability and efficiency of deep RL algorithms.\n2. The use of deep ensemble learning in the proposed method is innovative and practical. Deep ensemble learning has shown effectiveness in domains such as image classification and RL, and leveraging the diversity gain of ensemble agents can enhance performance and prevent performance collapses.\n3. The paper provides a comprehensive analysis of the proposed method, including its underlying operations and how it effectively prevents performance collapses. This analysis adds clarity and depth to the understanding of the proposed method.", "weaknesses": "1. The paper stated that RDE was conducted for tasks with safety requirements, but it only did one experiment on safe RL benchmark. Thus the paper lacks effective validations for RDE on violations. Simply stating that RDE does not cause performance collapse in general continuous control tasks cannot explain its role in safe RL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Regarding \"Results on safe RL benchmark\"**\n\n- As described in the common response, we have conducted two additional experiments on the safe RL benchmark to show the effectiveness of the proposed method. The corresponding results are presented in Figure 1 of the rebuttal PDF file. These results clearly indicate that RDE outperforms WCSAC in both test performance and cost. As demonstrated earlier in the main paper, RDE addresses a critical issue of the naive reset method in the safe RL domain—the rapid increase in safety costs during training. Summarizing the aforementioned additional results along with the result on safe RL domain in the main paper, it becomes evident that the proposed RDE not only ensures safety but also enhances sample efficiency.\n\n"}, {"review_id": "vfNSdIHT5C", "reviewer": "Reviewer_UDiR", "summary": "The work proposes an extension to the resetting strategy proposed by Nikishin et al. The extension intends to mitigate the catastrophic performance collapse often observed for the simple resetting strategy while keeping the properties that help avoid the primacy bias.\nTo this end, the work makes use of ensembling techniques, such that the policy/value network of an RL agent is not completely reset, but can fall back to different checkpoints.\nThe method is evaluated on a variety of environments and a further extension is presented that enables application in safety critical systems.", "questions": "How would PBT style methods compare to resetting strategies. Are they doing some form of resetting?", "limitations": "Limitations of the method have not been discussed. An obvious limitation is that ensembles likely will increase the computational overhead. For example, in line 131 it is said that the simple resetting strategy often requires high replay ration and therefor more resources. This should likely be worse for the presented method and the conducted experiments are not convincing enough to show that the novel method would require fewer resources.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The presented method cleverly combines ensembling with resetting to improve RL agents.\n* It seems general enough that it should be usable as a plug and play method for a broad variety of RL agents.\n* The work mostly easy to follow\n* It is demonstrated that the method is flexible and can incorporate auxiliary information, such as safety-critical information to provide a safer method than the vanilla resetting one", "weaknesses": "The presentation could be improved:\n* A lot of the discussed \"preliminaries\" seem irrelevant for the content of the paper. For example, the paragraph on \"Off-policy RL.\" seems not necessary and the content in \"Primacy Bias and Resetting Deep RL Agent.\" is largely a repetition of the introductory text.\n* Algorithm 1 is never discussed in the text and feels wholly redundant with Fig. 1. This half page might be used to show more experimental results.\n* Algorithm 1 is not consistent with the text. From the Algorithm it looks like every ensemble member is reset every $N\\times T_{reset}$ time-steps and not $T_{reset}/N$ as stated in line 149. \n* Lines 178 - 180 are concerned with expressing that the \"oldest Q-function\" is used to normalize in the selection mechanism. This can be expressed in much clearer terms \"oldest Q-function\" as is shown in line 183.\n* In line 195 it is claimed that RDE effectively prevents performance collapses, however I disagree with this wording. It can mitigate it to some extent, but the experiments clearly show that there is still performance collapses happening.\n* The description of Figure 2 is wholly confusing. The long sentence explaining what the y-axis is showing is expressed in a very convoluted way.\n* In figure 2c it should not be possible that the performance of the baseline \"Base\" is below 1 when using an RR of 1.\n* Design decisions are often not well enough explained.\n* Figure 1 does not explain what RR is\n* The paragraph heading in line 204 should be \"Baselines & RL Agents\" not \"Baselines & DNN Architectures\"\n* Line 246 claims that there is no significant drop on the humanoid-run example. The reward for the RDE method drops from 150 - 100. This represents a significant drop in my opinion since 1/3 of the performance is lost.\n* The choice for designing the selection mechanism is very unclear. The discussion about using the oldest Q-function in the selection mechanism seems more to point to only using the oldest Q-function. Indeed, this seems to be supported in the experiments and should have been at least an ablation. Additionally the initial paragraph of Section 3.2 seems to also point to only using the oldest Q-function\n* It is claimed that a $\\beta$ set to 50 \"nearly eliminated\" performance collapses (line 270). However, the performance collapse in the Figure is again the 1/3 loss in performance.\n* Section 5 feels like an added afterthought. I don't see why it warrants an own section. It repeats some of the discussion of safe RL from the preliminaries section. Instead the \"safe\" selection mechanism should have been discussed in Section 3 and then only the results should be a subsection of Section 4.\n* Where does the value for the reset frequency $4\\times 10^5$ in line 221 come from?\n\nThe experiments seem to have an unfair comparison and are likely not reproducible:\n* All details about how hyperparameters were determined seems to be missing\n* The value of $N$ is never stated. Only from the plots can it be assumed to be 2\n* The ablation does not take into account all confounding factors and should be redone\n* It is claimed that using a reset frequency for the individual members being equal to the single network case is fair. I fail to see how this is fair. This seems to just give benefit to the RDE method since it can benefit $N$ times as much from resetting.\n* Without stating how hyperparameters were set of the methods, it seems like an arbitrary comparison of the methods.\n* N is not really ablated and it's not clear how the ensemble increases computational overhead.\n\nThe description of an MDP is wrong. An MDP is an abstract representation of an environment. The MDP does not consist of an agent.\nThe bounds for the discount factor in line 74 should be $\\gamma \\in [0,1]$ not $\\gamma \\in [0,1)$. It is totally valid to have undiscounted cases.\n\nOverall, the work would need significant rewriting and an overhaul for the experiments for me to accept it. I am very doubtful that this can be done in the time-frame of a rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Regarding \"Presentation\"**\n\n\n- The subsection on 'Off-policy RL' is necessary in our work for two reasons. Firstly, the reset methods depend on an off-policy algorithm, as a recently reinitialized RL agent needs training using experiences generated by the previous RL agent. Secondly, this section allows us to introduce the base algorithms used in our experiments (DQN and SAC) and essential definitions for our work, such as the value function, which plays a key role in the adaptive integration mechanism (Equation 2).\n\n- In the 'Safe RL' subsection of Preliminaries, we introduce the definition of safe RL and provide an example of a safe RL algorithm (WCSAC). In contrast, Section 5 explores a specific challenge that arises when applying the reset method to safe RL domains: the rapid increase in safety costs. We then describe how we incorporate the proposed method into WCSAC to tackle this challenge and present the corresponding results. Thus, we believe both sections are necessary to establish the background and motivation of our work.\n\n- We agree that there is overlap between the content of \"Primacy Bias and Resetting Deep RL Agent\" and the Introduction. We intend to streamline and simplify the duplicated portions within \"Primacy Bias and Resetting Deep RL Agent\" for the final manuscript.\n\n- We plan to relocate Algorithm 1 to the Appendix and offer a comprehensive explanation of it in the final manuscript. Furthermore, we will adjust the reset frequency in Algorithm 1 to $T_{reset}/N$.\n\n- We apologize for the confusion regarding Fig. 2. The y-axis of Fig. 2 represents the IQM metric of test return. In Fig. 2 (a) and (b), we normalized the test return using the base algorithm with a replay ratio of 1, while in Fig. 2 (c), we utilized the unnormalized test return (which is why the \"base\" baseline performance is below 1). To ensure consistency, we will make the necessary revision to display the normalized value in the final manuscript.\n\n- The value of $4\\times 10^5$ is one example.\n\n\n**Regarding \"Prevention of performance collapse\"**\n\n- As discussed in Section 4.3, we can address the degree of performance collapses by adjusting $\\beta$. In the rebuttal PDF file, we have incorporated a result from RDE using a higher $\\beta$ on the Humanoid-run environment. The result, illustrated in Figure 4 of the rebuttal PDF file, shows that RDE with $\\beta=300$ avoids performance collapse, affirming the effectiveness of our method. In addition, RDE with $\\beta=50$ still performs better than the base algorithm even if performance collapses exist. \n\n\n**Regarding \"Experiments and Hyperparameters\"**\n\n- The hyperparameters have been tuned within the range of values used in prior work [2-1]. Furthermore, to ensure fair comparison, we established the hyperparameters to be consistent; for example, both SR and RDE employ the same number of reset operations per agent (see common response). Regarding the reset interval, we have included a performance comparison by adjusting it, as mentioned in the common  response for the reviewer's information. The range of hyperparameters will be detailed in the final manuscript.\n\n\n- We have mentioned the value of $N$ in the Appendix; however, we will relocate it to the main body. For Atari-100k and DMC, we used $N=2$, while for Minigrid, we used $N=4$. Notably, we conducted an ablation study on the number of ensemble agents, $N$, as outlined in \"Ensemble and Reset Effect\" in Section 4.3. During the rebuttal period, we also included the additional ablation study on $N$, and the corresponding results are shown in Figure 3 of the rebuttal PDF file. \n\n\n**Regarding \"Reset frequency\"**\n\n- We have discussed the concept of fair comparison related to reset frequency (interval) in the common response. To summarize, the reset operation of an individual agent in RDE should be the same with that of the SR agent, considering that the recovery times for DNNs of the same size tend to be similar. Notably, we noticed a decrease in performance when the reset frequency for an SR agent corresponds to the frequency at which ensemble agents are reset. Please see the common response and Figure 2 in the rebuttal PDF file (the green line represents SR with the same number of reset operations as ensemble agents).\n\n\n\n**Regarding \"PBT style method\"**\n\n- Population-Based Training (PBT) involves training networks in parallel with diverse parameters and hyper-parameters. It periodically evaluates performance, selects the best hyper-parameters, and distributes them to other learners for training. While both PBT and our method involve multiple networks, it's important to note that PBT is not directly related to reset methods. Reset methods, on the other hand, focus on reinitializing the parameters of a deep neural network to avoid overfitting to early experiences and facilitate convergence towards the global maximum (or minimum).\n\n\n**Regarding \"Using the oldest Q-function in adaptive composition\"**\n\n- The rationale behind utilizing the oldest Q-function is that the estimated Q-value function of a recently reset network can be unreliable due to limited time for recover its performance. We have conducted an ablation study regarding this, and the corresponding result is shown in Figure 5 of the rebuttal PDF file in the common response. The result indicates that utilizing the recently reinitialized (newest) Q-function for adaptive composition yields poorer performance due to inaccurate cumulative return estimation. The first, second, and third oldest Q-functions exhibit similar performance since they all recover their performance to approximate the cumulative return. From a conservative standpoint, we believe employing the oldest Q-function is the most suitable choice for our method.\n\n\n\n[2-1] E. Nikishin et al., \"The primacy bias in deep reinforcement learning,\" ICML 2022\n\n"}, {"review_id": "cf15Gc8b6Z", "reviewer": "Reviewer_fWBK", "summary": "This paper combines the resetting method proposed by (Nikishin et al., 2022) as a remedy to the primacy bias affecting deep RL algorithms with the use of ensembles of agents. The proposed RDE method, apart from generally improving performance, has the goal of minimizing the regret associated to a learning agent that uses periodic resets, mitigating the severeness of the performance drops it experiences right after a reset. To do this, without sacrificing too much on exploration and online data collection, the probability of executing an action in the environment is evaluated according to the oldest value function. Empirical results show benefits in using this approach, at both low and high replay ratio, in standard robotic locomotion and navigation tasks, as well as a safety domain.", "questions": "I ask the authors to provide answers to my concerns expressed above.", "limitations": "The authors do not discuss any computational consideration resulting from their use of ensembles of agents. I encourage the authors to add such a discussion to the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality**: despite the combination of resets and ensembles is not particularly original, the focus on developing a technique for leveraging their combination to avoid performance collapse while resetting is, to the best of my knowledge, new.\n\n**Quality**: the quality of the work is generally good. The experiments cover a reasonable number of domains and the ablations mostly answer natural questions.\n\n**Clarity**: the clarity of the writing is good. The paper would benefit from some small tuning to the presentation here and there, but the overall flow makes clear what the contribution is.\n\n**Significance**: harnessing the performance benefits originating from the mitigation of the primacy bias while at the same time not incurring a cost in terms of regret is a worthy research direction which could be interesting to many practitioners and researchers.", "weaknesses": "**Major Concerns**\n- If I understand it correctly from Algorithm 1, a different policy could be potentially selected at each step in the environment. This could conceptually create problems in terms of inconsistent behavior, since each policy will have to deal with the actions previously sampled from a potentially very different policy, and in terms of lack of \"deep exploration\", since this would harm temporally-consistent behaviors. A reader would benefit from a discussion or analysis of this aspect, to understand whether this is happening at all, or, if not, why that might be not happening in this kind of tasks of setting.\n- The idea of combining periodic reset and ensembles of agents in continuous control has been explored in \"Unleashing The Potential of Data Sharing in Ensemble Deep Reinforcement Learning\" (Lin et al, 2022). I find the idea of the paper of using this combination to avoid performance drops to still be valuable, but discussing the relationship with that paper could better contextualize the contribution.\n\n**Minor Concerns**\n- Y and X labels are missing from all Figures in the paper, making it hard to parse the plots at first sight. For most plots, it is either quite easy to infer the quantities of interest, or they are explicitly mentioned in the caption, but it can still be very misleading for many readers.\n- The paper would benefit from the extension of some of the ablations to more tasks. In particular, I find the performance of the ensemble-based approach without any resetting on top to be an important baseline to contextualize the results in Figure 3, and I think results concerning it would be a good addition to the Figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Regarding \"Inconsistent behavior\"**\n\n\n- It is entirely true that a different policy can be chosen at each time step. While this might result in inconsistent behavior, such inconsistency doesn't negatively affect exploitation since our method primarily relies on off-policy learning (the resetted policy is trained using experiences generated by the previous RL agent). Moreover, in terms of exploration, diverse policy behaviors can improve the process by generating new trajectories that have not been encountered before. In specific, we typically introduce Gaussian noise to actions in continuous action domains to encourage exploration, which can also potentially constrain the exploration of an RL agent. Through the utilization of policies from other ensemble agents, we expect that the adaptively composited agent can visit unexplored state-action spaces. We plan to include a discussion on this discussion in the final manuscript.\n\n\n\n\n**Regarding \"Ensemble agents without reset\"**\n\n- We have already compared our method to the ensemble-based approach without reset, and you can see the corresponding result in Figure 5 (purple line) of the main paper. This demonstrates that the combination of our ensemble learning and the reset method enhances performance.\n\n\n\n**Regarding \"Relationship with a prior work\"**\n\n\nAs the reviewer mentioned, [3-1] also combines ensemble learning and the reset method. However, [3-1] primarily focuses on exploiting diversity gain by sharing data among ensemble agents, while our approach concentrates on both enhancing diversity gain and preventing performance collapses. The specific differences are as follows:\n\n- [3-1] employs parallel learning, training $N$ ensemble agents across $N$ corresponding environments. In contrast, RDE adapts and combines ensemble agents into a single agent within a single environment.\n\n- [3-1] resets ensemble agents simultaneously, whereas RDE performs a sequential reset to prevent performance collapse.\n\n- Additionally, we provide results in various RL domains, encompassing both discrete and continuous action spaces, as well as safe RL benchmarks.\n\n\n**Regarding \"Figures\"**\n\n- We will make more concrete revisions to the figures based on the comments from the reviewer.\n\n\n\n\n[3-1] Lin et al., \"Unleashing The Potential of Data Sharing in Ensemble Deep Reinforcement Learning,\" arxiv, 2022\n\n"}, {"review_id": "aHgQ7EB8ST", "reviewer": "Reviewer_EaBR", "summary": "Overfitting in deep Q-learning agents is a recent topic of interest in the RL community, and several methods have been proposed to mitigate this problem, including data augmentation (DrQ), random ensembles (RedQ, DroQ), and resets (Nikishin et al.). This paper builds upon prior work on periodically resetting weights and addresses one of its core limitations: by resetting weights, an agent will perform poorly immediately after resetting, although it eventually recovers and often exceeds its performance before resetting. This paper proposes to learn an *ensemble* of agents and periodically reset only one of the agents at a time (in sequence). Experiments are conducted on tasks from DMControl, Minigrid, and Atari100k, and indicate that resetting with ensembles is effective at mitigating deterioration of performance immediately after a reset.", "questions": "I would like the authors to address my comments listed in \"weaknesses\". If the authors are not able to conduct some or all of the experiments required to answer my questions during the rebuttal, I'd like to see a discussion of what the authors would expect the results to be based on their experience.", "limitations": "There is sufficient discussion of limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well written, considers an interesting problem, and experiments appear sound. The description of the proposed method is easy to follow, and especially Figure 4 is useful for understanding how the ensemble resetting works in practice.", "weaknesses": "- It would be useful to include more discussion on related works that seek to understand and mitigate overfitting in deep Q-learning besides resetting. There is a lot of literature in this area and I imagine that the authors are familiar with the literature, so I will refrain from mentioning any specific references (besides the methods mentioned in my summary) to remain impartial.\n\n- I would like to see more ablations to get a deeper understanding of the trade-offs in ensemble resetting vs. prior works. How does sample-efficiency and performance drop change when the number of ensemble agents $N$ varies? How often should one reset agents? Is the rate of resets and number of agents dependent on the replay ratio? What is the computational cost of using additional agents vs. increasing the replay ratio? Would the proposed method benefit from using the ensemble in other ways as well, e.g. by using the RedQ trick for computing TD-targets? Addressing some or all of these questions would likely increase the impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Regarding \"Related works\"**\n\n- We will include related works regarding overfitting problem in RL in the final manuscript, as the reviewer recommended.\n\n\n**Regarding \"Sample efficiency and performance collapse with respect to the number of ensemble agents\"**\n\n- We have conducted additional experiments concerning the number of ensemble agents, as detailed in the common response. As depicted in Figure 3 of the rebuttal PDF file, the results demonstrate that increasing the number of ensemble agents, denoted as $N$, improves both final performance and sample efficiency. This improvement is attributed to each agent being trained with experiences from a diverse set of ensemble agents, contributing to increased diversity. Moreover, the result shows that a higher $N$ leads to more efficient mitigation of performance collapse.\n\n\n**Regarding \"Reset interval\"**\n\n- We have included additional experiments concerning the reset interval, as outlined in the common response. It is seen that highly frequent resets can negatively impact learning. Consequently, determining an appropriate reset interval, a critical hyperparameter, is imperative. Importantly, our proposed method consistently outperforms the vanilla reset method across all considered reset intervals.\n\n\n**Regarding \"Computational costs of increasing the number of ensemble agents and the replay ratio\"**\n\n\n- Computational costs increase linearly as we raise the number of ensemble agents or the replay ratio. A limitation of our approach is the escalated computational cost due to ensemble agents. Nonetheless, it is important to highlight that, typically, the challenge in reinforcement learning lies more in sample efficiency, owing to the substantial costs tied to environment interaction, rather than computational intricacies. We are convinced that our method substantially enhances sample efficiency and safety, especially in environments with ample computational resources. We will include this discussion in the final manuscript. \n\n\n**Regarding \"relationship between (the rate of resets, the number of agents) and the replay ratio\"**\n\n- As discussed in our work, the reset method allows us to increase the replay ratio, resulting in improved performance. Therefore, a higher replay ratio allows for a higher rate of reset (more frequent resets). The number of ensemble agents is independent of the replay ratio, but we need to choose the appropriate number of ensemble agents and replay ratio within our computational budget, as both increase computational costs.\n\n\n\n\n**Regarding \"Benefit from other ensemble learning methods\"**\n\n\n- Ensemble learning offers a range of advantages, including improved exploration [4-1] and reduced estimation variance [4-2], depending on its purpose and approach. In our work, we leverage ensemble learning to prevent performance collapse in reset methods and attain diversity gains. Additionally, we concur that ensemble learning could yield further benefits, like diminished variance during Q-function estimation, as pointed out by the reviewer. While we recognize the potential impact of the recently reinitialized agent on ensemble learning benefits, this aspect could be explored in future work, prompting us to consider slight modifications.\n\n[4-1] Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning\n\n[4-2] Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning\n\n"}]

Author rebuttal:
We thank all reviewers for their valuable comments. 

In this paper, we propose a novel method that incorporates ensemble learning into the resetting method to harness diversity gain and mitigate performance collapse. We provide various experiments on both standard and safety RL benchmarks, as well as ablation studies that demonstrate how the proposed method mitigates performance collapses. We believe the proposed method can contribute to practical RL algorithms by addressing sample efficiency and safety, which are critical challenges in RL.

In response to the reviewers' comments, we conducted additional experiments and included the corresponding results in the rebuttal PDF file. Referring to the PDF file, we present our common response to the major concerns raised by the reviewers below:


**1. Two Additional Experiments on Safe RL Benchmark**


In addition to the results on Safexp-PointGoal1-v0, we have included results on the SafexpPointButton1-v0 and SafexpCarGoal1-v0 environments. The results indicate that RDE consistently outperforms WCSAC in terms of both test performance and cost across all considered environments, as shown in Figure 1 of the rebuttal PDF file. As RDE improves final performance while minimizing safety cost regret, we believe that RDE can significantly contribute to the safe RL domain. 


**2. Further Experiments Regarding Reset Interval**


- The reset interval is an important hyperparameter in reset-based algorithms. We have conducted performance comparisons by varying the reset interval, and the corresponding results are as follows:


| RR |\| | 1 |  | \| | 2 | | \| | 4 | |
| :------------ | :------------ | :-------------: | :-------------: | :------------ | :-------------: | :------------: | :------------ | :------------: | :-------------: |
| $ T_{reset} $ | \| | $ 2 \times 10^5 $ | $ 4 \times 10^5$ | \| | $ 2 \times 10^5 $ | $ 4 \times 10^5 $ | \| | $ 2 \times 10^5 $ | $ 4 \times 10^5 $ |
| **SR+SAC** | \| | 1.08 | 1.02 | \| | 1.15 | 1.15 | \| | 1.13 | 1.21 |
| **RDE+SAC** | \| | 1.10 | 1.20 | \| | 1.10 | 1.17 | \| | 1.16 | 1.25 |

It is seen that RDE outperforms the vanilla reset method.

- To ensure  fair comparison, we set the reset interval of one ensemble agent to match that of an SR agent. This is important because excessively frequent resetting can have a negative impact on the learning process, causing reset operations to occur before DNNs have fully recovered their performance. To illustrate this, we've included additional experiments using the reset interval of $T_{reset}^{rr=1}/(N\times rr)$ for SR (vanilla reset) in DMC and Minigrid environments. The corresponding results are shown in Figure 2 of the rebuttal PDF file (indicated by the green line). Notably, the highly frequent vanilla reset (with the same number of reset operations as in RDE) performs worse, even more poorly than the base algorithm in the Minigrid environment.



**3. Further Experiments Regarding # of Ensemble Agents (N)**

We have included additional experiments to confirm the effectiveness of ensemble learning. Increasing the number of ensemble agents, denoted as $N$, can lead to greater diversity gain. Additionally, the presence of $N-1$ non-reset agents can aid in effectively mitigating performance collapses after resetting. Illustrated in Figure 3 of the rebuttal PDF file, RDE with $N=4$ demonstrates improved sample efficiency in both Atari-100k and the Minigrid environment. It is also observed in the Minigrid environment that the proposed method more effectively prevents performance collapse in comparison to RDE with $N=2$.


**4. Further Experiments Regarding Performance Collapse**

We have presented a result from RDE with a higher $\beta$ on the humanoid-run environment to determine whether RDE can completely prevent performance collapse. The corresponding result is depicted in Figure 4 of the rebuttal PDF file. It is observed that RDE with $\beta=300$ does not experience any performance collapse.



**5. Further Experiments Regarding Adaptive Composition of Ensemble Agents**


We outlined the reasoning for employing the oldest Q-function in the main paper, considering that the recently reinitialized Q-function may not provide accurate estimations. To demonstrate this, we conducted an ablation study using alternative Q-functions, such as the 2nd oldest, 3rd oldest, and newest Q-functions. Figure 5 of the rebuttal PDF file illustrates that utilizing the newest Q-function leads to notably worse performance. From a cautious perspective, using the oldest Q-function for the adaptive composition is the most appropriate decision for our approach.


**6. Presentation**



- We will enhance the presentation of our paper using the insightful comments provided by the reviewers. For instance, in response to Reviewer UDiR's feedback, we plan to relocate Algorithm 1 to the Appendix and provide a comprehensive explanation of its details to address the concern that Algorithm 1's content may appear redundant with the information presented in Fig. 1. Furthermore, we intend to refine the figures to offer a clearer representation.


- The computational overhead almost linearly increases as $N$ increases, which is a limitation of our work, and we will include it in the final manuscript.





Meta-review: This paper proposes a solution for primary bias in deep RL which does not have the adverse safety issues of the plain resetting solution. Most of the reviewers agree that this is an interesting work.

The only reviewer who rejected the paper asked for more safe RL experiments and the authors added 2 more experiments during the rebuttal. Another reviewer had concerns about the writing. However, given the very positive review by the reviewer fWBK, I recommend an acceptance and encourage the authors to improve their writing in the final version of the paper.
Decision: Accept (poster)