Title: Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agents
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep reinforcement learning

primacy bais

reset

deep ensemble learning
Abstract: Deep reinforcement learning (RL) has achieved remarkable success in solving complex tasks through its integration with deep neural networks (DNNs) as function approximators. However, the reliance on DNNs has introduced a new challenge called primacy bias, whereby these function approximators tend to prioritize early experiences, leading to overfitting. To alleviate this bias, a reset method has been proposed, which involves periodic resets of a portion or the entirety of a deep RL agent while preserving the replay buffer. However, the use of this method can result in performance collapses after executing the reset, raising concerns from the perspective of safe RL and regret minimization. In this paper, we propose a novel reset-based method that leverages deep ensemble learning to address the limitations of the vanilla reset method and enhance sample efficiency. The effectiveness of the proposed method is validated through various experiments including those in the domain of safe RL. Numerical results demonstrate its potential for real-world applications requiring high sample efficiency and safety considerations.

Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agents

Abstract

Deep reinforcement learning (RL) has achieved remarkable success in solving complex tasks through its integration with deep neural networks (DNNs) as function approximators. However, the reliance on DNNs has introduced a new challenge called primacy bias, whereby these function approximators tend to prioritize early experiences, leading to overfitting. To mitigate this primacy bias, a reset method has been proposed, which performs periodic resets of a portion or the entirety of a deep RL agent while preserving the replay buffer. However, the use of the reset method can result in performance collapses after executing the reset, which can be detrimental from the perspective of safe RL and regret minimization. In this paper, we propose a new reset-based method that leverages deep ensemble learning to address the limitations of the vanilla reset method and enhance sample efficiency. The proposed method is evaluated through various experiments including those in the domain of safe RL. Numerical results show its effectiveness in high sample efficiency and safety considerations.

1 Introduction

With the remarkable success of deep learning in diverse fields such as image classification [10], the integration of reinforcement learning (RL) with deep learning, called deep RL, has been considered a promising approach to solving complex control tasks. In particular, using deep neural networks (DNNs) as function approximators enables the representation of state-action value function over a huge state-action space without requiring large storage as in tabular methods, which is often infeasible in complex tasks [12]. Despite its effectiveness, however, DNN-based function approximators in deep RL can lead to an overfitting problem called primacy bias , which was first investigated in Nikishin et al. [13]. Primacy bias means that DNN-based function approximators in RL may overfit to early experiences, impeding their ability as function approximators for subsequent later experiences. This phenomenon basically results from the use of an replay buffer in deep RL serving as a dynamic dataset to which experience samples are added during the training process so that experiences collected early in the training process are sampled more in training than those collected later [6]. Due to this primacy bias, it has been shown that performance tends to deteriorate as the replay ratio, defined as the number of updates per environment interaction, increases [6, 13].

To mitigate the primacy bias, Nikishin et al. [13] recently proposed a simple but effective method based on resetting. The reset method periodically resets either a portion or the entirety of a deep RL agent while preserving the replay buffer. This method enhances both the steady-state performance and sample efficiency by allowing the deep RL agent to increase the replay ratio, whereas the performance of the vanilla deep RL agent deteriorates with the increasing replay ratio. Despite its potential as a means to solve primacy bias, the reset method has a problem. That is, it causes periodic

collapses in performance immediately after the resets. The occurrence of periodic collapses is an undesirable side effect, which is an issue from the perspective of safe RL and regret minimization [13]. That is, performance collapse can lead to the violation of safety constraints, thereby limiting the applicability of the reset method to real-world RL scenarios such as autonomous vehicles. In addition, the reset method can be ineffective when the replay ratio is low, or the environment requires extensive exploration with balance with exploitation, for example, MiniGrid, as we will see in Sec. 4.

In this paper, we propose a new reset-based method that is safeavoiding performance collapses -and achieves enhanced sample efficiency , by leveraging deep ensemble learning. In the proposed method, we first construct N -ensemble agents that adaptively integrate into a single agent based on the action value function, and then reset each agent in the ensemble sequentially. The adaptive integration of the ensemble agents prevents performance collapses after the reset by minimizing the probability of selecting the action from the recently-reset agent, also improves the sample efficiency by leveraging the diversity gain of the ensemble agents. We evaluate the proposed method combined with several deep RL algorithms such as deep Q-network (DQN) [12] and soft-actor-critic (SAC) [7], on various environments including Minigrid [5], Atari-100k [4], and DeepMind Control Suite (DMC) [17]. We show that the proposed method significantly improves the performance of the baseline algorithm in the considered environments. In addition, we apply the proposed method to a safe RL algorithm called Worst-Case SAC (WCSAC). The result demonstrates that the proposed method improves return performance while simultaneously ensuring the safety constraint. The main contributions of this work are summarized as follows:

· To the best of our knowledge, this is the first work that addresses the issue of primacy bias while simultaneously avoiding performance collapses, which can be detrimental in safety-critical real-world scenarios.

· We present a novel method that adaptively combines N -ensemble agents into a single agent, incorporating sequential resets for each ensemble agent, to effectively harness the diversity gain of the ensemble agents and prevent performance collapses.

· We empirically demonstrate that the proposed method yields superior performance compared with the base algorithm and the vanilla reset method on various environments. We also provide a comprehensive analysis of the underlying operations, including how the proposed method effectively prevents performance collapses.

· We provide a further experiment in the domain of safe RL, demonstrating that the proposed method surpasses the baseline approaches in terms of reward performance and effectively reduces the occurrence of safety constraint violations of the reset method during the training process.

2 Preliminaries and Related Works

We consider a Markov decision process (MDP). At each time step t , the agent selects an action a t based on the current state s t according to its policy π ( a t | s t ) . The environment makes a transition to a next state s t +1 and yields a reward r t to the agent according to the transition probability p ( s t +1 | s t , a t ) and the reward function r ( s t , a t ) , respectively. Through this iterative process, the policy π is optimized to maximize the discounted return R t = ∑ ∞ τ = t γ τ r τ , where γ ∈ [0 , 1) is the discounted factor.

Deep Ensemble Learning Deep ensemble learning has emerged as a promising approach, showing effectiveness in domains such as image classification and RL [3, 14]. It involves aggregating an ensemble of multiple DNNs to exploit their diversity and improve performance. RL with deep ensemble learning combines policies from multiple agents with the same architecture but with different initial parameters, leveraging their diversity to enhance robustness and efficiency. For example, Anschel et al. [3] proposed training and averaging multiple Q-networks for stabilization and improved performance. Lee et al. [11] employed an ensemble-based weighted Bellman backup technique that re-weights the target Q-value based on an uncertainty estimate from the Q-ensemble.

Off-Policy RL Off-policy RL algorithms aim to optimize a target policy by using experiences generated by a behavior policy to increase sample efficiency compared with on-policy RL algorithms [13]. Off-policy learning typically uses a replay buffer to store experiences and use them to train the policy. One representative off-policy RL algorithm is DQN [12], which employs a DNN as a function approximator for the Q-function, defined as Q π DQN ( s t , a t ) := E { a t }∼ π [∑ ∞ l = t γ l -t r l | s t , a t ] . The

DNN parameters of the Q-function are trained to minimize the temporal difference (TD) error. Another example is SAC [7], which aims to maximize the cumulative sum of reward and entropy to improve exploration. SAC is built on the actor-critic architecture, consisting of a policy π ( ·| s t ) and soft-Q function, defined as Q π SAC ( s t , a t ) := r t + E τ t +1 ∼ π [∑ ∞ l = t +1 γ l -t ( r l + α H ( π ( ·| s l ))) | s t , a t ] . The policy is updated through soft policy iteration [7].

Safe RL Ensuring safety in RL is a crucial issue when applying RL to real-world applications such as autonomous vehicles and robotics. To address this issue, safe RL has emerged as an area of research that aims to learn policies satisfying safety constraints under a Constrained Markov decision Process (CMDP), which incorporates a cost function c : S × A → R to the conventional MDP framework. Recent safe RL research has focused on optimizing the return-based objective function while imposing constraints on the safety-related cost function [1, 9, 18]. For example, Yang et al. [18] introduced a SAC-based safety-constrained RL algorithm named WCSAC that aims to maintain the Conditional Value-at-Risk (CVaR) below a certain threshold. CVaR quantifies the level of risk aversion with respect to safety and CVaR for risk level α is defined as

where ϕ and Φ denote the PDF and the CDF of the standard normal distribution, respectively, and Q c π ( s, a ) and Var c π ( s, a ) respectively denote the conditional mean and the conditional standard deviation of the discounted sum of costs C π ( s, a ) = ∑ ∞ t =0 γ t c ( s t , a t ) given s 0 = s and a 0 = a , which are called safety critic. WCSAC uses DNNs to parameterize the safety critic as well as the reward critic. Based on both the safety critic and the reward critic, the actor is trained to maximize the reward while not violating the risk level α risk .

Primacy Bias and Resetting Deep RL Agent. Enhancing sample efficiency in terms of environment interactions is a fundamental challenge in RL. One simple approach to improve sample efficiency is to increase the replay ratio , which is defined as the number of policy or value parameter updates per environment time step. In deep RL, however, the reliance on DNNs for function approximators can degrade performance as the replay ratio increases. This is because the deep RL agent is susceptible to overfitting on early experiences, which deteriorates the function approximation ability of DNNs for subsequent experiences. This phenomenon is referred to as primacy bias, first investigated in the context of RL by Nikishin et al. [13]. To overcome the primacy bias, Nikishin et al. [13] proposed a simple method based on increasing the replay ratio and periodically resetting either a portion or the entirety of the RL agent while preserving the replay buffer. It was shown that the deep RL algorithms employing both periodic resets and high replay ratios achieved impressive performance across several environments. In this line of research, D'Oro et al. [6] recently proposed fixing the number of updates for resetting, rather than the number of environment time steps, to maximally increase the replay ratio. This approach leads to more frequent parameter resets as the replay ratio increases, but results in improved sample efficiency in terms of interaction with the environment.

Although the existing reset methods mentioned above were shown to be effective in many tasks, certain limitations of the current reset methods exist. First, performance collapses are unavoidable immediately after executing the reset, which is not desired in the context of safe RL and regret minimization [13]. Second, the current reset method is ineffective for some off-policy RL algorithms such as DQN as well as on some environments requiring extensive exploration such as MiniGrid. Lastly, a high replay ratio is necessary to achieve the desired performance enhancement, which may not be supported in computing environments with limited resources.

3 Methodology

To overcome the aforementioned limitations and increase sample efficiency further, we propose a simple R eset-based algorithm by leveraging D eep E nsemble learning (RDE). The proposed method involves a sequential resetting process of ensemble agents that are adaptively integrated into a single agent. The algorithm is comprised of three main components: (1) the construction of ensemble agents, (2) the sequential resetting mechanism, and (3) the adaptive integration of the ensemble agents into a single agent. With RDE, we aim to achieve superior performance in various environments while simultaneously addressing safety concerns. The overall operation of RDE is shown in Fig. 1.

Environment

Composited

Agent

N-Ensemble Agents

Adaptive Composition

if (timestep) % (reset frequency) == 0;

then reset

Select a

single agent k

Reset

Replay

Buffer

Train RR times

Store transition

3.1 Sequential Resetting Ensemble Agents

Wefirst construct N ensemble agents, each with an identical neural network architecture but initialized with distinct parameters. In this paper, we denote the parameters of the k -th agent by θ k , where k ∈ { 1 , 2 , · · · , N } . For instance, in the case of SAC, θ k includes all parameters for both the actor and critic of the k -th agent. Note that these ensemble agents are combined into a single agent that interacts with the environment. The integration of the ensemble into a single agent will be presented in Section 3.2.

Next, we perform the sequential and periodic resetting of the ensemble agents, while preserving the replay buffer. Specifically, every T reset time-step, we reset each parameter θ 1 , θ 2 , · · · , θ N in sequential order with time-step gap T reset /N , returning back to θ 1 after we reset θ N . Consequently, the existence of N -1 non-reset agents at the time of each reset operation serves to mitigate performance collapses, whereas the vanilla reset method with a single agent inevitably encounters performance collapse after each reset. As the reset operation is applied to all ensemble agents, our method can still tackle the issue of primacy bias effectively.

3.2 Adaptive Ensemble Agents Composition

Despite the potential effectiveness of utilizing an ensemble of N agents to mitigate performance collapses, the use of an ensemble alone is not sufficient. Suppose that we adopt a naive approach that selects one agent uniformly from the ensemble of the N agents and follows the action that the agent generates. Then, performance collapses may still occur since the most recently initialized agent can be selected to generate untrained random actions or biased actions. Therefore, we need a method to judiciously integrate the ensemble into a single agent, considering that a lower probability should be assigned to the selection of the recently reset agent.

To achieve this, we propose the adaptive integration of the N -ensemble agents into a single agent that interacts with the environment, inspired by Zhang et al. [19]. The proposed integration method first generates a set of actions ˆ a = ( a 1 , a 2 , · · · , a N ) given a state s from the policies ( π θ 1 , π θ 2 , · · · , π θ N ) of the ensemble agents. Subsequently, we select a single action from the set of N actions with a probability distribution that is based on the action-value function. We assign a higher selection probability to the action with a higher action value, thereby reducing the chance that the most recently reset policy will be selected. Thus, the most recently reset policy is seldom selected immediately

after the reset operation, and the probability of selecting this element policy gradually increases as it is trained. The designed selection mechanism effectively mitigates performance collapses, as demonstrated in Sec. 4.3.

Let us explain the adaptive integration mechanism in detail. We design the probability distribution p select of selecting an action from the set of N actions as a categorical distribution whose probability values are determined as

where p k denotes the probability of selecting the action from Agent k and α is the temperature parameter. To deal with the scale of the Q-function, we set the temperature parameter as α = max( ˆ Q ( s, a 1 ) , ˆ Q ( s, a 2 ) , · · · , ˆ Q ( s, a N )) /β and adjust the coefficient β . Here, we choose ˆ Q to be the estimated action-value function of the agent that underwent a reset operation at the earliest point with respect to the current time step among the ensemble agents. For example, after the k -th agent is reset, the earliest reset agent is the ( k +1) -th agent (or the first agent if k = N ). Then, ˆ Q = Q θ k +1 (or Q θ 1 if k = N ). The underlying rationale for adopting this approach is that the estimated Q-value function of a recently reset network is inaccurate due to lack of training time. By leveraging the oldest Q-function, which provides a more precise estimation of the true action value function, we can effectively decrease the probability of selecting action a k that has a low return. We initialize p select before the first reset operation as p i = 1 /N for all i .

In addition to its role in preventing performance collapse, our use of deep ensemble learning provides the potential to enhance performance through the diversity gained from ensemble agents. With ensemble agents initialized differently, we can effectively avoid getting trapped in local optima. Moreover, the adaptive integration allows the RL agent to balance the exploration-exploitation tradeoff, in contrast to the vanilla reset method, which disrupts this balance by restarting from scratch and consequently losing its exploitation ability after a reset operation. The proposed method leverages the presence of N -1 non-reset agents, ensuring that the exploitation ability is preserved even after a reset. Thus, the proposed reset method is a more balanced approach in terms of exploitation-exploration trade-off, and still preserves some level of exploitation while exploring the environment.

In summary, the advantages of RDE are threefold: (1) it effectively prevents performance collapses, (2) it balances the exploration-exploitation trade-off through the adaptive ensemble integration, and (3) it improves sample efficiency by addressing the primacy bias and leveraging the benefits of deep ensemble learning. The pseudo-code of the overall algorithm is provided in Appendix A.

4 Experiments

4.1 Experimental Setup

Environments We consider both continuous and discrete tasks including DeepMind Control Suite (DMC) [17], Minigrid[5], and Atari-100k[4] environments. Here, we consider 9 tasks in DMC, 26 Atari games, and 5 tasks in Minigrid. The details are provided in Appendix B.

Baselines We built the vanilla reset method [13] and the proposed method, both of which were implemented on top of the base algorithms. We used SAC [8] and DQN [12] as the base algorithms in DMC and Minigrid/Atari-100K environments, respectively. For each environment, we compared three algorithms: the base algorithm (denoted as X), the vanilla reset method (denoted as SR+X), and the proposed method (denoted as RDE+X).

DNN Architectures SAC employed an actor network and two critic networks, each consisting of three multi-layer perceptions (MLPs). For Atari-100k, DQN used a Convolutional Neural Network (CNN) for the first 3 layers and followed by 2 MLPs. For Minigrid, DQN employed 5 layers MLPs.

Reset Depth Both the vanilla reset and our proposed method involve resetting parts or the entirety of DNNs. The degree of reset, referred to as the reset depth, is a hyperparameter that varies depending on the specific environment. We follow [13] in resetting the entire layers of DNNs in the SAC for the DMCenvironments. For the Minigrid environment, we observed that a higher reset depth is more advantageous for tasks requiring extensive exploration, which we will see in Sec. 4.3. The used reset depth in Minigrid and Atari-100k are provided in Appendix B.

Base

SR + X

RDE + X  (ours)

IQM

IQM

IQM

(a) DMC

(b) Atari-100k

(c) MiniGrid

1.4

1.2

0.4

1.1

1.2

1.0

0.3

1.0

0.9

1

2

1

2

0.5

1

2

Reset Frequency The reset frequency T reset is a hyperparameter that is fixed with respect to the number of updates, as proposed in D'Oro et al. [6]. Accordingly, for each environment, the reset frequency in terms of timesteps is a linearly decreasing function of the replay ratio. For example, let us assume that the reset frequency is 4 × 10 5 timesteps when the replay ratio is 1 . Then, a replay ratio of 2 and 4 correspond to reset frequencies of 2 × 10 5 and 10 5 timesteps, respectively. Note that the replay ratio can be smaller than 1 . For instance, a replay ratio of 0 . 5 implies training DNNs every 2 timesteps. To ensure a fair evaluation, the reset frequency of each ensemble agent in RDE is equivalent to the reset frequency in the vanilla reset method. In summary, when the reset frequency with the replay ratio 1 in the vanilla reset is T v,rr =1 reset , and the reset frequency with the replay ratio of rr with N -ensemble agents is T v,rr =1 reset / ( N × rr ) . The reset frequency values for each environment is provided in Appendix B.

4.2 Performance Comparison

We first evaluated the performance of the proposed method in comparison with the base algorithm and the vanilla reset method by varying the replay ratio. The results on DMC, Atari-100k, and MiniGrid are shown in Fig. 2, using the interquartile mean (IQM) [2] performance metric. Note that the primacy bias is observed in the base algorithm, where its performance deteriorates as the replay ratio increases, as reported by Nikishin et al. [13] and D'Oro et al. [6]. While the vanilla reset method demonstrates better performance than the base algorithm, particularly with high replay ratios, our proposed method outperforms both baselines even with low replay ratios, as shown in Fig. 2.

The performance results of the proposed method and the baselines on the considered environments are shown in Fig. 3, where the considered tasks are hopper-hop and humanoid-run in DMC and Fourroom and GotoDoor in MiniGrid. It is seen that the proposed method not only yields superior performance to the baselines but also prevents performance collapses. In the Fourroom environment, the proposed method solves the game whereas both the base algorithm and the vanilla reset method fail to learn. The inherent loss of exploitation ability following a reset operation in the vanilla reset method prevents effective learning of this hard task. In other environments, as we expected, the vanilla reset method suffers from performance collapses. Especially in humanoid-run , there is a clear difference between our method and the vanilla reset method: immediately after a reset operation, our method sustains performance without a significant drop, whereas the vanilla reset method collapses to the initial performance.

More results on Atari-100k are provided in Appendix C.

4.3 Analysis

Performance Collapse The results in Sec. 4.2 demonstrate the effectiveness of the proposed method in eliminating performance collapses. As aforementioned, the probability of selecting an action from the ensemble of N actions is determined based on Eq. (2), which assigns a lower selection

Base

SR + X

RDE + X  (ours)

Test Return

Time step (× 10 6

)

Test Return

Time step (× 10 6

)

Test Return

Time step (× 10 6

)

Test Return

Time step (× 10 6

)

(a) hopper-hop (1)

(b) hopper-hop (4)

(c) humanoid-run (1)

(d) humanoid-run (4)

Test Return

Time step (× 10 6

)

Test Return

Time step (× 10 6

)

Test Return

Time step (× 10 6

)

Test Return

Time step (× 10 6

)

(e) FourRoom (0.5)

(f) FourRoom (2)

(g) GotoDoor (0.5)

(h) GotoDoor (2)

500

500

200

200

400

400

150

150

300

300

100

100

200

200

50

50

100

100

0

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.25

0.25

1.00

1.00

0.20

0.20

0.15

0.15

0.75

m

0.75

0.50

0.50

0.10

0.10

0.05

0.05

0.25

0.25

0.00

0.00

0.00

0.00

W

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

probability to an action of a recently reset agent. This effectively prevents performance collapses, and the degree of this effect can be controlled by adjusting the parameter β . To investigate the impact of the parameter β , we conducted experiments on the humanoid-run task using RDE+SAC with

N = 2 . We varied β across {-10 , 0 , 50 , 300 } and measured the corresponding empirical probability p 1 , which denote the probability of selecting the action generated by θ 1 . In Fig. 4 (b), we observe that the empirical p 1 experiences a sudden drop immediately after resetting θ 1 . On the other hand, when θ 2 is reset, p 1 jumps up, indicating the sudden drop of p 2 . Note that the occurrence of performance collapses was nearly eliminated when β was set to 300 . This demonstrates the effectiveness of our adaptive integration in mitigating performance collapses

𝜷

= - 10

𝜷

= 0

𝜷

= 50

𝜷

= 300

Test Return

Time step (× 10 6

)

Reset

𝜃

1

Reset

𝜃

2

(a) Performance

(b) Empirical

p

1

1.0

150

0.8

100

Empirical

0.6

0.4

50

0.2

0

0.0

0.00

0.25

0.50

0.75

1.00

10.2510.5010.75

1.00

Time step (x 106)

by appropriately reducing the probability of selecting actions from recently-initialized agents. Furthermore, increasing the value of β quickly adapt the values of p select , leading to effective reduction of performance collapses.

Reset Depth The choice of reset depth is a crucial hyperparameter that significantly impacts the final performance of reset-based methods. We observed that a higher reset depth can provide an advantage in more challenging tasks that require hard exploration, whereas it may present a slight disadvantage in relatively easy tasks. We provide the performance of RDE+DQN with two different reset depths: reset-part , which resets the last two layers, and reset-all , which completely reset all layers. These evaluations were conducted on three MiniGrid environments, comprising two challenging environments, FourRoom and SimpleCrossing9 , and one comparatively easier environment, GotoDoor . As shown in Fig. 5, reset-part exhibits inferior performance and instability compared with reset-all in two relatively challenging tasks. However, in two relatively easy tasks, the reset-pt performs marginally better than reset-all .

RDE (reset all)                RDE (reset part)                RDE (w/o reset)              RDE (N = 2)

Test Return

Time step (× 10 6

)

Test Return

Time step (× 10 6

)

Test Return

Time step (× 10 6

)

(a) FourRoom (1)

(b) SimpleCrossing9 (1)

(c) GotoDoor (2)

0.25

0.25

1.00

0.20

0.20

0.15

0.15

0.75

0.50

0.10

0.10

0.05

0.05

0.25

0.00

0.00

0.00

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

Ensemble and Reset Effect The effectiveness of our proposed method is due to two key factors: ensemble learning and reset. To assess the contributions of these factors, we conducted two ablation studies by evaluating the performance of RDE by (1) varying the number of ensemble agents N , and (2) eliminating the reset mechanism. In the first ablation study, we compared the performance of RDE+DQN with N = 4 and N = 2 on the MiniGrid environments. The results, shown in Fig. 5, demonstrate that increasing the number of ensemble agents improves the performance of the proposed method. In the second ablation study, we examined the performance of RDE+DQN with and without reset. Here, the latter approach solely relies on deep ensemble learning. As shown in Fig. 5, the reset mechanism significantly contributes to the enhanced performance of the proposed method, especially in the FourRoom environment.

5 Experiment: Safe Reinforcement Learning

As previously mentioned, Safe RL aims to find a high-return policy while avoiding unsafe states and actions during training as much as possible. Ray et al. [16] formulated the problem as a constrained RL problem, where the agent incurs a cost when it visits a hazardous state. The goal is to maximize the expected return while ensuring that the cumulative cost remains below a predefined threshold. On the other hand, Jung et al. [9] considered maximizing the expected return while satisfying the probability of outage events. When applying the reset mechanism in such setting, a challenge arises because the deep RL agent loses all its knowledge about states that should be avoided after operating a reset. Consequently, until the safety critic is sufficiently retrained, the agent may continue to visit these undesirable states, thereby increasing the overall training cost. We expect that RDE with a slight modification taking into account both reward and cost can maintain the training cost low while benefiting from the performance gains associated with the reset operation still.

For safe RL, we adopted WCSAC, which maximizes the expected return while constraining the CVaR defined as Eq. (1), and modified the adaptive integration method proposed in Sec. 3.2. Specifically, we modified the probability of action selection in Eq. (2) by incorporating the cost function. The modified probability of adaptive action selection is now defined as p safe select = κp select +(1 -κ ) p c select , where κ is the mixing coefficient and p c select is given by

where α c = max {| ˆ C ( s, a 1 ) | , | ˆ C ( s, a 2 ) | , . . . , | ˆ C ( s, a N ) |} /β . Here, we choose ˆ C to be the estimated CVaR value function that underwent a reset operation at the earliest point with respect to the current time step among the ensemble agent. Note that the sign of ˆ C ( s, a ) in Eq. (3) is inverted to prioritize actions with low CVaR values, as we aim to minimize the cost function.

5.1 Result

We compared our algorithm with WCSAC and SR-WCSAC, which incorporates the reset method into WCSAC, on the 3 environments in Safety-Gym . The details are provided in Appendix B.

Base

SR + X

RDE + X  (ours)

Test Return

Epoch

Test Return

Epoch

Test Return

Epoch

Train Cost

Epoch

Train Cost

Epoch

Train Cost

Epoch

(a) PointGoal

(b) PointButton

(c) CarGoal

30

20

10

20

10

0

0

-10

0

-10

-20

-20

-20

-30

-30

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

600

800

400

400

600

200

400

200

200

0

0

0

200

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Fig. 6 shows the test return and the training cost. As expected, although the application of the simple reset method leads to a slight performance improvement compared with the baseline WCSAC agent, it also incurred a higher training cost. In particular, the soaring cost makes the naive reset method impractical in real-world scenarios concerning safety. On the other hand, our algorithm achieved superior test performance compared with the simple reset method, while significantly reducing the training cost. Note that the presence of non-reset agents and adaptive composition prevent cost from soaring after the reset operation.

6 Conclusion

We have proposed a novel RDE ( R eset with D eep E nsemble) framework that ensures safety and enhances sample efficiency by leveraging deep ensemble learning and the reset mechanism. The proposed method effectively combines N ensemble agents into a single agent to mitigate performance collapse that follows a reset. The experimental results demonstrate the effectiveness of the proposed method, which significantly improves sample efficiency and final performance, while avoiding performance collapse in various reinforcement learning tasks. In addition, in the context of safe reinforcement learning, our method outperforms the vanilla reset approach without incurring high costs, whereas the latter suffers from prohibitive training cost.

Acknowledgments

This work was supported in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00469, Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones) and in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2021R1A2C2009143 Information Theory-Based Reinforcement Learning for Generalized Environments).

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017. Constrained policy optimization. In International conference on machine learning . PMLR, 22-31.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. 2021. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems 34 (2021), 29304-29320.

Oron Anschel, Nir Baram, and Nahum Shimkin. 2017. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning . PMLR, 176-185.

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research 47 (2013), 253-279.

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. 2018. Minimalistic gridworld environment for openai gym. (2018).

Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. 2023. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. In The Eleventh International Conference on Learning Representations .

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning . PMLR, 1861-1870.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. 2018. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 (2018).

Whiyoung Jung, Myungsik Cho, Jongeui Park, and Youngchul Sung. 2022. Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability. Advances in Neural Information Processing Systems 35 (2022), 6437-6449.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84-90.

Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. 2021. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning . PMLR, 6131-6141.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529-533.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. 2022. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning . PMLR, 16828-16847.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. 2016. Deep exploration via bootstrapped DQN. Advances in neural information processing systems 29 (2016).

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22, 268 (2021), 1-8. http://jmlr.org/papers/ v22/20-1364.html

Alex Ray, Joshua Achiam, and Dario Amodei. 2019. Benchmarking Safe Exploration in Deep Reinforcement Learning. (2019).

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. 2018. Deepmind control suite. arXiv preprint arXiv:1801.00690 (2018).

Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. 2021. WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35. 10639-10646.

Haichao Zhang, Wei Xu, and Haonan Yu. 2023. Policy Expansion for Bridging Offline-toOnline Reinforcement Learning. In International Conference on Learning Representations (ICLR) .

A Algorithm Pseudo-code

B Experimental Details and Hyperparameter

DeepMind Control Suite. DeepMind Control Suite (DMC) is a collection of continuous control tasks that involve the manipulation of high-dimensional systems [17]. We considered nine distinct tasks of DMC, listed in Table 2. Our implementation is based on Stable Baseline3 [15] and the hyperparameters are provided in Table 1.

2. Atari 100k. Atari 100k is a benchmark that tests an agent's abilities by allowing it to interact with 100k environment steps (equivalent to 400k frames with a frameskip of 4) in 26 Atari games [4]. Each game has different mechanics, providing a diverse evaluation of the agent's capabilities. The benchmark imposes a restriction of 100k actions per environment, which roughly corresponds to 2 hours of human gameplay. The hyperparameters are provided in Table 3 and Table 4. Our implementation is based on Stable Baseline3 [15].

3. MiniGrid. Minigrid is a collection of goal-oriented tasks in 2D grid-world. The agent receives a sparse reward R 1 = 10 if the goal is achieved. The spare reward is penalized based on the number of interaction steps. In this paper, we considered 5 tasks of Minigrid including FourRooms , SimpleCrossingS9N1 , LavaCrossingS9N1 , SimpleCrossingS9N1 , and GoToDoor-8x8 . We now provide the brief introduction of each environment.

FourRooms This environment is designed with four rooms and comprises of a single agent and a green goal. At the start of each episode, both the agent and the green goal are randomly placed within the four rooms. The goal of the agent is to navigate through the environment and ultimately reach the green goal.

SimpleCrossingS9N1, LavaCrossingS9N1 This environment is designed with two rooms that are blocked by obstacles, such as lava (for LavaCrossing) and walls (for SimpleCrossing). The objective is to successfully reach a goal while avoiding these obstacles. In LavaCrossing, the episode comes to an end if the agent collides with the obstacle, whereas in SimpleCrossing, the episode continues despite the collision.

GoToDoor-8x8 This environment is designed with a single room, four doors, and a single mission text string. The string provides instructions on which door the agent should reach.

LavaGapS7 This environment is designed with a single room, a strip of lava, and a green goal. The objective is to successfully reach the goal while avoiding the lava.

We provide the hyperparameters used in MiniGrid as follows.

4. Safety-Gym. Safety Gym presents a collection of environments having safety constraints. Within each of these environments, a robotic entity operates within a intricate setting with the objective of fulfilling a task while maintaining observance to predefined constraints governing its interactions with nearby objects and spatial regions. These environments incorporate separate reward and cost functions, which respectively describe task-specific objectives and safety prerequisites. We considered 3 environments including PointGoal, PointButton , and CarGoal . We now provide the brief introduction of each environment.

PointGoal A singular point robot is tasked with traversing toward a specific goal while effectively evading hazards. The point robot's mobility is confined to a 2-dimensional plane, facilitated by separate actuators enabling rotational motion and moving forward/backward.

PointButton This environment requires the point robot to sequentially press a series of goal buttons. These buttons, immobile in nature, are dispersed throughout the environment, compelling the agent to navigate and press towards the currently highlighed button, serving as the immediate goal.

CarGoal This environment introduces a robot of increased complexity, equipped with two independently driven parallel wheels and an additional freely rotating rear wheel. Effective coordination of both turning and forward/backward locomotion necessitates concurrent manipulation of both actuators. The objective of this environment coincides with that of the PointGoal environment.

Table 6: Hyperparameters in Safety Gym.

C Experimental Results

We provide the entire results of DQN, SR+DQN, and RDE+DQN on Atari 100k and Minigird in Table 7 and Table 8, respectively. We report the per-environment learning curves of Minigird in Fig. 9. The learning curves of DMC are provided in Fig. 7 and Fig. 8.

Base

SR + X

RDE + X  (ours)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

100

200

300

400

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

100

200

300

400

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

100

200

300

400

(a) acrobot-swingup(1)

(b) acrobot-swingup(2)

(c) acrobot-swingup(4)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

(d) cheetah-run(1)

(e) cheetah-run(2)

(f) cheetah-run(4)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

(g) finger-turn_hard(1)

(h) finger-turn_hard(2)

(i) finger-turn_hard(4)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

(j) fish-swim(1)

(k) fish-swim(2)

(l) fish-swim(4)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

100

200

300

400

500

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

100

200

300

400

500

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

100

200

300

400

500

(m) hopper-hop(1)

(n) hopper-hop(2)

(o) hopper-hop(4)

Base

SR + X

RDE + X  (ours)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

50

100

150

200

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

50

100

150

200

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

50

100

150

200

(a) humanoid-run(1)

(b) humanoid-run(2)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

(d) quadruped-run(1)

(e) quadruped-run(2)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

(g) swimmer-swimmer15(1)

(h) swimmer-swimmer15(2)

(i) swimmer-swimmer15(4)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

(j) walker-run(1)

(k) walker-run(2)

(l) walker-run(4)

Base

SR + X

RDE + X  (ours)

(a) FourRoom(0.5)

(b) FourRoom(1)

(c) FourRoom(2)

(d) Gotodoor(0.5)

(e) Gotodoor(1)

(f) Gotodoor(2)

(g) Lavacrossing9(0.5)

(h) Lavacrossing9(1)

(i) Lavacrossing9(2)

(j) Lavagap7(0.5)

(k) Lavagap7(1)

(l) Lavagap7(2)

(m) Simplecrossing(0.5)

(n) Simplecrossing(1)

(o) Simplecrossing(2)

0.25

0.25

0.25

0.20

0.20

0.20

0.15

0.15

0.15

0.10

0.10

0.10

0.05

0.05

0.05

0.00

0.00-

0.00-

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.00

1.00

1.00

335S13-5

0.75

WW

0.75

0.75

0.50

0.50-

0.50

0.25

0.25

0.25

0.00

0.00

0.00

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0.0七

0.0七

0.0七

0.000.25

0.500.751.00

0.00

0.250.500.751.00

0.00

0.250.500.751.00

1.00

1.00

1.00

0.75

0.75

0.75

0.50

0.50

0.50

0.25

0.25

0.25

0.00

0.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.500.751.00

0.0

0.20.4（

0.60.8

1.0

0.25

0.25

0.25

0.20

0.20

0.20 -

0.15

0.15

0.15

0.10

0.10-

0.10

0.05

0.05

0.05

0.00

0.00

0.00-

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

D Reset depth for Continuous Environments

In Section 4.3, we investigated the impact of reset depth in Minigird environment. In order to investigate the effect of reset depth in continuous tasks, we conduct the additional experiments of RDE with two different depths: reset-1 , which only resets the last layer, and reset-all , which entails a complete reset of all layers in the DMC environments. As shown in Fig. 10, it is observed that reset-1 exhibits comparable performance to reset-all in the cheetah-run, finger-turn_hard, hopper-hop, swimmer-swimmer15, walker-run tasks. However, reset-all performs better than reset-1 in the acrobot-swingup, fish-swim, humanoid-run, quadruped-run tasks. Furthermore, in the cases of humanoid-run , the performance of reset-1 deteriorates with increasing replay ratio, suggesting that shallower levels of resetting render it more susceptible to primacy bias.

reset - 1    (RR = 1)

reset - 1    (RR = 2)

reset - 1    (RR = 4)

reset - all  (RR = 1)

reset - all  (RR = 2)

reset - all  (RR = 4)

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

100

200

300

400

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

(a) acrobot-swingup

(b) cheetah-run

(c) finger-turn_hard

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

100

200

300

400

500

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

50

100

150

200

(d) fish-swim

(e) hopper-hop

(f) humanoid-run

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0

200

400

600

800

1000

(g) quadruped-run

(h) swimmer-swimmer15

(i) walker-run

E Further Experiments Regarding Reset Interval

The reset interval is an important hyperparameter in reset-based algorithms. We conduct performance comparisons by varying the reset interval, and the corresponding results are in Table 9. It is seen that RDE outperforms the vanilla reset method. To ensure fair comparison, we set the reset interval of one ensemble agent to match that of an SR agent. This is important because excessively frequent resetting can have a negative impact on the learning process, causing reset operations to occur before DNNs have fully recovered their performance. To illustrate this, we include additional experiments using the reset interval of T rr =1 reset / ( N × rr ) for SR (vanilla reset) in DMC and Minigrid environments. The corresponding results are shown in Fig. 11 (indicated by the green line). Notably, the highly frequent vanilla reset (with the same number of reset operations as in RDE) performs worse, even more poorly than the base algorithm in the Minigrid environment.

IQM

(a) DMC

(b) GotoDoor (0.5)

(c) GotoDoor (2)

1.00

Base

1.00

1.2

Test Return

0.75

SR+DQN

RDE+DQN

53

TestReturn

0.75

SR+DQN(HRF)

1.1

Base

0.50

0.50

SR+SAC

1.0

RDE+SAC

0.25

0.25

SR+SAC (HRF)

0.9

0.00

0.00

1

2

4

0.0

Time step (x 10°)

0.5

1.0

1.5

0.0

Time step (× 10°)

0.2

0.4

0.6

F Further Experiments Regrading # of Ensemble Agents ( N )

We include additional experiments to confirm the effectiveness of ensemble learning. Increasing the number of ensemble agents, denoted as N , can lead to greater diversity gain. Additionally, the presence of N -1 non-reset agents can aid in effectively mitigating performance collapses after resetting. Illustrated in Fig. 12, RDE with N = 4 demonstrates improved sample efficiency in the Minigrid environment. It is also observed in the Minigrid environment that the proposed method more effectively prevents performance collapse in comparison to RDE with N = 2 .

(a) GotoDoor (0.5)

(b) LavaCrossing9 (0.5)

(c) Lavagap7 (0.5)

0.4

1.00

1.00

Return

0.3

Return

0.75

Return

0.75

0.2

0.50

0.50

Test

Test

0.25

0.1

0.25

RDE (N=4)

0.00

RDE (N=2)

0.0

0.00

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Time step (x 106)

Time step (× 106)

Time step (x 106)

G Limitations

Computational costs increase linearly as we raise the number of ensemble agents or the replay ratio. A limitation of our approach is the escalated computational cost due to ensemble agents. Nonetheless, it is important to highlight that, typically, the challenge in reinforcement learning lies more in sample efficiency, owing to the substantial costs tied to environment interaction, rather than computational intricacies. We are convinced that our method substantially enhances sample efficiency and safety, especially in environments with ample computational resources.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOverfitting in deep Q-learning agents is a recent topic of interest in the RL community, and several methods have been proposed to mitigate this problem, including data augmentation (DrQ), random ensembles (RedQ, DroQ), and resets (Nikishin et al.). This paper builds upon prior work on periodically resetting weights and addresses one of its core limitations: by resetting weights, an agent will perform poorly immediately after resetting, although it eventually recovers and often exceeds its performance before resetting. This paper proposes to learn an *ensemble* of agents and periodically reset only one of the agents at a time (in sequence). Experiments are conducted on tasks from DMControl, Minigrid, and Atari100k, and indicate that resetting with ensembles is effective at mitigating deterioration of performance immediately after a reset.\n\nSTRENGTHS:\nThis paper is well written, considers an interesting problem, and experiments appear sound. The description of the proposed method is easy to follow, and especially Figure 4 is useful for understanding how the ensemble resetting works in practice.\n\nWEAKNESSES:\n- It would be useful to include more discussion on related works that seek to understand and mitigate overfitting in deep Q-learning besides resetting. There is a lot of literature in this area and I imagine that the authors are familiar with the literature, so I will refrain from mentioning any specific references (besides the methods mentioned in my summary) to remain impartial.\n\n- I would like to see more ablations to get a deeper understanding of the trade-offs in ensemble resetting vs. prior works. How does sample-efficiency and performance drop change when the number of ensemble agents $N$ varies? How often should one reset agents? Is the rate of resets and number of agents dependent on the replay ratio? What is the computational cost of using additional agents vs. increasing the replay ratio? Would the proposed method benefit from using the ensemble in other ways as well, e.g. by using the RedQ trick for computing TD-targets? Addressing some or all of these questions would likely increase the impact of the work.\n\nQUESTIONS:\nI would like the authors to address my comments listed in \"weaknesses\". If the authors are not able to conduct some or all of the experiments required to answer my questions during the rebuttal, I'd like to see a discussion of what the authors would expect the results to be based on their experience.\n\nLIMITATIONS:\nThere is sufficient discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper combines the resetting method proposed by (Nikishin et al., 2022) as a remedy to the primacy bias affecting deep RL algorithms with the use of ensembles of agents. The proposed RDE method, apart from generally improving performance, has the goal of minimizing the regret associated to a learning agent that uses periodic resets, mitigating the severeness of the performance drops it experiences right after a reset. To do this, without sacrificing too much on exploration and online data collection, the probability of executing an action in the environment is evaluated according to the oldest value function. Empirical results show benefits in using this approach, at both low and high replay ratio, in standard robotic locomotion and navigation tasks, as well as a safety domain.\n\nSTRENGTHS:\n**Originality**: despite the combination of resets and ensembles is not particularly original, the focus on developing a technique for leveraging their combination to avoid performance collapse while resetting is, to the best of my knowledge, new.\n\n**Quality**: the quality of the work is generally good. The experiments cover a reasonable number of domains and the ablations mostly answer natural questions.\n\n**Clarity**: the clarity of the writing is good. The paper would benefit from some small tuning to the presentation here and there, but the overall flow makes clear what the contribution is.\n\n**Significance**: harnessing the performance benefits originating from the mitigation of the primacy bias while at the same time not incurring a cost in terms of regret is a worthy research direction which could be interesting to many practitioners and researchers.\n\nWEAKNESSES:\n**Major Concerns**\n- If I understand it correctly from Algorithm 1, a different policy could be potentially selected at each step in the environment. This could conceptually create problems in terms of inconsistent behavior, since each policy will have to deal with the actions previously sampled from a potentially very different policy, and in terms of lack of \"deep exploration\", since this would harm temporally-consistent behaviors. A reader would benefit from a discussion or analysis of this aspect, to understand whether this is happening at all, or, if not, why that might be not happening in this kind of tasks of setting.\n- The idea of combining periodic reset and ensembles of agents in continuous control has been explored in \"Unleashing The Potential of Data Sharing in Ensemble Deep Reinforcement Learning\" (Lin et al, 2022). I find the idea of the paper of using this combination to avoid performance drops to still be valuable, but discussing the relationship with that paper could better contextualize the contribution.\n\n**Minor Concerns**\n- Y and X labels are missing from all Figures in the paper, making it hard to parse the plots at first sight. For most plots, it is either quite easy to infer the quantities of interest, or they are explicitly mentioned in the caption, but it can still be very misleading for many readers.\n- The paper would benefit from the extension of some of the ablations to more tasks. In particular, I find the performance of the ensemble-based approach without any resetting on top to be an important baseline to contextualize the results in Figure 3, and I think results concerning it would be a good addition to the Figure.\n\nQUESTIONS:\nI ask the authors to provide answers to my concerns expressed above.\n\nLIMITATIONS:\nThe authors do not discuss any computational consideration resulting from their use of ensembles of agents. I encourage the authors to add such a discussion to the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes an extension to the resetting strategy proposed by Nikishin et al. The extension intends to mitigate the catastrophic performance collapse often observed for the simple resetting strategy while keeping the properties that help avoid the primacy bias.\nTo this end, the work makes use of ensembling techniques, such that the policy/value network of an RL agent is not completely reset, but can fall back to different checkpoints.\nThe method is evaluated on a variety of environments and a further extension is presented that enables application in safety critical systems.\n\nSTRENGTHS:\n* The presented method cleverly combines ensembling with resetting to improve RL agents.\n* It seems general enough that it should be usable as a plug and play method for a broad variety of RL agents.\n* The work mostly easy to follow\n* It is demonstrated that the method is flexible and can incorporate auxiliary information, such as safety-critical information to provide a safer method than the vanilla resetting one\n\nWEAKNESSES:\nThe presentation could be improved:\n* A lot of the discussed \"preliminaries\" seem irrelevant for the content of the paper. For example, the paragraph on \"Off-policy RL.\" seems not necessary and the content in \"Primacy Bias and Resetting Deep RL Agent.\" is largely a repetition of the introductory text.\n* Algorithm 1 is never discussed in the text and feels wholly redundant with Fig. 1. This half page might be used to show more experimental results.\n* Algorithm 1 is not consistent with the text. From the Algorithm it looks like every ensemble member is reset every $N\\times T_{reset}$ time-steps and not $T_{reset}/N$ as stated in line 149. \n* Lines 178 - 180 are concerned with expressing that the \"oldest Q-function\" is used to normalize in the selection mechanism. This can be expressed in much clearer terms \"oldest Q-function\" as is shown in line 183.\n* In line 195 it is claimed that RDE effectively prevents performance collapses, however I disagree with this wording. It can mitigate it to some extent, but the experiments clearly show that there is still performance collapses happening.\n* The description of Figure 2 is wholly confusing. The long sentence explaining what the y-axis is showing is expressed in a very convoluted way.\n* In figure 2c it should not be possible that the performance of the baseline \"Base\" is below 1 when using an RR of 1.\n* Design decisions are often not well enough explained.\n* Figure 1 does not explain what RR is\n* The paragraph heading in line 204 should be \"Baselines & RL Agents\" not \"Baselines & DNN Architectures\"\n* Line 246 claims that there is no significant drop on the humanoid-run example. The reward for the RDE method drops from 150 - 100. This represents a significant drop in my opinion since 1/3 of the performance is lost.\n* The choice for designing the selection mechanism is very unclear. The discussion about using the oldest Q-function in the selection mechanism seems more to point to only using the oldest Q-function. Indeed, this seems to be supported in the experiments and should have been at least an ablation. Additionally the initial paragraph of Section 3.2 seems to also point to only using the oldest Q-function\n* It is claimed that a $\\beta$ set to 50 \"nearly eliminated\" performance collapses (line 270). However, the performance collapse in the Figure is again the 1/3 loss in performance.\n* Section 5 feels like an added afterthought. I don't see why it warrants an own section. It repeats some of the discussion of safe RL from the preliminaries section. Instead the \"safe\" selection mechanism should have been discussed in Section 3 and then only the results should be a subsection of Section 4.\n* Where does the value for the reset frequency $4\\times 10^5$ in line 221 come from?\n\nThe experiments seem to have an unfair comparison and are likely not reproducible:\n* All details about how hyperparameters were determined seems to be missing\n* The value of $N$ is never stated. Only from the plots can it be assumed to be 2\n* The ablation does not take into account all confounding factors and should be redone\n* It is claimed that using a reset frequency for the individual members being equal to the single network case is fair. I fail to see how this is fair. This seems to just give benefit to the RDE method since it can benefit $N$ times as much from resetting.\n* Without stating how hyperparameters were set of the methods, it seems like an arbitrary comparison of the methods.\n* N is not really ablated and it's not clear how the ensemble increases computational overhead.\n\nThe description of an MDP is wrong. An MDP is an abstract representation of an environment. The MDP does not consist of an agent.\nThe bounds for the discount factor in line 74 should be $\\gamma \\in [0,1]$ not $\\gamma \\in [0,1)$. It is totally valid to have undiscounted cases.\n\nOverall, the work would need significant rewriting and an overhaul for the experiments for me to accept it. I am very doubtful that this can be done in the time-frame of a rebuttal.\n\nQUESTIONS:\nHow would PBT style methods compare to resetting strategies. Are they doing some form of resetting?\n\nLIMITATIONS:\nLimitations of the method have not been discussed. An obvious limitation is that ensembles likely will increase the computational overhead. For example, in line 131 it is said that the simple resetting strategy often requires high replay ration and therefor more resources. This should likely be worse for the presented method and the conducted experiments are not convincing enough to show that the novel method would require fewer resources.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel reset-based method that leverages deep ensemble learning to address the primacy bias issue in deep reinforcement learning. The authors construct N-ensemble agents and reset each ensemble agent sequentially to prevent performance collapses and improve sample efficiency. The proposed method is evaluated through experiments on various environments, including safe RL scenarios, and the results demonstrate its effectiveness and potential for real-world applications.\n\nSTRENGTHS:\n1. The paper addresses an important issue in deep reinforcement learning, namely primacy bias, and proposes a novel method to mitigate its effects. This is valuable as primacy bias can lead to overfitting and performance deterioration, which affects the applicability and efficiency of deep RL algorithms.\n2. The use of deep ensemble learning in the proposed method is innovative and practical. Deep ensemble learning has shown effectiveness in domains such as image classification and RL, and leveraging the diversity gain of ensemble agents can enhance performance and prevent performance collapses.\n3. The paper provides a comprehensive analysis of the proposed method, including its underlying operations and how it effectively prevents performance collapses. This analysis adds clarity and depth to the understanding of the proposed method.\n\nWEAKNESSES:\n1. The paper stated that RDE was conducted for tasks with safety requirements, but it only did one experiment on safe RL benchmark. Thus the paper lacks effective validations for RDE on violations. Simply stating that RDE does not cause performance collapse in general continuous control tasks cannot explain its role in safe RL.\n\nQUESTIONS:\n1. Can you show more experimental results on safe RL benchmark?\n\nLIMITATIONS:\nThe paper stated that RDE was conducted for tasks with safety requirements, but it only did one experiment on safe RL benchmark. The paper has no negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}