SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The performance results of the proposed method and the baselines on the considered environments are shown in Fig. 3, where the considered tasks are hopper-hop and humanoid-run in DMC and Fourroom and GotoDoor in MiniGrid. It is seen that the proposed method not only yields superior performance to the baselines but also prevents performance collapses. In the Fourroom environment, the proposed method solves the game whereas both the base algorithm and the vanilla reset method fail to learn. The inherent loss of exploitation ability following a reset operation in the vanilla reset method prevents effective learning of this hard task. In other environments, as we expected, the vanilla reset method suffers from performance collapses. Especially in humanoid-run, there is a clear difference between our method and the vanilla reset method: immediately after a reset operation, our method sustains performance without a significant drop, whereas the vanilla reset method collapses to the initial performance.
More results on Atari-100k are provided in Appendix C.
Performance Collapse The results in Sec. 4.2 demonstrate the effectiveness of the proposed method in eliminating performance collapses. As aforementioned, the probability of selecting an action from the ensemble of  $N$  actions is determined based on Eq. (2), which assigns a lower selection
(a) hopper-hop (1)
(b) hopper-hop (4)
(c) humanoid-run (1)
(d) humanoid-run (4)
(e) FourRoom (0.5)
(f) FourRoom (2)
Figure 3: Performance comparison on hopper-hop and humanoid-run in DMC and Fourroom and GotoDoor in MiniGrid. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is  $10^{6}$ . Performances are averaged over 5 seeds.
(g) GotoDoor (0.5)
(h) GotoDoor (2)
probability to an action of a recently reset agent. This effectively prevents performance collapses, and the degree of this effect can be controlled by adjusting the parameter  $\beta$ . To investigate the impact of the parameter  $\beta$ , we conducted experiments on the humanoid-run task using RDE+SAC with
$N = 2$ . We varied  $\beta$  across  $\{-10,0,50,300\}$  and measured the corresponding empirical probability  $p_1$ , which denote the probability of selecting the action generated by  $\theta_{1}$ . In Fig. 4 (b), we observe that the empirical  $p_1$  experiences a sudden drop immediately after resetting  $\theta_{1}$ . On the other hand, when  $\theta_{2}$  is reset,  $p_1$  jumps up, indicating the sudden drop of  $p_2$ . Note that the occurrence of performance collapses was nearly eliminated when  $\beta$  was set to 300. This demonstrates the effectiveness of our adaptive integration in mitigating performance collapses
(a) Performance
(b) Empirical  $p_1$
Figure 4: Performance and empirical probability of action selection from  $\theta_{1}$  for RDE+SAC with varying  $\beta$  values on humanoid-run.
by appropriately reducing the probability of selecting actions from recently-initialized agents. Furthermore, increasing the value of  $\beta$  quickly adapt the values of  $p_{select}$ , leading to effective reduction of performance collapses.
Reset Depth The choice of reset depth is a crucial hyperparameter that significantly impacts the final performance of reset-based methods. We observed that a higher reset depth can provide an advantage in more challenging tasks that require hard exploration, whereas it may present a slight disadvantage in relatively easy tasks. We provide the performance of RDE+DQN with two different reset depths: reset-part, which resets the last two layers, and reset-all, which completely reset all layers. These evaluations were conducted on three MiniGrid environments, comprising two challenging environments, FourRoom and SimpleCrossing9, and one comparatively easier environment, GotoDoor. As shown in Fig. 5, reset-part exhibits inferior performance and instability compared with reset-all in two relatively challenging tasks. However, in two relatively easy tasks, the reset-pt performs marginally better than reset-all.
(a) FourRoom (1)
(b) SimpleCrossing9 (1)
(c) GotoDoor (2)
Figure 5: Ablation studies regarding reset depth, ensemble learning, and reset mechanism. Performances are averaged over 5 seeds.
Ensemble and Reset Effect The effectiveness of our proposed method is due to two key factors: ensemble learning and reset. To assess the contributions of these factors, we conducted two ablation studies by evaluating the performance of RDE by (1) varying the number of ensemble agents  $N$ , and (2) eliminating the reset mechanism. In the first ablation study, we compared the performance of RDE+DQN with  $N = 4$  and  $N = 2$  on the MiniGrid environments. The results, shown in Fig. 5, demonstrate that increasing the number of ensemble agents improves the performance of the proposed method. In the second ablation study, we examined the performance of RDE+DQN with and without reset. Here, the latter approach solely relies on deep ensemble learning. As shown in Fig. 5, the reset mechanism significantly contributes to the enhanced performance of the proposed method, especially in the FourRoom environment.
5 Experiment: Safe Reinforcement Learning
As previously mentioned, Safe RL aims to find a high-return policy while avoiding unsafe states and actions during training as much as possible. Ray et al. [16] formulated the problem as a constrained RL problem, where the agent incurs a cost when it visits a hazardous state. The goal is to maximize the expected return while ensuring that the cumulative cost remains below a predefined threshold. On the other hand, Jung et al. [9] considered maximizing the expected return while satisfying the probability of outage events. When applying the reset mechanism in such setting, a challenge arises because the deep RL agent loses all its knowledge about states that should be avoided after operating a reset. Consequently, until the safety critic is sufficiently retrained, the agent may continue to visit these undesirable states, thereby increasing the overall training cost. We expect that RDE with a slight modification taking into account both reward and cost can maintain the training cost low while benefiting from the performance gains associated with the reset operation still.
For safe RL, we adopted WCSAC, which maximizes the expected return while constraining the CVaR defined as Eq. (1), and modified the adaptive integration method proposed in Sec. 3.2. Specifically, we modified the probability of action selection in Eq. (2) by incorporating the cost function. The modified probability of adaptive action selection is now defined as  $p_{select}^{safe} = \kappa p_{select} + (1 - \kappa)p_{select}^c$ , where  $\kappa$  is the mixing coefficient and  $p_{select}^c$  is given by
$$
p _ {s e l e c t} ^ {c} = \left[ p _ {1} ^ {c}, p _ {2} ^ {c}, \dots , p _ {N} ^ {c} \right] = \operatorname {s o f t m a x} \left[ - \hat {C} (s, a _ {1}) / \alpha_ {c}, - \hat {C} (s, a _ {2}) / \alpha_ {c}, \dots , - \hat {C} (s, a _ {N}) / \alpha_ {c} \right], \tag {3}
$$
where  $\alpha_{c} = \max \{|\hat{C}(s, a_{1})|, |\hat{C}(s, a_{2})|, \ldots, |\hat{C}(s, a_{N})|\} / \beta$ . Here, we choose  $\hat{C}$  to be the estimated CVaR value function that underwent a reset operation at the earliest point with respect to the current time step among the ensemble agent. Note that the sign of  $\hat{C}(s, a)$  in Eq. (3) is inverted to prioritize actions with low CVaR values, as we aim to minimize the cost function.
5.1 Result
We compared our algorithm with WCSAC and SR-WCSAC, which incorporates the reset method into WCSAC, on the 3 environments in Safety-Gym. The details are provided in Appendix B.
Base SR + X RDE + X (ours)
(a) PointGoal
(b) PointButton
(c) CarGoal
Figure 6: Performance Comparison in Safety Gym Environments. The first and second rows show the test returns and training costs, respectively. The performance metrics are averaged over five different seeds.
Fig. 6 shows the test return and the training cost. As expected, although the application of the simple reset method leads to a slight performance improvement compared with the baseline WCSAC agent, it also incurred a higher training cost. In particular, the soaring cost makes the naive reset method impractical in real-world scenarios concerning safety. On the other hand, our algorithm achieved superior test performance compared with the simple reset method, while significantly reducing the training cost. Note that the presence of non-reset agents and adaptive composition prevent cost from soaring after the reset operation.
We have proposed a novel RDE (Reset with Deep Ensemble) framework that ensures safety and enhances sample efficiency by leveraging deep ensemble learning and the reset mechanism. The proposed method effectively combines  $N$  ensemble agents into a single agent to mitigate performance collapse that follows a reset. The experimental results demonstrate the effectiveness of the proposed method, which significantly improves sample efficiency and final performance, while avoiding performance collapse in various reinforcement learning tasks. In addition, in the context of safe reinforcement learning, our method outperforms the vanilla reset approach without incurring high costs, whereas the latter suffers from prohibitive training cost.
This work was supported in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00469, Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones) and in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2021R1A2C2009143 Information Theory-Based Reinforcement Learning for Generalized Environments).
[18] Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. 2021. WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 10639-10646.
[19] Haichao Zhang, Wei Xu, and Haonan Yu. 2023. Policy Expansion for Bridging Offline-to-Online Reinforcement Learning. In International Conference on Learning Representations (ICLR).
A Algorithm Pseudo-code
Algorithm 1 RDE+X
Input: Algorithm  $X$  Ensemble size  $N$  Reset frequency  $T_{\mathrm{reset}}$  ,Replay Ratio RR, Coefficient  $\beta$  Initialize  $N$  -agents parameters,  $(\theta_{1},\theta_{2},\dots ,\theta_{N})$  , and the replay buffer  $\mathcal{D}$  . Set index  $k = 0$  for each episode do for each timestep t do for  $i = 1$  to  $N$  do Collect action  $a_{t}^{i}\sim \pi_{i}(a_{t}^{i}|s_{t})$  end for Calculate  $p_{select}$  according to Eq. 2 and sample  $a_{t}$  based on  $p_{select}$  Execute  $a_{t}$  and get  $r_t$  and  $s_{t + 1}$  . Store a transition to  $\mathcal{D}$  for  $j = 1$  to  $RR$  do Sample a random minibatch from  $\mathcal{D}$  Update  $(\theta_{1},\theta_{2},\dots ,\theta_{N})$  based on  $X$  end for if  $t \% (T_{\mathrm{reset}} / N) == 0$  then Reset  $\theta_{k}$  and  $k\gets (k + 1)\% N$  end if end for end for
B Experimental Details and Hyperparameter
1. DeepMind Control Suite. DeepMind Control Suite (DMC) is a collection of continuous control tasks that involve the manipulation of high-dimensional systems [17]. We considered nine distinct tasks of DMC, listed in Table 2. Our implementation is based on Stable Baseline3 [15] and the hyperparameters are provided in Table 1.
Hyperparameters	Value
# of ensemble agents	2
Training steps	1 × 106
Discount factor	0.99
Initial collection steps	5000
Minibatch size	1024
Optimizer (all)	Adam
Optimizer (all) : learning rate	0.0001 humanoid-run
0.0003 otherwise
Networks (all) : activation	ReLU
Networks (all) : n. hidden layers	2
Networks (all) : hidden units	1024
Initial Temperature	1
Replay Buffer Size	1 × 106
Updates per step (Replay Ratio)	(1, 2, 4)
Target network update period	1
τ	0.005
Reset Interval (gradient steps)	4 × 105
β (action select coefficient)	50
Environment	Task
acrobot	swingup
cheetah	run
finger	turn_hard
fish	swim
hopper	hop
humanoid	run
quadruped	run
swimmer	swimmer15
walker	run
Table 2: The considered tasks of DMC
Table 1: Hyperparameters for RDE+SAC on DMC.
2. Atari 100k. Atari 100k is a benchmark that tests an agent's abilities by allowing it to interact with 100k environment steps (equivalent to 400k frames with a frameskip of 4) in 26 Atari games [4]. Each game has different mechanics, providing a diverse evaluation of the agent's capabilities. The benchmark imposes a restriction of 100k actions per environment, which roughly corresponds to 2 hours of human gameplay. The hyperparameters are provided in Table 3 and Table 4. Our implementation is based on Stable Baseline3 [15].
Reset Interval	Environments
4 × 104	Assault, Asterix, Battle Zone, Boxing, Crazy CLimber, Freeway, Frostbite, Krull, Ms Pacman, Qbert, Seaquest, Up N Down
8 × 104	Alien, Amidar, Bank Heist, Breakout, Chopper Command, Demon Attack, Gopher, Hero, Jamesbond, Kangaroo, Kung Fu Master, Pong, Private Eye, Road Runner
Table 3: Reset Interval in terms of the gradient step for each environment of Atari-100k
Hyperparameters	Value
# of ensemble agents	2
Gray-scaling	True
Observation down-sampling	84 × 84
Frames stacked	4
Action repetitions	4
Reward clipping	[-1, 1]
Terminal on loss of life	True
Max gradient norm	10
Replay学期 every	1 step
Training steps	1 × 105
Discount factor	0.99
Initial collection steps	1 × 104
Minibatch size	32
Optimizer	Adam
Optimizer: learning rate	0.0001
Q network: channels	32, 64, 64
Q network: filter size	8 × 8, 4 × 4, 3 × 3
Q network: stride	4, 2, 1
Q network: activation	ReLU
Q network: hidden units	512
Replay Buffer Size	1 × 105
Updates per step (Replay Ratio)	(1, 2, 4)
Target network update period	1
Exploration	ε-greedy
ε-decay	1 × 104
τ	0.005
β (action select coefficient)	50
Reset depth	last 2 layers: Amidar, Asterix Bank Heist, Freeway, Frostbite Gopher, Hero, Kangaroo, Pong last 1 layer: otherwise
Table 4: Hyperparameters for RDE+DQN on Atari 100k.
3. MiniGrid. Minigrid is a collection of goal-oriented tasks in 2D grid-world. The agent receives a sparse reward  $R_{1} = 10$  if the goal is achieved. The spare reward is penalized based on the number of interaction steps. In this paper, we considered 5 tasks of Minigrid including FourRooms, SimpleCrossingS9N1, LavaCrossingS9N1, SimpleCrossingS9N1, and GoToDoor-8x8. We now provide the brief introduction of each environment.
FourRooms This environment is designed with four rooms and comprises of a single agent and a green goal. At the start of each episode, both the agent and the green goal are randomly placed within the four rooms. The goal of the agent is to navigate through the environment and ultimately reach the green goal.
SimpleCrossingS9N1, LavaCrossingS9N1 This environment is designed with two rooms that are blocked by obstacles, such as lava (for LavaCrossing) and walls (for SimpleCrossing). The objective is to successfully reach a goal while avoiding these obstacles. In LavaCrossing, the episode comes to an end if the agent collides with the obstacle, whereas in SimpleCrossing, the episode continues despite the collision.
GoToDoor-8x8 This environment is designed with a single room, four doors, and a single mission text string. The string provides instructions on which door the agent should reach.
LavaGapS7 This environment is designed with a single room, a strip of lava, and a green goal. The objective is to successfully reach the goal while avoiding the lava.
We provide the hyperparameters used in MiniGrid as follows.
Hyperparameters	Value
ε	0.9 → 0.05
ε-decay time step	10^5
target update period	10^3
Replay buffer size	5 × 10^5
Mini-batch size	256
Optimizer	RMSProp
Learning rate	0.0001
The maximum number of steps	100
Reset Interval (gradient steps)	2 × 10^5 (GoToDoor, LavaCrossing, LavaGap)
1 × 10^5 (FourRooms, SimpleCrossing)
β (action select coefficient)	50
Table 5: Hyperparameters in MiniGrid.
4. Safety-Gym. Safety Gym presents a collection of environments having safety constraints. Within each of these environments, a robotic entity operates within a intricate setting with the objective of fulfilling a task while maintaining observance to predefined constraints governing its interactions with nearby objects and spatial regions. These environments incorporate separate reward and cost functions, which respectively describe task-specific objectives and safety prerequisites. We considered 3 environments including PointGoal, PointButton, and CarGoal. We now provide the brief introduction of each environment.
PointGoal A singular point robot is tasked with traversing toward a specific goal while effectively evading hazards. The point robot's mobility is confined to a 2-dimensional plane, facilitated by separate actuators enabling rotational motion and moving forward/backward.
PointButton This environment requires the point robot to sequentially press a series of goal buttons. These buttons, immobile in nature, are dispersed throughout the environment, compelling the agent to navigate and press towards the currently highlighted button, serving as the immediate goal.
CarGoal This environment introduces a robot of increased complexity, equipped with two independently driven parallel wheels and an additional freely rotating rear wheel. Effective coordination of both turning and forward/backward locomotion necessitates concurrent manipulation of both actuators. The objective of this environment coincides with that of the PointGoal environment.
Hyperparameters	Value
# of ensemble agents	2
Epochs	100
Steps per epcoh	20000
Discount factor	0.99
Learning start steps	500
Minibatch size	256
Optimizer (all)	Adam
Optimizer (all) : learning rate	0.001
Networks (all) : activation	ReLU
Networks (all) : n. hidden layers	2
Networks (all) : hidden units	256
Initial Temperature	1
Entropy constraint	-1
Replay Buffer Size	\( 1 \times 10^6 \)
Target network update period	1
τ	0.005
Safety constraint	25
Risk level	0.5
Reset Interval (gradient steps)	\( 4 \times 10^5 \)
\( \beta \) (action select coefficient)	50
Table 6: Hyperparameters in Safety Gym.
We provide the entire results of DQN, SR+DQN, and RDE+DQN on Atari 100k and Minigird in Table 7 and Table 8, respectively. We report the per-environment learning curves of Minigird in Fig. 9. The learning curves of DMC are provided in Fig. 7 and Fig. 8.
RR	1	2	4
Game	DQN	SR+DQN	RDE+DQN	DQN	SR+DQN	RDE+DQN	DQN	SR+DQN	RDE+DQN
Alien	423.2	512.4	414.4	596.6	506.4	502.4	414.0	639.6	610.0
Amidar	46.8	43.2	47.8	54.6	58.2	68.2	31.6	66.4	55.2
Assault	438.8	354.5	409.1	409.9	369.2	431.8	372.1	455.3	462.0
Asterix	418.0	426.0	418.0	394.0	474.0	562.0	306.0	620.0	678.0
Bank Heist	14.0	13.6	16.8	23.2	27.2	21.2	14.4	26.8	33.6
Battle Zone	4040.0	3360.0	4040.0	2120.0	4520.0	7880.0	3840.0	7000.0	8240.0
Boxing	1.4	-7.7	-2.6	0.8	-0.6	4.2	5.1	3.6	1.9
Breakout	16.1	6.7	16.1	19.6	15.0	21.2	23.8	20.8	19.5
Chopper Command	828.0	836.0	760.0	1324.0	1120.0	1000.0	1080.0	1024.0	1044.0
Crazy Climber	12472.0	16240.0	22556.0	22100.0	22216.0	25784.0	16028.0	25072.0	56324.0
Demon Attack	490.8	166.8	324.6	1492.4	184.4	652.4	1088.4	355.6	284.8
Freeway	15.1	7.2	4.0	10.9	6.4	16.8	14.8	7.9	21.2
Frostbite	233.6	158.4	197.6	206.4	206.8	348.0	116.4	264.4	271.6
Gopher	225.6	390.4	470.4	460.8	720.6	911.2	434.0	1169.6	1045.6
Hero	621.6	738.4	1698.6	1068.8	2725.6	2819.4	754.0	3073.2	3564.2
Jamesbond	68.0	70.0	126.0	178.0	50.0	138.0	140.0	92.0	78.0
Kangaroo	168.0	104.0	72.0	160.0	160.0	176.0	48.0	232.0	208.0
Krull	1905.2	2262.4	5325.6	2637.5	2460.4	1854.0	2533.6	3144.0	3374.0
Kung Fu Master	8264.0	5908.0	7256.0	6244.0	8216.0	7524.0	6008.0	7996.0	8284.0
Ms Pacman	790.0	769.6	609.6	907.2	832.0	831.6	868.4	954.8	1223.2
Pong	-20.7	-20.6	-20.0	-19.1	-18.8	-17.1	-14.4	-16.0	-16.8
Private Eye	20.0	2.1	44.0	44.0	-69.1	64.0	0.0	40.0	83.5
Qbert	457.0	388.0	415.0	497.0	489.0	436.0	615.0	467.0	941.0
Road Runner	2288.0	576.0	1976.0	2288.0	2488.0	3180.0	1680.0	1924.0	3036.0
Seaquest	292.0	222.4	243.2	207.2	240.0	337.6	216.0	341.6	372.0
Up N Down	1396.8	1068.8	1734.4	1756.0	1769.2	2258.0	1662.4	1472.4	1503.6
IQM	1.000	0.852	0.987	1.203	1.063	1.377	1.073	1.351	1.422
Mean	1.000	0.603	1.092	1.323	1.050	1.695	1.181	1.641	1.902
Table 7: Results on Atari-100k
RR	0.5	1	2
Game	DQN	SR+DQN	RDE+DQN	DQN	SR+DQN	RDE+DQN	DQN	SR+DQN	RDE+DQN
GoToDoor-8x8	0.709	0.71	0.911	0.544	0.659	0.944	0.159	0.684	0.929
LavaCrossing	0.035	0	0.248	0.016	0.02	0.215	0.029	0	0.237
SimpleCrossingS9N1	0	0.012	0.186	0	0.013	0.159	0	0.014	0.137
FourRooms	0.002	0.03	0.148	0	0.072	0.159	d 0	0.034	0.155
LavaGapS7	0.747	0.655	0.793	0.761	0.729	0.793	0.674	0.706	0.791
Table 8: Results on MiniGrid
(a)acrobot-swingup(1)
Base SR + X RDE + X (ours)
(b)acrobot-swingup(2)
(c)acrobot-swingup(4)
(d) cheetah-run(1)
(e) cheetah-run(2)
(f) cheetah-run(4)
(g) finger-turn-hard(1)
(h) finger-turn-hard(2)
(i) finger-turn-hard(4)
(j) fish-swim(1)
(k) fish-swim(2)
(1) fish-swim(4)
(m) hopper-hop(1)
Figure 7: Per-environment performance in DMC with varying replay ratio values. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is  $10^{6}$ . Performances are averaged over 5 seeds.
(n) hopper-hop(2)
(o) hopper-hop(4)
(a) humanoid-run(1)
Base SR + X RDE + X (ours)
(b) humanoid-run(2)
(c) humanoid-run(4)
(d) quadruped-run(1)
(e) quadruped-run(2)
(f) quadruped-run(4)
(g) swimmer-swimmer15(1)
(h) swimmer-swimmer15(2)
(i) swimmer-swimmer15(4)
(j) walker-run(1)
Figure 8: Per-environment performance in DMC with varying replay ratio values. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is  $10^{6}$ . Performances are averaged over 5 seeds.
(k) walker-run(2)
(1) walker-run(4)
(a) FourRoom(0.5)
Base SR + X RDE + X (ours)
(b) FourRoom(1)
(c) FourRoom(2)
(d) Gotodoor(0.5)
(e) Gotodoor(1)
(f) Gotodoor(2)
(g) Lavacrossing9(0.5)
(h) Lavacrossing9(1)
(i) Lavacrossing9(2)
(j) Lavagap7(0.5)
(k) Lavagap7(1)
(1) Lavagap7(2)
(m) Simplecrossing(0.5)
(n) Simplecrossing(1)
Figure 9: Per-environment performance in minigird with varying replay ratio values. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is  $10^{6}$ . Performances are averaged over 5 seeds.
(o) Simplecrossing(2)
D Reset depth for Continuous Environments
In Section 4.3, we investigated the impact of reset depth in Minigird environment. In order to investigate the effect of reset depth in continuous tasks, we conduct the additional experiments of RDE with two different depths: reset-1, which only resets the last layer, and reset-all, which entails a complete reset of all layers in the DMC environments. As shown in Fig. 10, it is observed that reset-1 exhibits comparable performance to reset-all in the cheetah-run, finger-turn-hard, hopper-hop, swimmer-swimmer15, walker-run tasks. However, reset-all performs better than reset-1 in the acrobot-swingup, fish-swim, humanoid-run, quadruped-run tasks. Furthermore, in the cases of humanoid-run, the performance of reset-1 deteriorates with increasing replay ratio, suggesting that shallower levels of resetting render it more susceptible to primacy bias.
(a)acrobot-swingup
(b) cheetah-run
(c) finger-turn-hard
(d) fish-swim
(e) hopper-hop
(f) humanoid-run
(g) quadruped-run
(h) swimmer-swimmer15
(i) walker-run
Figure 10: Per-environment performance in DMC with varying replay ratio values. Note that the number in parentheses indicates the replay ratio. The scale of the x-axis is  $10^{6}$ . Performances are averaged over 5 seeds.
E Further Experiments Regarding Reset Interval