SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The reset interval is an important hyperparameter in reset-based algorithms. We conduct performance comparisons by varying the reset interval, and the corresponding results are in Table 9. It is seen that RDE outperforms the vanilla reset method. To ensure fair comparison, we set the reset interval of one ensemble agent to match that of an SR agent. This is important because excessively frequent resetting can have a negative impact on the learning process, causing reset operations to occur before DNNs have fully recovered their performance. To illustrate this, we include additional experiments using the reset interval of  $T_{\text{reset}}^{rr=1} / (N \times rr)$  for SR (vanilla reset) in DMC and Minigrid environments. The corresponding results are shown in Fig. 11 (indicated by the green line). Notably, the highly frequent vanilla reset (with the same number of reset operations as in RDE) performs worse, even more poorly than the base algorithm in the Minigrid environment.
RR	1	2	4
Treset	1 × 105	2 × 105	4 × 105	1 × 105	2 × 105	4 × 105	1 × 105	2 × 105	4 × 105
SR+SAC	1.00	1.08	1.02	1.00	1.15	1.15	1.02	1.13	1.21
RDE+SAC	0.99	1.10	1.20	1.02	1.10	1.17	1.02	1.16	1.25
Table 9: Results on DMC with varying reset interval
(b) GotoDoor (0.5)
Figure 11: The green line represents  $\mathrm{SR + X}$  with an equal number of reset operations as  $\mathrm{RDE + X}$ . (a) IQM metric normalized by SAC with a replay ratio of 1 on DMC. (b) & (c): Test performances on GotoDoor. High reset frequency (HRF) refers to the number of reset operations being the same as in RDE.
(c) GotoDoor (2)
F Further Experiments Regrading # of Ensemble Agents  $(N)$
We include additional experiments to confirm the effectiveness of ensemble learning. Increasing the number of ensemble agents, denoted as  $N$ , can lead to greater diversity gain. Additionally, the presence of  $N - 1$  non-reset agents can aid in effectively mitigating performance collapses after resetting. Illustrated in Fig. 12, RDE with  $N = 4$  demonstrates improved sample efficiency in the Minigrid environment. It is also observed in the Minigrid environment that the proposed method more effectively prevents performance collapse in comparison to RDE with  $N = 2$ .
(a) GotoDoor (0.5)
(b) LavaCrossing9 (0.5)
Figure 12: Ablation study in MiniGrid environments with varying the number of ensemble agents,  $N$ . The red line and brown line represent RDE with  $N = 4$  and RDE with  $N = 2$ , respectively.
(c) Lavagap7 (0.5)
Computational costs increase linearly as we raise the number of ensemble agents or the replay ratio. A limitation of our approach is the escalated computational cost due to ensemble agents. Nonetheless, it is important to highlight that, typically, the challenge in reinforcement learning lies more in sample efficiency, owing to the substantial costs tied to environment interaction, rather than computational intricacies. We are convinced that our method substantially enhances sample efficiency and safety, especially in environments with ample computational resources.