SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We call the step of computing  $\epsilon_{\rho}^{*}(\pmb{w})$  as the perturbation step of SAM. As a result of the perturbation step, we can approximately solve the problem (5) with any gradient update algorithm (i.e., base optimizer), such as Stochastic Gradient Descent (SGD) [56], SGD with momentum [54], or Adam [35], using the SAM gradient at current weight  $\pmb{w}_{t}$ :  $\nabla_{\pmb{w}}\mathcal{L}(\pmb{w}_{t} + \epsilon_{\rho}^{*}(\pmb{w}_{t}))$ . We call the step of updating the weights with SAM gradient as the update step of SAM. In essence, SAM is an iterative algorithm alternating perturbation and update steps. Readers can check a more detailed derivation of the SAM gradient in [21].
It is both theoretically and empirically proven that such an optimization scheme implicitly prefers the weights in smoother (i.e., less sharp) region of loss landscape, in terms of, e.g., the maximum eigenvalue and/or trace of Hessian [21, 31, 70, 62].
Naturally, in (deep) RL, we never have access to the full static loss function as described above. Hence, analogous to most of the applications of SAM, it is natural to randomly sample a mini-batch  $\mathcal{B}_t$  of  $m$  transitions from replay buffer and apply SAM update (often called  $m$ -SAM) using the stochastic gradient  $\nabla_{\boldsymbol{w}}\mathcal{L}_{\mathcal{B}_t}\left(\boldsymbol{w}_t + \epsilon_{\rho,\mathcal{B}_t}^*(\boldsymbol{w}_t)\right)$ , where
$$
\boldsymbol {\epsilon} _ {\rho , \mathcal {B}} ^ {*} (\boldsymbol {w}) := \frac {\rho}{\| \nabla_ {\boldsymbol {w}} \mathcal {L} _ {\mathcal {B}} (\boldsymbol {w}) \| _ {2}} \nabla_ {\boldsymbol {w}} \mathcal {L} _ {\mathcal {B}} (\boldsymbol {w}).
$$
There are several considerations for applying SAM to RL agents. When applying SAM to the Rainbow agent, the random noises from noisy layers embedded in the agent might hurt the SAM perturbation. However, we observe that the effect of regulating the noises is not significant. Moreover, there are options to apply SAM perturbation solely to the backbone or head. We find that applying SAM to the whole network is the most beneficial for enhancing generalization capability, yet we
observe that solely applying SAM to the backbone is quite similar to the case of applying it to the whole. On the other hand, when applying SAM to the SAC agent, there are multiple loss functions to be optimized, which share parameters. Again, applying SAM to both actor and critic is the most desirable option, yet we observe that the application to critic is more critical for performance improvement. More detailed ablation studies appear in Section B.4.
B.2 Deep Q-Learning with SAM and Noisy Layers
We used the Rainbow agent and its variants to learn discrete control problems of the Atari-100k benchmark. Although there are many other components inside the Rainbow agent, such as multi-step learning, distributional RL, andueling Deep Q-Network (DQN), it would be confusing to integrate all details together in the pseudocode. Thus, for simplicity of the display, we provide a pseudocode of applying SAM to vanilla DQN with noisy layers. See Algorithm 1.
Algorithm 1: Deep Q-Learning with Noisy Layers & SAM
1 Input: Learning rate  $\eta$ ; SAM parameter  $\rho$ ; Replay ratio  $R$ ; Discount factor  $\gamma$ ; Polyak averaging factor  $\tau$ ;  
2 Initialize: Replay memory  $\mathcal{D}$ ; Q-function weight  $\boldsymbol{w}$ ; Target weight  $\boldsymbol{w}^{-} \leftarrow \boldsymbol{w}$ ;  
3 foreach environment step  $t = 1, 2, \ldots$  do  
4 Collect a trajectory  $(s, a, r, s', d)$  with  $\varepsilon$ -greedy ( $d = \mathbb{1}[s' \text{ is terminal}]$ );  
5 Store the trajectory to  $\mathcal{D}$ ;  
6 If  $d = 1$ , Reset environment state;  
7 foreach optimization step  $i \in \{1, \ldots, R\}$  do  
8 Sample a minibatch of transitions  $\mathcal{B} = \{(s, a, r, s', d)\}$  from  $\mathcal{D}$ ;  
9 Compute TD target  $y(r, s', d) = r + \gamma (1 - d) \max_{a'} Q_{\boldsymbol{w}^-}(s', a')$ ;  
// Perturbation step  
10 Sample a random noise  $\xi$  of noisy layers inside Q-function;  $\boldsymbol{w} \leftarrow \boldsymbol{w} + \xi$ ;  
11 Compute gradient  $\boldsymbol{g}(\boldsymbol{w}) = \nabla_{\boldsymbol{w}} \frac{1}{|\mathcal{B}|} \sum_{(s, a, r, s', d) \in \mathcal{B}} (y(r, s', d) - Q_{\boldsymbol{w}}(s, a))^2$ ;  
12 Compute SAM perturbation  $\tilde{\boldsymbol{w}}^{\mathrm{SAM}} = \boldsymbol{w} + \frac{\rho}{\|\boldsymbol{g}(\boldsymbol{w})\|_2} \boldsymbol{g}(\boldsymbol{w})$ ;  
// Update step  
13 Sample a random noise  $\xi'$  of noisy layers inside Q-function;  $\tilde{\boldsymbol{w}}^{\mathrm{SAM}} \leftarrow \tilde{\boldsymbol{w}}^{\mathrm{SAM}} + \xi'$ ;  
14 Compute SAM gradient  $\boldsymbol{g}^{\mathrm{SAM}} = \boldsymbol{g}(\tilde{\boldsymbol{w}}^{\mathrm{SAM}})$ ;  
15 Gradient descent update  $\boldsymbol{w} \leftarrow \boldsymbol{w} - \eta g^{\mathrm{SAM}}$ ; // Could be modified as any gradient-based optimizer  
16 Update target weight  $\boldsymbol{w}^{-} \leftarrow \tau \boldsymbol{w}^{-} + (1 - \tau) \boldsymbol{w}$ ;
B.3 Soft Actor-Critic with SAM
We also provide a pseudocode for applying SAM to the SAC algorithm. See Algorithm 2.
Algorithm 2: Soft Actor-Critic with SAM
1 Input: Learning rates  $\eta_{Q},\eta_{\pi},\eta_{\alpha}$ ; SAM parameters  $\rho_{Q},\rho_{\pi},\rho_{\alpha}$ ; Replay ratio  $R$ ; discount factor  $\gamma$ ; Polyak averaging factor  $\tau$ ; minimum expected entropy  $\mathcal{H}$ ;
2 Initialize: Replay memory  $\mathcal{D}$ ; Q-function weights  $\pmb{w}_1, \pmb{w}_2$ ; Q-function weights  $\pmb{w}_1^-, \pmb{w}_2^-$ ; Policy weight  $\pmb{\theta}$ ; Entropy regularization coefficient  $\alpha$ ;
3 foreach environment step  $t = 1,2,\ldots$  do
4 | Observe state  $s$  and sample/execute an action  $a \sim \pi_{\theta}(|s|)$ ;
5 Observe reward  $r$  , next state  $s^\prime$  , and done signal  $d = \mathbb{1}[s^{\prime}$  is terminal];
6 Store the trajectory  $(s,a,r,s^{\prime},d)$  to  $\mathcal{D}$
7 If  $d = 1$ , Reset environment state;
8 | foreach optimization step  $i \in  \{ 1,\ldots ,R\}$  do
9 Sample a minibatch of transitions  $\mathcal{B} = \{(s,a,r,s',d)\}$  from  $\mathcal{D}$ ;
// Perturbation step
10 | Compute target
$$
y (r, s ^ {\prime}, d; \boldsymbol {\theta}) = r + \gamma (1 - d) \left(\min  _ {i = 1, 2} Q _ {\boldsymbol {w} _ {i} ^ {-}} \left(s ^ {\prime}, a ^ {\prime}\right) - \alpha \log \pi_ {\boldsymbol {\theta}} \left(a ^ {\prime} | s ^ {\prime}\right)\right) \quad \left(a ^ {\prime} \sim \pi_ {\boldsymbol {\theta}} (\cdot | s)\right);
$$
11 | Compute Q-function gradient:
$$
\boldsymbol {g} _ {Q, i} (\boldsymbol {w} _ {i}) = \nabla_ {\boldsymbol {w} _ {i}} \frac {1}{| \mathcal {B} |} \sum_ {(s, a, r, s ^ {\prime}, d) \in \mathcal {B}} (y (r, s ^ {\prime}, d; \boldsymbol {\theta}) - Q _ {\boldsymbol {w} _ {i}} (s, a)) ^ {2}; \quad (i = 1, 2)
$$
12 Compute policy gradient by sampling differentiable action  $\tilde{a}_{\pmb{\theta}}(s)\sim \pi_{\pmb{\theta}}(\cdot |s)$
$$
\pmb {g} _ {\pi} (\pmb {w}) = \nabla_ {\pmb {\theta}} \frac {1}{| \mathcal {B} |} \sum_ {s \in \mathcal {B}} \left(\min _ {i = 1, 2} Q _ {\pmb {w} _ {i}} (s, \tilde {a} _ {\pmb {\theta}} (s)) - \alpha \log \pi_ {\pmb {\theta}} (\tilde {a} _ {\pmb {\theta}} (s) | s)\right) ^ {2};
$$
13 Compute temperature gradient:  $\pmb{g}_{\mathrm{temp}}(\alpha) = \nabla_{\alpha}\frac{1}{|\pmb{B}|}\sum_{s\in \mathcal{B}}(-\alpha \log \pi_{\pmb{\theta}}(\tilde{a}_{\pmb{\theta}}(s)|s) - \alpha \mathcal{H})$
SAM-perturbations:
$$
\begin{array}{l} \tilde {\boldsymbol {w}} _ {i} ^ {\text {S A M}} = \boldsymbol {w} _ {i} + \frac {\rho_ {Q}}{\| \boldsymbol {g} _ {Q , i} (\boldsymbol {w} _ {i}) \| _ {2}} \boldsymbol {g} _ {Q, i} (\boldsymbol {w} _ {i}); \quad (i = 1, 2) \\ \tilde {\boldsymbol {\theta}} ^ {\text {S A M}} = \boldsymbol {\theta} + \frac {\rho_ {\pi}}{\| \boldsymbol {g} _ {\pi} (\boldsymbol {\theta}) \| _ {2}} \boldsymbol {g} _ {\pi} (\boldsymbol {\theta}); \\ \tilde {\alpha} ^ {\mathrm {S A M}} = \alpha + \frac {\rho_ {\alpha}}{\| \boldsymbol {g} _ {\mathrm {t e m p}} (\alpha) \| _ {2}} \boldsymbol {g} _ {\mathrm {t e m p}} (\alpha); \\ \end{array}
$$
15 | Compute target  $y(\mathbf{x})$
16 Compute SAM gradients:
$$
\boldsymbol {g} _ {Q, i} ^ {\text {S A M}} = \boldsymbol {g} _ {Q, i} (\tilde {\boldsymbol {w}} _ {i} ^ {\text {S A M}}), \quad \boldsymbol {g} _ {\pi} ^ {\text {S A M}} = \boldsymbol {g} _ {\pi} (\tilde {\boldsymbol {\theta}} ^ {\text {S A M}}), \quad \boldsymbol {g} _ {\text {t e m p}} ^ {\text {S A M}} = \boldsymbol {g} _ {\text {t e m p}} (\tilde {\alpha} ^ {\text {S A M}});
$$
17 Gradient descent updates:
$$
\boldsymbol {w} _ {i} \leftarrow \boldsymbol {w} _ {i} - \eta_ {Q} \boldsymbol {g} _ {Q, i} ^ {\text {S A M}}; \boldsymbol {\theta} \leftarrow \boldsymbol {\theta} - \eta_ {\pi} \boldsymbol {g} _ {\pi} ^ {\text {S A M}}; \alpha \leftarrow \alpha - \eta_ {\alpha} \boldsymbol {g} _ {\text {t e m p}} ^ {\text {S A M}};
$$
// Could be modified as any gradient-based optimizer
Update target weight  $\pmb{w}_i^{-} \gets \tau \pmb{w}_i^{-} + (1 - \tau) \pmb{w}_i$ ;
B.4 Ablations on the usage of SAM
We conduct some ablation studies aiming to answer the following questions:
1. How should we deal with noisy layers during training?
2. What if we solely apply SAM perturbation to the backbone or head of Rainbow agent?
3. What if we solely apply SAM perturbation to the actor or critic of SAC agent?
The first two questions are answered by experiments with the DrQ algorithm tested on the Atari-100k benchmark, whereas for the last question we test on the DMC-M benchmark with the SAC agent.
(a) Noisy layers on/off.
(b) Backbone/Head SAM.
(c) Actor/Critic SAM.
Figure 5: Ablation study on utilizing SAM.
Noisy Layers. For the case of the Rainbow agent, we compare three different schemes of SAM updates regarding random noises from noisy layers: (i) perturbation with independent noise, using independently sampled (e.g., different) noise between perturbation and update step (default setting); (ii) update with reused noise, turning the noise on at the perturbation step and reusing that noise at the update step; (iii) noise-less perturbation, turning the noise off at the perturbation step and turning it on at the update step. Despite the variations, the overall impact on the agent's performance was relatively consistent. Note that option (i) is the most naive approach which can be implemented by simply adding a SAM optimizer code block into any RL implementation with noisy layers.
Backbone-SAM v.s. Head-SAM. We experiment with resectioning the parts to be updated with SAM in the RL agent architecture. We temporarily turn off the gradient of either backbone or head at the perturbation step and then turn on that gradient back at the update step. The result of the experiment, presented in Figure 5b, says that updating with backbone-only perturbation is more beneficial than using head-only perturbation, although applying both is much better. Indeed, SAM perturbation of the head alone hurts the performance without any other SAM perturbation or resetting.
We want to explain why these happen from the point of view of the number of active neurons in the backbone/head. Interestingly, we observe that the SAM perturbation of backbone parameters induces more active units in the head, whereas the SAM perturbation of head parameters results in less active units in the backbone. Recall that both tendencies are observed when we apply SAM perturbation to both the backbone and head at once. From this, we can say that the rise of active units in the head by applying SAM is due to the SAM perturbation of the backbone, whereas the reason for the backbone getting sparser is the SAM perturbation of the head.
Actor-SAM v.s. Critic-SAM. SAC is an actor-critic algorithm that employs three distinct sets of updating parameters, namely the actor, critic, and alpha. Since alpha has just one parameter, we focus on investigating the impact of SAM on actor and critic parameters. We conduct an ablation study utilizing the identical configuration as the DMC experiment presented in Section 4.2. Figure 5c demonstrates that exclusively applying SAM to critic parameters is more critical than solely applying SAM to actor parameters while using both is much better.
In the context of SAC, the actor and critic networks are constructed to jointly utilize the backbone layer. In order to ensure stable learning, updates to the backbone layer are exclusively driven by critic losses, while actor losses do not influence the backbone layer [73, 72]. Consequently, if SAM is exclusively applied to the actor parameters, the backbone layer remains unaffected by SAM as
there is no gradient propagation. As a result, the performance of the SAM solely used on the actor parameters, which has no impact on the backbone, is inferior to other approaches incorporating SAM, which corroborates the aforementioned findings.
(a) SAM parameter  $\rho$
Figure 6: More ablation studies on using SAM.
(b) Batch sizes.
SAM parameter  $(\rho)$ . Being the main hyperparameter of SAM, the  $\rho > 0$  represents the intensity of SAM perturbation. In every other experiment with DER and  $\mathrm{DrQ}(-\mathrm{Rainbow})$ , we do not tune the value of  $\rho$  but fix them as 0.1. Here, however, we run two other values of  $\rho$ 's, 0.03 and 0.3, to examine the effect of  $\rho$ . The result is shown in Figure 6a. We definitely observe the trade-off due to the size of  $\rho$ . If the  $\rho$  is too small, the scale of SAM perturbation becomes negligible and it cannot drastically improve the generalization capability compared to the case without SAM. However, a larger value of  $\rho$  is not always beneficial; the generalization capability does not always scale with the size of SAM perturbation. We, however, claim that finding an appropriate value of  $\rho$  is not too difficult; our recommendation is to test a few values between 0.01 and 1.
Batch size. We further study robustness with respect to batch sizes. We investigate the batch sizes among 16, 32 (baseline), and 64. From the resulting plot Figure 6b, we observe that SAM consistently improves the performance regardless of the choice of batch size, by comparing solid lines (SAM+Adam) and dashed lines (Adam). We also observe that the performance improvement between reset and non-reset becomes much larger when SAM is applied.
C Per Environment Results
C.1 Atari-100k
C.2 DMC-M
Table 7: Mean trajectory scores on DrQ algorithm. We report the mean trajectory scores on the 26 Atari games, evaluated on top of DrQ. For each random seed, the results are averaged over 100 different trajectories at the end of training. The results are then averaged over 10 random seeds.
Game	Random	Human	Base	SAM	LayerNorm	CReLU	Reset	PLASTIC
Alien	227.8	7127.7	757.62	854.84	899.14	1011.84	926.73	1032.0
Amidar	5.8	1719.5	164.12	187.76	176.12	146.69	158.22	201.6
Assault	222.4	742.0	582.59	597.9	566.26	585.57	694.17	888.5
Asterix	210.0	8503.3	823.3	780.6	889.3	887.8	921.15	1066.0
BankHeist	14.2	753.1	48.36	185.25	56.16	79.88	141.14	161.2
BattleZone	2360.0	37187.5	4088.0	13999.0	14416.0	11306.0	4700.0	2099.0
Boxing	0.1	12.1	11.56	16.43	21.2	18.75	44.14	44.5
Breakout	1.7	30.5	13.16	12.64	13.71	10.56	18.46	21.0
ChopperCommand	811.0	7387.8	778.0	977.7	879.2	1150.6	777.5	891.2
CrazyClimber	10780.5	35829.4	13182.2	17927.7	15852.8	13957.6	19082.8	31223.8
DemonAttack	152.1	1971.0	701.38	735.54	556.33	531.05	1074.29	2117.8
Freeway	0.0	29.6	22.32	27.79	30.49	24.54	26.92	27.1
Frostbite	65.2	4334.7	1856.64	2052.95	1409.34	763.64	1257.33	1802.3
Gopher	257.6	2412.5	374.28	476.02	433.92	445.88	711.14	839.4
Hero	1027.0	30826.4	5096.67	6496.56	5645.98	6332.89	6987.43	7007.2
Jamesbond	29.0	302.8	348.8	333.3	287.2	239.7	369.2	461.1
Kangaroo	52.0	3035.0	3773.2	3453.8	3589.0	4919.8	2830.8	1636.1
Krull	1598.0	2665.5	3612.06	3385.05	3483.26	3710.24	4379.46	5019.5
KungFuMaster	258.5	22736.3	20412.4	16380.3	8646.2	15188.4	12737.4	16105.0
MsPacman	307.3	6951.6	1210.02	1471.33	1226.5	1191.2	1362.05	1245.6
Pong	-20.7	14.6	-8.82	-6.42	-12.85	1.96	-10.0	-17.7
Qbert	11.5	7845.0	3776.3	3495.45	2648.4	2429.2	3492.18	3986.3
RoadRunner	68.4	42054.7	15696.6	11919.5	13045.6	13703.4	12939.5	15073.8
Seaquest	24.9	69571.3	525.88	519.2	394.24	383.8	645.26	635.9
PrivateEye	163.9	13455.0	100.0	96.04	100.0	99.4	100.13	100.0
UpNDown	533.4	11693.2	3690.74	4963.46	5596.2	4565.04	15463.98	66473.0
IQM	0.0	1.0	0.258	0.325	0.259	0.256	0.343	0.421
Median	0.0	1.0	0.277	0.327	0.247	0.193	0.291	0.347
Mean	0.0	1.0	0.476	0.501	0.463	0.498	0.660	0.933
OG	1.0	0.0	0.633	0.589	0.627	0.628	0.579	0.535
Table 8: Mean trajectory scores on State-of-the-art methods. We report the individual scores on the 26 Atari games. For IRIS, DreamerV3, EfficientZero, PlayVirtual, MLR, SR-SPR, and PLASTIC the results are averaged over 5, 5, 32, 15, 3, 10, and 5 seeds respectively.
Game	EfficientZero	IRIS	DreamerV3	PlayVirtual	MLR	SR-SPR:4	SR-SPR:8	PLASTIC†:8	PLASTIC:8
Alien	808.5	420.0	1095.8	947.8	990.1	964.4	1015.5	1021.7	1368.6
Amidar	148.6	143.0	142.9	165.3	227.7	211.8	203.1	186.2	220.2
Assault	1263.1	1524.4	638.0	702.3	643.7	987.3	1069.5	821.5	1074.2
Asterix	25557.8	853.6	982.8	933.3	883.7	894.2	916.5	1184.7	1188.0
BankHeist	351.0	53.1	617.8	245.9	180.3	460.0	472.3	370.1	313.8
BattleZone	13871.2	13074.0	12800.0	13260.0	16080.0	17800.6	19398.4	15728.0	16868.0
Boxing	52.7	70.1	67.8	38.3	26.4	42.0	46.7	45.9	41.8
Breakout	414.1	83.7	18.9	20.6	16.8	26.1	28.8	21.9	20.9
ChopperCommand	1117.3	1565.0	400.0	922.4	910.7	1933.7	2201.0	799.4	871.2
CrazyClimber	83940.2	59324.2	71620.0	23176.7	24633.3	38341.7	43122.3	31652.6	47893.0
DemonAttack	13003.9	2034.4	545.0	1131.7	854.6	3016.2	2898.1	1562.9	2460.9
Freeway	21.8	31.1	0.0	16.1	30.2	24.5	24.9	29.4	30.4
Frostbite	296.3	259.1	1108.4	1984.7	2381.1	1809.9	1752.8	2152.5	2195.4
Gopher	3260.3	2236.1	5828.6	684.3	822.3	717.5	711.2	582.7	612.6
Hero	9315.9	7037.4	10964.6	8597.5	7919.3	7195.7	7679.6	11195.5	9000.7
Jamesbond	517.0	462.7	510.0	394.7	423.2	408.8	392.8	401.3	428.8
Kangaroo	724.1	838.2	3550.0	2384.7	8516.0	2024.1	3254.9	6218.2	2249.2
Krull	5663.3	6616.4	8012.0	3880.7	3923.1	5364.3	5824.8	5201.9	5647.6
KungFuMaster	30944.8	21759.8	29420.0	14259.0	10652.0	17656.5	17095.6	20839.6	19546.2
MsPacman	1281.2	999.1	1388.5	1335.4	1481.3	1544.7	1522.6	1662.0	1292.4
Pong	20.1	14.6	18.5	-3.0	4.9	-5.5	-3.0	0.3	-3.5
Qbert	13781.9	745.7	3117.9	3620.1	3410.4	3699.8	3850.6	4372.8	4967.2
RoadRunner	17751.3	9614.6	14036.6	13429.4	12049.7	14287.3	13623.5	16254.0	20709.0
Seaquest	1100.2	661.3	582.0	532.9	628.3	766.6	800.5	574.7	859.1
PrivateEye	96.7	100.0	1124.0	93.9	100.0	95.8	95.8	100.0	100.0
UpNDown	17264.2	3546.2	9234.0	10225.2	6675.7	91435.2	95501.1	34342.4	33203.3
IQM	n/a	0.501	0.497	0.374	0.432	0.544	0.589	0.583	0.571
Median	0.227	0.289	0.466	n/a	n/a	0.523	0.560	0.542	0.494
Mean	0.562	1.046	1.097	n/a	n/a	1.111	1.188	0.939	0.968
OG	n/a	0.512	0.505	0.558	0.522	0.470	0.452	0.448	0.461
Table 9: Mean trajectory scores for each value of replay ratio (2, 4, 8). We report the individual scores on the 26 Atari games. Here, the results are averaged over 5 random seeds.
Game	PLASTIC†:2	PLASTIC:2	PLASTIC†:4	PLASTIC:4	PLASTIC†:8	PLASTIC:8
Alien	1063.1	1032.0	1251.2	1138.3	1021.7	1368.6
Amidar	195.2	201.6	151.4	206.1	186.2	220.2
Assault	763.6	888.5	767.2	939.7	821.5	1074.2
Asterix	1007.6	1066.0	1098.1	1058.0	1184.7	1188.0
BankHeist	217.7	161.2	328.1	268.0	370.1	313.8
BattleZone	17767.0	2099.0	17762.0	9292.0	15728.0	16868.0
Boxing	39.1	44.5	53.6	46.4	45.9	41.8
Breakout	16.4	21.0	22.4	27.9	21.9	20.9
ChopperCommand	972.4	891.2	798.0	875.4	799.4	871.2
CrazyClimber	21967.5	31223.8	24615.0	42557.6	31652.6	47893.0
DemonAttack	1016.2	2117.8	1630.6	2402.2	1562.9	2460.9
Freeway	30.1	27.1	29.6	29.6	29.4	30.4
Frostbite	1582.0	1802.3	2119.4	1899.3	2152.5	2195.4
Gopher	616.4	839.4	635.2	811.6	582.7	612.6
Hero	6582.8	7007.2	10350.0	8118.2	11195.5	9000.7
Jamesbond	397.5	461.1	383.7	438.7	401.3	428.8
Kangaroo	2836.3	1636.1	6860.8	2386.6	6218.2	2249.2
Krull	4485.6	5019.5	4719.8	5266.4	5201.9	5647.6
KungFuMaster	11873.9	16105.0	17613.4	19828.6	20839.6	19546.2
MsPacman	1399.6	1245.6	1401.9	1457.2	1662.0	1292.4
Pong	-6.3	-17.7	0.1	-7.1	0.3	-3.5
Qbert	4006.1	3986.3	3794.1	4736.4	4372.8	4967.2
RoadRunner	15883.0	15073.8	17704.2	19487.8	16254.0	20709.0
Sequester	576.1	635.9	692.8	537.0	574.7	859.1
PrivateEye	46.6	100.0	94.1	100.0	100.0	100.0
UpNDown	15470.0	66473.0	11398.0	51764.5	34342.4	33203.3
IQM	0.396	0.421	0.518	0.545	0.583	0.571
Median	0.425	0.347	0.517	0.407	0.542	0.494
Mean	0.702	0.933	0.858	1.002	0.968	0.939
OG	0.541	0.535	0.478	0.475	0.448	0.461
Table 10: Mean trajectory scores on DrQ algorithm. We report the mean trajectory scores on the DMC medium environments, evaluated on top of DrQ. For each random seed, the results are averaged over 10 trajectories at the end of training. The results are then averaged over 10 random seeds.
Game	Base	SAM	LayerNorm	CReLU	Reset	PLASTIC
acrobot-swingup	16.34	15.36	23.45	40.79	57.83	68.24
cartpole-swingup_sparse	79.03	197.91	164.57	321.83	740.0	691.16
cheetah-run	727.44	736.61	591.47	835.33	665.67	708.04
finger-turneasy	207.02	129.92	325.21	221.0	232.76	379.69
finger-turn_hard	80.95	60.07	485.70	117.5	89.11	546.56
hopper-hop	284.78	255.84	277.48	293.52	284.69	254.62
quadruped-run	155.06	261.03	266.84	78.89	466.93	426.57
quadruped-walk	88.08	130.85	173.13	80.02	632.33	528.12
reacher-easy	505.08	738.80	926.94	914.3	957.96	879.71
reacher-hard	450.96	495.32	746.55	795.77	794.53	798.10
walker-run	632.93	639.94	652.41	643.24	564.95	632.55
IQM	213	278	412	338	514	565
Median	288	341	415	398	491	540
Mean	293	332	421	394	498	537
Our research possesses broader impacts in two different aspects. The first impact lies in the sample efficiency of Reinforcement Learning (RL) algorithms. We incorporate principles of preventing the loss of plasticity, leading to more efficient and adaptive solutions. This advance can improve RL's sample efficiency, opening new avenues for future research in this domain.
The second impact is on inclusivity and accessibility in RL. By decreasing data and computational demands, this work can aid underprivileged communities with fewer resources to participate in RL research and benefit from its many applications. This approach encourages a diverse range of perspectives and experiences in the field, enriching the community.
However, while our research has these positive implications, it is important to also acknowledge potential risks associated with RL technologies, particularly in robotics. We must continue to uphold ethical standards and prioritize safety to prevent misuse or harm that could arise from these advancements.