SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Section 5.3, we considered three alternative ways of dynamically addressing plasticity loss during training: resets, Shrink-and-Perturb (SnP), and naive width scaling. Resets re-initialize parameters of the last layers (using our notation, it corresponds to replacing  $h_{\theta}(\cdot)$  with  $h_{\theta_1^{\prime}}(\cdot))$ ) and rely on a replay buffer to transfer knowledge before and after the intervention. Resets require the specification of the number of last layers and the application timestep. We ran a sweep over [1, 2] layers and two choices of timesteps: either once at 50M frames or trice at 50M, 100M, and 150M. Afterwards, we reported the results that attain the highest IQM score.
Figure 11: A demonstration of the Assault game evolution when a high-performing agent found on the Internet reaches a score of around 2800: before, the agent had to shoot only upwards; afterwards, it has to shoot up, left, and right. We interpret that the failure to improve upon the 2800 score is explained by exploration.
Shrink-and-Perturb modify all network weights  $w$  as  $w \gets \lambda w + \sigma \epsilon$  at the given application timesteps, where  $\epsilon$  is a random vector with the same dimensionality as  $w$  sampled from the standard Gaussian distribution. SnP has three hyperparameters: the shrink coefficient  $\lambda$ , the noise scale  $\sigma$ , and the application timesteps. We performed a grid search over  $\lambda$  in [0.1, 0.3, 1],  $\sigma$  in [0.01, 0.1, 1], and the same choices of timesteps as for resets.
The best hyperparameters ended up being the ones that somewhat minimized the effect of both resets (1 layer, 1 application time) and SnP ( $\lambda = 1$ ,  $\sigma = 0.01$ , 3 application times); other hyperparameters resulted in even worse performance. The paper on resets [Nikishin et al., 2022] demonstrates results on the Atari 100k benchmark [Kaiser et al., 2019] that focuses on a data-efficient regime with  $10^{5}$  interactions only and contains a subset of 26 (out of 57) games. In this setting, the replay buffer has all experiences encountered during the agent's lifetime; this data can be sufficient for recovering the performance after a reset. In the Atari 200M setting though, the replay buffer has only 4M frames which might not be enough to recover fast after a reset. We speculate that similar reasoning applies to SnP since it can be seen as a soft version of resets [D'Oro et al., 2023].
For the width scaling method, we modify the last two layers by doubling their width. In detail, suppose the weight matrices are  $W_{1} \in \mathbb{R}^{N \times K}$  and  $W_{2} \in \mathbb{R}^{K \times |\mathcal{A}|}$ , where  $|\mathcal{A}|$  is the action space dimensionality. We create two new matrices  $W_{1}' \in \mathbb{R}^{N \times 2K}$  and  $W_{2}' \in \mathbb{R}^{2K \times |\mathcal{A}|}$  and fill the first  $K$  columns of  $W_{1}'$  with values of  $W_{1}$  and the first  $K$  rows of  $W_{2}'$  with values of  $W_{2}$ . The remaining entries are sampled from the random initializer. We perform a modification to the bias term  $b_{1}' \in \mathbb{R}^{2K}$  by copying values from  $b_{1} \in \mathbb{R}^{K}$  and setting the rest to zero. The width is scaled once at 50M.
Such a naive approach increases plasticity but its inability to improve over the standard Double DQN might be caused by adverse effects on the predictions after the intervention without output correction.
D The Assault Game Analysis
We searched for a high-scoring behavior demonstration in the Assault environment on YouTube<sup>3</sup>. The screenshots in Figure 11 demonstrate the change of the environment around the score of 2800: before, the enemies were appearing only above the controlled starship, while afterwards, they start to appear from the left and from the right. Before the transition, the algorithm learned that actions "shoot left" and "shoot right" were irrelevant, while afterwards, it has to start using these actions, suggesting that the performance plateau can be attributed to exploration challenges.
We highlight that it was the suggested protocol for diagnosis that led to the insight: after seeing that the post-injection agent has the same performance plateau as the baseline, we decided to investigate the behavior in the game and realized that previously irrelevant actions became critical.
3https://youtu.be/HwWJrb2PQQ0