Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps
Authors: Benjamin Ellis
Matthew Thomas Jackson
Andrei Lupu
Alexander D. Goldie
Mattie Fellows
Shimon Whiteson
Jakob Nicolaus Foerster
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Deep Learning
Primary area: reinforcement_learning

Abstract: In reinforcement learning (RL), it is common to apply techniques used broadly in 
machine learning such as neural network function approximators and momentum-based optimizers. However, such tools were largely developed for supervised learning rather than nonstationary RL, leading practitioners to adopt target networks, clipped policy updates, and other RL-specific implementation tricks to combat this mismatch, rather than directly adapting this toolchain for use in RL. 
In this paper, we take a different approach and instead address the effect of nonstationarity by adapting the widely used Adam optimiser. 
We first analyse the impact of nonstationary gradient magnitude --- such as that caused by a change in target network --- on Adam's update size, demonstrating that such a change can lead to large updates and hence sub-optimal performance.
To address this, we introduce Adam-Rel.
Rather than using the global timestep in the Adam update, Adam-Rel uses the *local* timestep within an epoch, essentially resetting Adam's timestep to 0 after target changes.
We demonstrate that this avoids large updates and reduces to learning rate annealing in the absence of such increases in gradient magnitude. Evaluating Adam-Rel in both on-policy and off-policy RL, we demonstrate improved performance in both Atari and Craftax.
We then show that increases in gradient norm occur in RL in practice, and examine the differences between our 
theoretical model and the observed data.

Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps
Benjamin Ellis University of Oxford
Matthew T. Jackson University of Oxford
Andrei Lupu University of Oxford
Alexander D. Goldie University of Oxford
Mattie Fellows University of Oxford
Shimon Whiteson
Jakob N. Foerster
University Oxford
University of Oxford
Abstract
In reinforcement learning (RL), is common to apply techniques used broadly machine learning such as neural network function approximators and momentum- based optimizers [ However, such tools were largely developed for super- vised learning rather than nonstationary RL, leading practitioners to adopt target networks [3], clipped policy updates [4]. and other RL-specific implementation tricks (5) 6 to combat this mismatch, rather than directly adapting this toolchain for use in RL. In this paper, we take a different approach and instead address the effect of nonstationarity by adapting the widely used Adam optimiser (7). We first analyse the impact of nonstationary gradient magnitude- such as that caused by change in target network
Adam's update size, demonstrating that such a change can lead to large updates and hence sub-optimal performance. To address this, we introduce Adam with Relative Timesteps, or Adam-Rel. Rather than using the global timestep in the Adam update, Adam-Rel uses the local timestep within an epoch, essentially resetting Adam's timestep demonstrate that this avoids large updates and reduces to learning rate annealing in the absence of such increases in gradient magnitude. Evaluating Adam-Rel in both on-policy and off-policy RL, we demonstrate improved performance in both Atari and Craftax. We then show that increases gradient norm occur in RL in practice, and examine the differences between our theoretical model and the observed data.
after target changes, We
Introduction
Reinforcement Learning (RL) aims learn robust policies from agent's experience. This has the potential for large scale real-world impact in areas such as autonomous driving or improving logistic chains. Over the last decade, number of breakthroughs in supervised learning-such as convolutional neural networks and the Adam optimizer-have expanded the deep learning toolchain and been transferred to RL, enabling
begin fulfilling this potential.
However, since RL agents are continuously learning from new data they collect under their changing policy, the optimisation objective is fundamentally nonstationary. Furthermore, temporal difference (TD) approaches bootstrap the agent's update from its own value predictions, exacerbating the nonstationarity in the objective function. This is in stark contrast to the stationary supervised learning setting for which the deep learning toolchain was originally developed. Therefore, to apply these tools successfully, researchers have developed a variety of implementation tricks on top of this base to stabilise training 8 6 5) This has resulted proliferation of litte--documentd design choices that are vital for performance, contributing the reproducibility crisis in RL 9
Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We believe that in the long term, a more robust approach is to angment this toolchain for RL, rather than building on top it. To this end, in this paper we examine the interaction between nonstationarity and the Adam optimizer 77. Adam's update rule, where equations are applied element-wise (i.e. per parameter), is defined by
m me
m = 3pm=1 + (1 31)gt.
(1 Bi ')
0, = -20,-1 + (1- -=))
61 =
1t
(1 B2
m
04
0: = 81-1 ant4.
Voc +
Here, St is the gradient, 0₂ a parameter to be optimized, and a the learning rate. The resulting update is the ratio of two different momentum terms: one for the first moment, my, and one for second moment, 2tz of the gradient. These terms use different exponential decay coefficients, 31 and 82. Under stationary gradients, the (1 - 3.) weighting ensures that, in the limit, the overall magnitude of the two momenta is independent of the value chosen for each of the coefficients. However, since both momentum estimates are initialised to 0, they must be renormalised for a given (finite) timestep 4, to account for the "missing parts" of the geometric series 77) resulting in ūn and mh.
Crucially, counts the update steps since the beginning of training and thus bakes in the assumption of stationarity that is common in supervised learning. In particular, this renormalisation breaks down if the loss is nonstationary. Consider a task change late in training, which results in gradients orders of magnitudes higher than those of the prior (near convergence) task. Clearly, this is analogous to the situation at the beginning of training where all momentum estimates are O. However, the t parameter, and therefore the renormalisation, does not account for this.
In this paper, we demonstrate that changes in the gradient scale can lead to large updates that persist over long horizon. Previous work [T0 I1 has suggested that old momentum estimates can contaminate an agent's update and propose resetting the entire optimizer state when the target changes as a solution. However, by discarding previous momentum estimates, we hypothesise that this approach needlessly sacrifices valuable information for optimization. Instead, we propose retaining momentum estimates and only resetting t, which we refer to as Adam-Rel. In the limit of gradient sparseness,
show that the Adam-Rel update size remains bounded, converging to 1 in the limit of a large gradient, unlike Adam. Furthermore, if such gradient magnitude increases do not occur, Adam-Ra Rel reduces to learning rate annealing, a common method for stabilising optimization.
When evaluated against the original Adam and Adam with total resets, we demonstrate that our method improves PPO's performance in Craftax-Classic [12 and the Atari-57 challenge from the Arcade Learning Environment [13]. Additionally, we demonstrate improved performance in the off-policy setting by evaluating DQN on the Atari-10 suite of tasks [14]. We then examine the gradients in practice and show that there are significant increases in gradient magnitude following changes in the objective. Finally, we examine the discrepancies between our theoretical model and observed gradients to better understand the effectiveness of Adam-Red Rel.
2 Background
2.1 Reinforcement Learning
Definition Reinforcement learning agents learn a policy in a Markov Decision Process 15 MDP],
a tuple M = -S, A. T, R, 7) where S is the set of states, AA is the set of actions, T: x A + P(S) is the transition function, R : S A IR is the reward function and 7 is the discount factor. At each timestep i, the agent observes a state 8, E S and takes an action a, drawn from (-((s)) before transitioning new state 8t-1 S and receiving reward r. drawn from R(se,a)) The goal of the agent is to maximise the expected discounted return E,,T 12re rr))
Nonstationarity in RL In contrast with supervised learning, where a single stationary objective is typically optimised, reinforcement learning is inherently nonstationary. Updates to the policy induce changes not only the distribution of observations seen at a given timestep, but also the return distribution, and hence value function being optimised. This arises regardless of how these updates

---

are performed. However, one particular reason for nonstationarity in RL is the use of bootstrapped value estimates via TD learning [15], which optimises the below objective
(() = [sg {rr: + 7VO" (st+1)} VV" (st))²,
where sg is the stop-grand nop-gradient operator. In this update, the target r. + 7 Vo (8c+1) depends on the parameters 0 and therefore changes as these are updated.
These target changes can either be more gradual, in the case of continuous updates the value function in TD learning, or more abrupt, as in the case of the use of target networks in DQN.
Sequentially Optimized Stationary Objectives In this work, we focus abrupt objective changes; changes of objectives that do not involve a smoothing method such as Polyak averaging M. and the resulting sudden change of supervised learning problem. More explicitly, we consider optimising a stationary loss function L(0, 6), where 0 are the parameters to be optimised and Φ is the other parameters of the loss function (such the parameters of a value network), which are not updated throughout optimisation, but does not include the training data.
We consider setting where at certain timestep t in our training. we transition from optimising L(O,, Φ1) to optimising L(0 Q2) for some ø1. 62. Such individual objectives are still non- stationary. For example, significant changes in the policy would induce changes in the data dis- tribution, which would then affect the underlying loss landscape, but we do not consider such non-stationarity in this work.
This setting is very common throughout RL. Bootstrapped value estimates are the most common cause of this, but it also occurs in PPO's s actor update, where each new rollout induces different supervised learning problem due to the actor and critic updates. This is optimised for a fixed number of updates before collecting new data.
We refer to these sequences of supervised learning problems as sgquentially-optiiiss stationary objectives. In this work, we use this framing to propose an approach that is consistent throughout each stationary period of optimization and applies corrections to make optimization techniques valid when nonstationarity is introduced via objective changes. Bengio et al. [1] propose the gradient contamination hypothesis, which states that current optimizer momentum estimates can point in the opposite direction to the gradient following change in objective, thereby hindering optimization. A previous approach to this problem is that of Asadi et al. [10], where they propose resetting Adam's ' momentum estimates and timestep to 0 throughout training. We refer to this method as Adam-MR. Finally, Dohare et al. [66 propose setting the Adam hyperparameters to equal values, such that 31 B2, suggesting that this can help avoid performance collapse.
Proximal Policy Optimization Proximal Policy Optimization 4 PPOJ is policy optimisation based RL method. It uses learned critic VT trained by TD loss to estimate the value function, and clipped actor update of the form
min [clip (r(e.o): 1 + (81, ((e,)) At (se, ax)]
where the policy ratio T(o,e) = aa(arl is the ratio of the stochastic policy to optimise सo and म, [()](ac)
the previous policy. A is the advantage, which is typically estimated using generalised advantage estimation [1]. Clipping the policy ratio aims to avoid performance collapse by preventing policy updates larger than e.
Optimisation of the PPO objective proceeds by first rolling out the policy to collect data, and then iterating over this data in a sequence of epochs. Each of these epochs splits the collected data into a sequence of mini mini-hatchss, over which the above update is calculated.
2.2 Momentum-Basee Optimization
2 is method for enhancing stochastic gradient descent by accumulating gradients in the direction of repeated improvement. The typical formulation of momentum for each element i is
Momentum
mi = 13mm + gi.
0: = 0 amm
3

---

where B is the momentum coefficient, Ir E IR" is the gradient at the current step, my E R" is the gradient incorporating momentum, the scalar learning rate and 0 I are the parameters to be optimised. With momentum, update directions with low curvature have their contribution to the gradient amplified, considerably reducing the number of steps required for convergence.
In the introduction, we described the update equations for Adam [7]. the most popular optimizer that uses momentum. Adam's 's update is designed to keep its updates within a trust region, which depends on a learning rate a.
3 Nonstationary Optimization with Adam
We now investigate the effect of nonstationarity Adam by analysing its update rule after sudden change in gradient. As a simplified model of gradient instability, we assume optimization with Adam starts at timestep t = -t' with constant gradient g'_ g. 0 < g < oo until timestep 0. Following t = O, we model instability by increasing the gradient by a factor of k. as might occur in a nonstationary optimization setting. This gives
gl
9, kgg,
-1'
120.
For larger values E', the short term effects of Adam's initialisation on the momentum terms dissipate and my and é, converge to stable values. By taking the limit of t' - oc, investigate the effect of sudden change in gradient 9; on the update size u, after a long period training. This allows for any effects from the initialisation of momentum terms m and o_ . dissipate:
Theorem 3.1. Assume that O. Let gf be defined as in Equation (2) and nn' v.a and und--'er v be the
momentum terms at timestep L given Adam starts at timestep -l. It Jollows that:
mi
31 31 + 1(1 31 (1)
lim u lim 1-000
(3)
1-000
Voes
B22 + h2 82 11)
Proof. See Appendix A
For large k. Theorem 3.1 proves that the element-wise momentum term after the change in gradient
0 approximately 1-8 8, For the most commonly used values of 31 0.9 and B3 0.999,
VI-22"
this is v10, which is much larger than the intended unit update which Adam is designed to maintain. The top plot in Figure [ which shows the Adam update size against t for different values of k demonstrates that the update peaks significantly higher than the desired 1 before slowly converging back to 1.
4 Adam with Relative Timesteps
To fix the problems analysed in the previous section, introduce Adam-Rel. At the start of each new supervised learning problem, Adam-Rel resets Adam's parameter to 0, rather than incrementing from its previous value. This one-line change is illustrated for PPO in Algorithm T
At the start of training, both momentum terms in Adam are Therefore, at the first timestep, when the first gradient is encountered, the magnitude of the gradient is infinite relative the current momentum estimate. As explained in Section this induces large update. However, dividing the momentum estimates by (1 3, and (1 32" ') fixes this issue by correcting for this sparsity, Therefore, by resetting to 0, Adam handles changes in gradient magnitude resulting from the change of supervised learning problem.
If we examine the same update as in the previous section adjusted by Adam-Red Rell, assuming that we reset Adam's 's & just before the gradient scales to kg, we find it comes to
mi mi
V1
32 FII
B ++
+ kk(1 - 1+1
lim 10000
(4)
31
32
41
+ +k²(1 - 32 1

---

Algorithm 1 Pseudocode for PPO with Adam, Adam- Rel, and Adam-MR.
Adam
10a
Update Size
mg Vo 0,
Initialise Adam state Initialise number of updates to O
0 for = 1 K do
Rollout policy no, to collect batch D
-10
0
40
80
100
f = m- 0, 0j 0
Adam-Rel
for epoch to E do
for mini-batch B in D do
1.0
10:
Vo,_,
PPO (0)- 1) + L LT (8)-1))
0.8
j j j+1
t = t+1
mj 81mj
(1 B1)095
0.2
10a
1j 32
32)99
20
40
100
Mj = (T-M)
6 [1 [1-B2))
Figure 1: Update size of Adam and Adam- Rel versus when considering nonstationary gradients. Assumes that optimization starts at time -t', which is large, and that the gradi- ents up until time are and then there is increase in the gradient to kg.
0 0 totc
12
end for end for end for
As - oo, this tends to 1. This means that Adam-Rel ensures approximately unit update size in the case of a large increase in magnitude in the gradient, at the expense of potentially smaller update at the point reset. Figure shows the update size of Adam- Rel as - increases. The update size is smaller at the start, but never reaches significantly above 1.
However, the above analysis does not show how Adam and Adam-Rel differ in practice, where large changes in gradient magnitude may not occur. Examining the bottom of Figure [ we can see that for lower values of k. Adam-Rel rapidly decays the update size before increasing it. Functionally, this behaves like a learning rate schedule. Over a short horizon (e.g.. 16 steps is common in PPO), this effect is similar to learning rate annealing, whilst over a longer horizon (e.g., approximately 1000 steps in DQN) it is akin to learning rate warmup, both of which are popular techniques in optimising stationary objectives. Therefore, the benefits of Adam-Rec Rel are twofold: first, it guards against large increases in gradient magnitude by capping the size of potential updates, and secondly, if such large gradient increases do not occur, it reduces to form of learning rate annealing, which is commonly employed in optimising stationary objectives.
5
Experiments
5.1 Experimental setup
To evaluate Adam-Rel, we explore its impact DQN and PPO, two of the most popular algorithms in off-policy and on-policy RL respectively.
To do so, first train DQN [8 [9]
agents with Adam-Rel the Atari-10 benchmark for 40M
frames, evaluating performance against agents trained with Adam and Adam-MR. We then extensively evaluate method's 's impact on PPO [4] 19 20), training agents
Craftax-Classic- IB [12
JAX-based reimplementation of Crafter 21 where the agent is allocated billion environment interactions and the Atari-57 suite [13 for 40 million frames. In doing so, benchmarks respectively evaluate the performance of Adam-Rel on exceedingly long training horizons and its
"We exclude out of the 57 games, Montezuma's Revenge and Venture, after observing that all algorithms achieve Homan-normalizee score of O.
5

---

PPO Atari-57 IQM
DQN Atari-10 Regressed Median
Adam (B1 = 82) Adam-MR Adam-Rel Adam
Adam-MR Adam-Rel
Adam
0.25
0.50
0.75
1.00
0.2
0.4
0.6
Human Normalized Score
Human Normalized Score
Figure 2: Performance of Adam-Rel. Adam, Adam-MR, and Adam (3) = (22) for PPO and Adam, Adam-MR and Adam-Rel for DQN on Atari-57 and Atari- 10 respectively. Atari-10 uses subset of Atari tasks to estimate median performance across the whole suite. Details can be found in [14 Error bars are 95% stratified bootstrapped confidence intervals. Results are across 10 seeds except for Adam (31 = 52), which is seeds.
robustness when applied to a diverse range of environments. We then analyse the differences between Adam-Rel and Adam's 's updates. We compare seeds on the Traftax-Classte environment for this purpose, recording the update norm, maximum update, and gradient norm of every update.
5.2 Off-policy RL
shows the performance of DQN agents trained with Adam-Rel against those trained with Adam-MR and Adam on the Atari-10 benchmark [14]. We tune the learning rate of each method, keeping all other hyperparameters fixed at values tuned for Adam in CleanRL [9] Adam-Rel outperforms Adam, achieving 65.7% % vs. 28.8 8% human-normalized performance. Furthermore, the stark performance difference between Adam-Rel and Adam-MR (23.5%) demonstrates the advantage of retaining momentum information across target changes (so long as appropriate corrections are applied), thereby contradicting the gradient contamination hypothesis discussed in Bengio et al. (I]] and Asadi et al. [10).
Figure 2
More surprisingly, Adam-MR performs substantially worse than Adam, contrasting with the findings of Asadi et al. [10]. We evaluate on a different set of Atari games and tune both Adam and Adam-MR separately, which may account for the differences. However, these results suggest that preventing any gradient information from crossing over target changes is an excessive correction and can even harm performance. We additionally evaluate the set of games used by Asadi al. [TO], the results of which can be found in Appendix B We find that Adam-Rel outperforms the Adam baseline in IQM. We also find that, although implementation of Adam-MR again significantly under-perfonee relative to the Adam baseline, we approximately match the returns listed in their work.
We also evaluate Adam-Rel Rel when soft target changes are used, by comparing Adam and Adam-Rel on Atari- 10 when using DQN with Polyak averaging. We find that Adam-Re also outperforms Adam in this setting. These results, along with more detailed discussion, be found in Appendix C
5.3 On-policy RL
Craftax Figure 3] shows the performance of PPO agents trained on Craftax- 1B over 8 seeds. Most strikingly. Adam-MR, which resets the optimizer completely when PPO samples new batch, achieves dramatically poorer performance across all metrics. This deficit unsurprising when compared to its performance on DQN, where the optimizer has many more updates between resets and so can achieve a superior momentum estimate, and demonstrates the impact of not retaining any momentum information after resets in on-policy RL. Similarly, Adam with B 3g [16] achieves poorer performance than Adam-Ret on all metrics and has no significant different against Adam with default hyperparameters.
Furthermore, Adam-Re Rel outperforms Adam on all metrics. Whilst the performance on the number of achievements is similar, we follow the evaluation procedure recommended in Hafner (2] and report score, calculated as the geometric mean of success rates for all achievements. This metric applies logarithmic scaling to the success rate of each achievement, thereby giving additional weight to those that are hardest to accomplish. We see that Adam-Rel clearly outperforms Adam score, as well as on the two hardest achievements (collecting diamonds and cating plant). These behaviours require

---

Number achievements
Collect Diamonds
Eat Plant
Adam (8, Bs)
Adam-MR
Adam- Rel
Adam
17.4 17.7 18.0
18.3
60
63
66
69
2.5
5.0
7.5
2
Figure 3: PPO on Craftax-1B IB - comparison of Adam-Rel against Adam, Adam-MR, and Adam
with 81 B [66.. Bars show the 95% stratified bootstrap confidence interval, with mean marked, over seeds 22
a strong policy to discover so are learned late in training, suggesting that Adam-Re improves the plasticity of PPO.
Atari-57 Figure 2 shows the performance of PPO agents on Atari-57. As before, entirely resetting the optimizer significantly harms per- formance when compared resetting only the count. Across all environments, Adam-Rel also improves over Adam, outperforming it in out of the 55 games tested and IQM across games. Adam with 81 = 32 also fails to improve over the baseline.
1.00
Fraction rurs with score >
Adam
0.50
0.25
To further analyse the impact of Adam- over Atari-57, we plot the performance profile of human-normalized score (Figure 4). Whilst the performance of the two methods is similar over the bottom half of the profile, we see major in- crease in performance in the top half. Namely, at the 75th percentile of scores Adam- -ell achieves performance of 338% % vs.
116%
0.00
5
Human Normalized Score (T)
Figure 4: Performance Profile of Adam and Adam- Rel on Atari-57. Error bars represent the stan- dard error across 10 seeds. Green-shaded areas represent Adam-Re outperforming Adam and red- shaded areas the opposite.
comman-normalian 220 % achieved by Adam. This demonstrates the ability of Adam-Rel to improve policy per- formance on tasks where Adam is successful but suboptimal, without sacrificing performance on harder tasks.
5.4 Method Analysis
In this section we connect our theoretical exposition in Section 3 to our experimental results. Specif- ically, we first examine whether gradients increase in magnitude due nonstationarity. what extent predictions from our model match the resulting updates, and how Adam's 's update differs from Adam-Rel's in practice.
To this end, we collect gradient (i.e., before passing through the optimizer) and update (i.e., the final change applied to the network) information from PPO Eraftax-Classie We follow the experimental setup in Section but truncate the Craftax- -Caassi runs to 250M steps to reduce the data processing required. The results are shown in Figure 5
Comparing Theory and Practice In Figure both Adam and Adam-Rel face a significant increase in gradient norm immediately after starting optimisation on new objective resulting from new batch of trajectories collected under updated policy and value function. While this matches the assumptions we make in our work, the magnitude of the increase is much less than some of the values explored in Section 3
For Adam, this is approximately 29% and for Adam-Re it is around 45%. The grad norm profiles look similar in each case, with the norm peaking early before decreasing below its initial average value. This decrease and the initial ramp both deviate from step function we assume in our model. It is obvious that our theoretical model of gradients, which requires an increase in the gradient magnitude on each abrupt change in the objective, cannot hold throughout training in its entirety because this would require the gradient norm increase without bound.

---

nomm
grad_norm
0.46
0.063
-130
0.42
0.07
0.30
0.32
0.60
Upsate Norm
0.061
-225
Per Paramae
0.40
0.38
0.06
0.28
0.26
Parameter
0.50
0.059
1.5
0.36
0.24
0.
1.00
0.22
0.45
0.0
1.05
16
16
8
16
8
12
Timestop Period
Timestep
Timestop Period
Timestep within Period
(a) Adam
(b) Adam-Rel
Figure 5: Adam and Adam-Rel compared to the theoretical model. To make this plot, divided all the updates the PPO run into chunks, each of which was optimising a stationary objective. We then averaged over all the chunks. The red dashed lines show the different epochs for each batch of data. The assumption about the gradient under the model is shown the grad norm plot. Note that the update norm plot for Adam and Adam-Rel has separate y-axes. The shading represents standard error.
However, we find that despite this discrepancy, for Adam-Re Meam-Rel the update predicted by model fairly closely matches the shape of the true update norm, i.c., a fast drop the beginning followed by flattening (the scaling is not comparable between observed and predicted values).
For Adam, our model explains the initial overshoot of the update norm but then fails predict the rapid decrease, which results from the fast drop in the true gradient norm. Given the simplicity of our modeling assumptions, we find these results overall encouraging.
Under the assumption of a step increase in gradients an infinite relative magnitude Adam-Rel results in a flat update, while Adam would drastically overshoot. Clearly, this assumption does not hold practice, as we have shown above. However, we believe that this mismatch between reality and assumption is encouraging, since experimental results show that Adam-Rel is still effective this regime. Our hypothesis is that there are two benefits to designing Adam-Rel under these assumptions. First of all, avoids overshoots and secondly, when there are less drastic gradient steps it undershoots, which might have similar effects to fast learning rate annealing. These kind of annealing schedules (over longer horizons) popular when optimising stationary losses 23
Spherical Cows
under large gradient steps
Related Work
Optimization in Reinforcement Learning Plasticity loss (25-27) refers the loss in ability of
models to fit new objectives as they are trained. This is particularly relevant in nonstationary settings such as RL and continual learning, where the model continuously fitting changing objectives. Many solutions have been proposed, including resetting network layers [28 32), policy distillation [25]. LayerNorm [4] regressing outputs to their initial values [26). resetting dead units (35) and adding output heads during training [36). These solutions, in particular resetting layers during training [2
32.. have contributed towards state-of-the thett-rtttrrtt performance on Atari 100k (30]. However, these works, only Lyle et al. 33 investigate the relationship between the optimizer and nonstationarity. demonstrating that by reducing the momentum coefficient of the second-moment moment gradient estimate in Adam, the fraction of dead units no longer increases. However, these works focus plasticity loss, which a symptom of nonstationarity, and only analyse off-policy RL. In contrast, we address nonstationarity directly and evaluate both on-policy and off-policy RL.
provides an alternative approach designing optimizers for reinforcement learning. Rather than manually identifying problems and handcrafting solutions for RL optimization, this line of work seeks to automatically discover these solutions by meta-learning components of the optimization process. Often these methods parameterize the agent's loss function with a neural network, allowing be optimized through meta gradiensss 42. or zeroth-orde Recently, Lan et al. 455 proposed veta-learning a black-box optimizer directly,
Meta-reinforreemen learning
methods 43 20 44

---

demonstrating competitive performance with Adam on range of RL tasks. However, these works are limited by the distribution of tasks they were trained on, and using handcrafted optimizers in RL is still far more popular.
Adam Extensions Cyclical update schedules 446 have previously been applied in supervised learning as mechanism for simplifying hyperparameter tuning and improving performance, and Loshchilov and Hutter (47 propose the use of warm learning rate restarts with cosine decay for improving the training of convolutional nets. Liu et al. [48] examine the combination of Adam and learning rate warmup, proposing RAdam to stabilise training. However, all of these methods focus on supervised learning and therefore assume stationarity.
There has also been some investigation of the interaction between deep RL and momentum-based optimization. Henderson et al. [49] investigate the effects of different optimizer settings and recom- mend sensible parameters, but do not investigate resetting the optimizer. Bengio al. [11] identify the problem of contamination of momentum estimates and propose solution based on a Taylor expansion. Dohare et al. [16] investigate policy collapse in RL when training for longer than methods were tuned for and propose setting 31 .2 to address this. By contrast, we investigate training for standard number of steps and focus on improved overall empirical performance, rather than avoiding policy collapse. Asadi et al. [10], which is perhaps the most similar to our work, also aim to tackle contamination, but do so differently, by simply resetting the Adam momentum states to whenever the target network changes in the value-based methods DQN and Rainbow. However, they do not consider resetting of Adam's 's timestep parameter, and explain their improved results by suggesting that old, bad, momentum estimates contaminate the gradients when training contrast, we demonstrate that resetting only the timestep suffices for better performance on a range of tasks and therefore that the contamination hypothesis does not explain the better performance of resetting the optimizer. We also demonstrate that retaining momentum estimates can be essential for performance, particularly in on-policy RL.
new objective. By
Adam in RL To adapt Adam for use in RL, prior work has commonly applied a number of modifications compared to its use in supervised learning [8]. The first is to set the parameter to 10-5, which is a higher value than the 10-8 typically used in supervised learning. Additionally many reinforcement learning algorithms use gradient clipping before passing the gradients Adam. Typically gradient vectors are clipped by their Le norm.
A higher value of reduces the sensitivity of the optimizer to sudden large gradients. If objective has been effectively optimized and hence the gradients are very small, then a sudden target change may lead to large gradients. á typically updates much more slowly than ń and therefore this causes the update size to increase significantly, potentially causing performance collapse. However, this implementation detail is not mentioned in the PPO paper (4), and subsequent investigations omit it [6 Clipping the gradient by the norm also aims at preventing performance collapse. Andrychowicz al. [6] find this increase performance slightly when set to 0.5.
7 Limitations and Future Work
In this work we have mostly examined abrupt nonstationarity, where there are distinct changes of target, it is in that setting where our method can be most cleanly applied. However, a range of RL methods face continuous nonstationarity, such as when applying Polyak averaging [1] to smoothly update target networks after every optimization step. We have demonstrated improved performance in this algorithm in Appendix C However, further investigation into how to apply resetting in this setting would be beneficial.
There are also many promising avenues for future work. First, while we have focused on RL, it would be interesting apply Adam-Rel to other domains that feature nonstationarity such as RLHF, training on synthetic data, or continual learning. Additionally, we also note that while our results are promising, it was not possible to investigate all RL settings and environments in this work, and we therefore encourage future work in settings such as continuous control. Secondly, Adan-Rel is designed with the principle that large updates can harm learning, but is not clear general what properties of update sizes are desirable in nonstationary settings. Understanding this more clearly may help produce meaningful improvements optimisation. Relatedly, it would be beneficial to better understand the nature of gradients in RL tasks, in particular how they change throughout training for

---

different methods and what effect this has on performance. Finally, re- re-examining other aspects of the RL toolchain that are borrowed from supervised learning could produce further advancements by designing architectures, optimisers and methods specifically suited for problems in RI..
8 Conclusion
We presented simple, theoretically -ottivaate method for handling nonstationarity via the Adam optimizer. By analysing the impact of large changes gradient size, we demonstrated how directly applying Adam to nonstationary problems can lead to unstable update sizes, before demonstrating how timestep resetting corrects for this instability. Following this, we performed an extensive evaluation of Adam-Rel against Adam and Adam-MR in both on on-policy and off-policy settings, demonstrating significant empirical gains. We then demonstrated that increases in gradient magnitude after abrupt objective changes occur in practice and compared the predictions of our simple theoretical model with the observed data complex environment, Adam-Re Rel can implemented as simple, single-line extension to any Adam-based algorithm with discrete nonstationarity (e.g. target network updates), leading to major improvements in performance across environments and algorithm classes. We hope that the ease of implementation and effectiveness of Adam-Re Rell will encourage researchers to use it as a de facto component of future RL algorithms, providing a step towards robust and performant RL.
Acknowledgements
BE and MJ are supported by the EPSRC centre for Doctoral Training in Autonomous and Intelligent Machines and Systems EP/S024050/I. MJ is also supported by Amazon Web Services and the Oxford-Singapore Human-Machine Collaboration Initiative. The experiments were made possible by generous equipment grant from NVIDIA.
References
[1] Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 1(5):1-17, 1964.
[2] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1121 1147. PMLR, 2013.
[3] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[4] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint orXiv:1707.0634/ 2017.
[5] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: case study on ppo and trpo. arXiv preprini arXiv:2005. 12729, 2020.
[6] Marein Andrychowicz, Anton Raichuk, Piotr Stañczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? large-scale empirical study. arXiv preprint arXiv:2006.05990, 2020.
[7] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[8] Shengyi Huang. Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The implementation details of proximal policy optimiza- tion. In ICLR Blog Truck, 2022. URL hunp:///dchhiii.rraa.aaiamiho/o2
03/25/ htt.:ppppppmmleeaaaaoeo---eeeeee impllementatioo-dettal..
10

---

[9] Joelle Pineau, Philippe Vincent-Lamaree Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility machine learning research (a report from the neurips 2019 reproducibility program). The Journal of Machine Learning Research, 22(1):7459-7477 -478, 2021.
[10] Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep rl: An empirical study. arXiv preprint 2rXiv:2306. /7833, 2023.
[11] Emmanuel Bengio, Joelle Pineau, and Doina Precup. Correcting momentum in temporal difference learning. arXiv preprint arXiv: /0.3990, 2021.
[12] Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Jackson, Samuel Coward, and Jakob Foerster. Craftax: lightning-fast benchmark for open-ended reinforcement learning. arXiv preprint arXiv:2402.16880 2024.
[13] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
[14] Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. In International Conference on Machine Learning, pages 421-438. PMLR, 2023.
[15] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL html
2nd
[16] Shibhansh Dohare, Qingfeng Lan, and A Rupam Mahmood. Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop Reinforcement Learning, 2023.
[17] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 1xXiv:/506.02436 2015.
[18] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, arXiv preprint 11X1v:1312.5602, 2013.
[19] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18 2022.
htRl:ttt:://mmll org/ papprs/v23/211111111//// 1342. html
[20] Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35:16455-16668, 2022.
[21] Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2021.
[22] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320 2021.
[23] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[24] Max Welling and Yee W Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681-688. Citeseer, 2011.
[25] Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference Learning Representations, 2020. 2020.

---

[26] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2021.
[27] Jordan Ash and Ryan Adams. On warm-starting neural network training, Advances in neural information processing systems, 19:3884-3894 2020.
[28] Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, PierreLuue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pages
PMLR, 2022.
[29] Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville, Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference Learning Representations, 2022.
[30] Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pages 30365-30380. PMLR, 2023.
[31] Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks. arXiv preprint arXiv:2202.00115, 2022.
[32] Charles Anderson. Q-learning with hidden-unit restarting. S. Hanson, J. Cowan, and
C. Giles, editors, Advunces in Neural Information Processing Systems, volume 5. Morgan- Kaufmann, 1992. //pp/ceepp/ neurips httpseeeeeeepppppp.rteeie...e
file/ 1tl1:/66643830066622226667777783
.df
[33] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernaro Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01486, 2023.
[34] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton, Layer normalization. arXiv preprint 160706460 2016.
[35] Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302.12902 2023.
[36] Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. arXiv preprint arXiv:2305.1555: 2023.
[37] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In Artificial Neurul NetWorks-ICANN 2001: International Conference Vienna, Austria, August 21-25, 2001 Proceedings 11, pages 87-94. Springer, 2001.
[38] Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint 16X1v:1611.05763. 2016.
[39] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 2016.
[40] Junhyuk Oh, Matteo Hessel, Wojeiech M Czarnecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. arXiv preprint arXiv:2007.08794 2020.
[41] Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti. Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. In 25th International Conference on Pattern Recognition (ICPR), pages 4161-4168. IEEE, 2021.
[42] Matthew Thomas Jackson, Minqi Jiang, Jack Parker-Holder, Risto Vuorio, Chris Lu, Gregory Farquhar, Shimon Whiteson, and Jakob Nicolaus Foerster. Discovering general reinforcement learning algorithms with adversarial environment design. In Thirty- Conference on Neural Information Processing Systems, 2023.
12

---

[43] Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint arXiv:1802.0482/ 04821,, 2018.
[44] Matthew Jackson, Chris Lu, Louis Kirsch, Robert T. Lange, Shimon Whiteson, and Jakob N. Foerster. Discovering temporally-awes reinforcement learning algorithms. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://operreee net/Forrun21d-M3Jce333
[45] Qingfeng Lan, A Rupam Mahmood, Shuicheng Yan, and Zhongwen Xu. Learning to optimize for reinforcement learning. arXiv preprint arXiv:2302.014700 2023.
[46] Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pages 464-472. IEEE, 2017.
[47] Uya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2016.
[48] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. International Conference Learning Representations, 2019.
[49] Peter Henderson, Joshua Romoff, and Joelle Pineau. Where did my optimum go?: An pirical analysis of gradient descent optimization in policy gradient methods. arXiv preprint arXiv:1810.00255,, 2018.
[50] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint 11X1v:1812.06110,, 2018.
13

---

A Proof of Theorem 1
Starting from the definition of the momentum term in Adam's update rule:
m = (1 (1)
B1)
31 g).
j=- v'
= -1-B11 9 Σ
B1 f)=1 + kg Σ
31
$=-1
5-0
(1 (1-8110 -1)8''
Σ
- + Σ
B1
5=0
= -1-
Σ
31' 1 Σ
(BT
1=0
From the solution to the sum of geometric series:
1 - 31 1¹
1 - B1 - ((+1)
m (1 31)31'
B1
+ k
1 B1
1 - 31
B1 1 - 31"
31 - $1
(1 31)31'0 'g
+ k
1 $1
B1
B1 ++ (1
81") + 1(1 - 1611-11 11
Similarly for v/, it follows:
Ut = g² 32 +1 32") + 33
Substituting of and m; into the Adam momentum updates with e = 0 yields:
m m've
V 32 "+6+1
1 31 14++1
B 11 + k(1 - 33, 41)
-3,
g
V g² 32 (1 (1- 32")
12(1 32 11)
Taking the limit L' > aa with 31, E [0, 1) yields our desired result:
miva
B1 ++ + +(1 - 81 +11
lim
VoLere 1 ++ 12 121- 11
B Results comparison with Asadi et al.
Asadi et al. [10] find in their paper that their method, when applied to DQN, gives roughly comparable performance to their Adam baseline. However, in our paper we find that Adam-MR performs significantly worse than the Adam baseline, even when compared on the same games as in Figure 6 There Adam-Rel performs better than Adam the inter-quartile mean, but worse on the median. However, given this is a selection of just 12 games of very different difficulties, the median is often likely in this case to reduce single game for most algorithms.
To investigate this disparity, we compare our results for Adam-MR to theirs in Table | We estimated their scores in each game from the appropriate figures in their paper. Overall we see that our
14

---

Table Comparison with the results from Asadi al. [O]. The scores are estimated by taking the performance at 40M frames from the figures in their paper. We compare to both K = 1000, which is our default hyperparameter, and K 8000, which their default hyperparameter.
Score
Nortalized Scone
Eivinoment
Adam XR (K-1000)
Adan MR/K--10000
Rel (K-
(K~1000)
(K-5000)
Rei (K-1000)
Aunidar
350
270:2:20
0.20
0.17
Aduric
1400
47000
5550 700
1.19
0164
0-40 0-000000000000
BeanBiler
3500
4300
4500 F20
0,21
0.24
I127 2: 50000
Brakose
160
300
300 -50
5.3
5.0
10.5 8:
ChanyCimber
85000
51300 (9210)
0.41
283
2.6 2: 0.3
DemomAilack
Gxpter
1500
0.02
Kangaroo
10500
K250
1230) 1830
35
2.75
1. 4-0.3
Phoenix
4250
4500
1503 1000
0.54
0.58
0.5 0.2
Scepued
1.000
1000
1500 300
0.05
0114
0L03 0.009
Zacion
1000
6200
2200
300
0.11
0U67
0.24 2: 0.04
Mesn
1.44
Quattll Mes
IQM
Median
Adam-MR Adam-Rel
Adam
0.8
1.2
1.6
2.0
0.5
1.0
1.5
Human Normalized Score
Figure 6: Comparison of the inter-quartile mean and median of Adam-MR, Adam-Rel and Adam on the Atari games evaluated on in Asadi al. (10).
implementation, which uses K 1000, performs significantly better than their implementation with = 1000. It is also better in mean but worse in median and inter-quartile mean than their K 8000 implementation. In short, our results broadly match theirs reported after similar amount of training. but our Adam baseline performs significantly better than theirs. However, there are number of differences in our evaluation. First run for 10M steps (40M frames) whereas they run for 30M steps (120M frames). Secondly, they use the Dopamine (50) settings for Atari, whereas we use the more standard ones used by DQN [18]. We kept these settings throughout our paper avoid sigmificant hyperparameter tuning by evaluating in as standard settings as possible. We believe these results demonstrate the correctness of our implementation of their work and that our method still performs favourably.
C Results with Polyak Averaging
We also run experiments on DQN with Polyak averaging to examine the effect of Adam-Rel in cases where there are soft-target updates.
set = 0.02. This was chosen so that after 250 steps, the previous target update frequency, the original target parameters would contribute just 0.5% the new target. We found the best learning rate to be lower, at 5 x 10 5 The results are shown in Figure 7 As shown in that figure, Adam Rel outperforms Adam in this setting as well, achieving a higher median value, although still retaining long tail of negative results. We also note that although Adam achieves better performance than the baseline without Polyak averaging, Adam-Rel performs used. This may be due to resetting t being less effective when soft-target changes are used, or that more extensive tuning may improve its performance.
than when Polyak averaging is not
D Code Repositories
For the Atari experiments (both DQN and PPO), we based our implementation CleanRL [19]. This code is available here For the Craftax experiments, we based our implementation on PureJaxRL 20
This code is available here)
'This is the inter quartie mean over environments as opposed to the more usual over environments and seeds. This is because Asadi al. [10 do not provide individual seed data.
15

---

DQN (Polyak) Atari-10 Regressed Median Adam-Rel Adam
0.2
0.3
0,4
0.5
Human Normalized Score
Figure 7: Comparison of the regressed median on the Atari-10 10 benchmark of Adam and Adam-Rel for DQN with Polyak Averaging.
E
Compute and Additional Experiments
For our DQN experiments, swept over learning rates for Adam-MR, Adam-Rel and Adam. For PPO experiments, we swept over learning rate, max gradient norm and GAE À values, as we found these to differ from the PPO defaults. Experiments were performed V100 GPUs. Experiments were scheduled using slurm, with 10 CPU cores per GPU.
internal cluster of NVIDIA
The Atari PPO experiments required around 10000 GPU hours to complete, including hyperparameter tuning. The DQN experiments, because of the computational inefficiency of DQN, take much longer to run (approximately 2 days per experiment), and hence used total of 14000 GPU hours, despite there being many fewer parallel runs. The Craftax-Classic experiments took around 300 GPU hours to complete.
F Hyperparameters
Table 2: Atari Adam PPO hyperparameters
Hyperparameter
Value
Learning Rate
0.00025
Number of Epochs
4
Minibatches
4
0.99
GAB X
0.95
Normalise Advantages
True
0.1
Value Function Clipping
True
Max Grad Norm
0.5
Number of Environments
Number of Rollout Steps 128
Table 3: Atari Adam-Rec and Adam-MR PPO hyperparameters
Hyperparameter
Value
Learning Rate
0.002
Number of Epochs
4
Minibatches
0.99
GAE
0.9
Normalise Advantages
True
0.1
Value Function Clipping
True
Max Grad Norm
5.0
Number of Environments
8
Number of Rollout Steps
128
16

---

Table 4: Atari-10 10 DQN hyperparameters
Hyperparameter
Value
Learning Rate
0.0001
Buffer Size
1 106
?
0.99
GAE X
0.9
Target Network Update Steps
1000
Batch Size
32
Start
End
0.01
Exploration Fraction
0.1
Number of Steps without Training
80000
Train Frequency
4
Table 5: Craftax Adam and Adam-MR PPO hyperparameters
Hyperparameter
Value
Learning Rate Number Epochs
0.0003
Minibatches
4
?
0.99
GAE X
0.9
Normalise Advantages
True
0.2
Value Function Clipping
True
Max Grad Norm
Number of Environments
512
Number Rollout Steps
64
Table 6: Craftax Adam-Rel hyperparameters
Hyperparameter
Value
Learning Rate
0.001
Number of Epochs
4
Minibatches
4
0.99
GAE
0.7
Normalise Advantages
True
0.2
Value Function Clipping
True
Max Grad Norm
5
Number of Environments
512
Number of Rollout Steps
64
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As claimed in the introduction, we provide an analysis of the Adam update rule under nonstationary gradients in Section 3 introduce and analyse Adam-Rel in Section then evaluate Adam, Adam-Rel, and Adam-MR on Atari and Craftax in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations, along with suggestions for future work, in Section We also examine how our theoretical assumptions match practice in Section 5.4
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "L_imitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: We We clearly state our assumptions about the gradient and optimiser in Equation 2
and Theorem 3.1
We provide the proof in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We detail how to reproduce the experiments in Section 5 as well as open- sourcing our code. We also describe the implementation of our method in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide anonymised links to code in the Appendix, and only run open-source environments, allowing for our experiments to be reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttps://nnis.cc)
atblcc/guudes//ddeeelliissss wwtilissssoooolllaoioo onPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (httpa
/
assisnsiiooooo asionPlicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details of our hyperparameter settings in Appendix as well as detailing our experimental setup in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In reporting our results, we follow the recommendations of Agarwal et al. 22 We provide details of the error bars in the figure captions for each plot.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the compute requirements in the Appendix. We also discuss there preliminary experiments that were included.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httup::/neurips..c.. http:///aaarrrr..cpppppattataaaa
Answer: [Yes]
Justification:
We have read and reviewed the ethics guidelines ensure work complies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is foundational machine learning research and as such has no direct path to negative societal consequences separate from advancement in machine learning.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper contains no such risky models or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite CleanRL, which our PPO and DQN implementations based and only rely on open-source freely available libraries.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.paperssttt www.ppppptsthhhcco thcode www.coo//aaaao datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide anonymised links to the released code in the Appendix and document how to run experiments.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper contains no crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "7uWJE9AVnj", "reviewer": "Reviewer_JkaJ", "summary": "This paper studies the effect of non-stationarity on the Adam optimizer. It shows that the standard use of the Adam optimizer can lead to large updates, which can cause sub-optimal performance. To address this issue, Adam-Rel is introduced, which resets Adam's timestep parameter to zero when the target network changes. Finally, experiments show that Adam-Rel provides performance improvements over Adam.", "questions": "How does Adam with $\\beta_1 = \\beta_2$ perform in your experiments?", "limitations": "The authors adequately discuss the limitations.\n\n-----------------\nUPDATE: I increased my score as the authors provide comparison with Dohare et al. (2023).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is generally well-written. The paper studies an important problem of large updates caused by Adam in non-stationary settings like reinforcement learning. Explicitly studying basic components of modern deep-RL, like Adam, in non-stationary settings is an important direction of research. The proposed solution, Adam-Rel, is simple and easy to implement.", "weaknesses": "The same problem of large updates by Adam in non-stationary problems has been studied before (Lyle et al., 2023; Dohare et al., 2023). They both use the solution of setting $\\beta_1 = \\beta_2$. The authors seem aware of this proposed solution because the work by Dohare et al. (2023) is discussed in the paper. However, Adam-rel is not compared with Adam with $\\beta_1 = \\beta_2$. A comparison with Adam with $\\beta_1 = \\beta_2$ is needed to accept this paper. Even if Adam with $\\beta_1 = \\beta_2$ performs better, this paper can be accepted, as it studies the problem in much more detail than any prior work.\n\nI'm willing to change my score if this comparison is added to the paper.\n\n\nLyle et al. Understanding plasticity in neural networks. ICML, 2023.\nDohare et la. (2023). Overcoming policy collapse in Deep RL. EWRL 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our work. We are pleased that you found the paper \"generally well-written\", studying an \"important problem\" and the proposed solution \"simple and easy to implement\". \nWe direct you to the common response for a summary of the issues raised by reviewers and our response to them.\n\nYour sole issue with the paper is that we do not include a comparison with the work of Dohare et al. [1]. Although it was not possible given time constraints to rerun all the experiments, we evaluated $\\beta_1 = \\beta_2$ in Craftax and found it achieves broadly similar performance as Adam, underperforming our method. See the uploaded PDF for details. \n\nDespite this, we were also surprised that you have initially rejected our paper, especially given that you state\n\n> Even if Adam with $\\beta_1 = \\beta_2$ performs better, this paper can be accepted, as it studies the problem in much more detail than any prior work.\n\nIf the paper can be accepted regardless of the result of this experiment, then the paper should also be accepted without it, albeit perhaps with a lower score. We hope that this reasoning and the new results will encourage you to raise your score significantly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n653ymxWqA", "reviewer": "Reviewer_uFqs", "summary": "This paper introduces a simple approach to address the issue of large updates commonly encountered with Adam optimizers in deep learning applications. The authors focus on a specific scenario prevalent in deep reinforcement learning: the updating of target networks. Instead of the conventional approach of resetting both the timestep and the momentum variables in the Adam optimizer, the authors propose only resetting the timestep. This modification leads to enhanced learning performance, as demonstrated through evaluations conducted in both the Atari and Craftax environments, utilizing a range of on-policy and off-policy algorithms.", "questions": "1. Have the authors considered extending the setting towards changes in data distribution rather than only due to a change in target network? \n2. The paper includes a comprehensive study using both off-policy and on-policy algorithms. Could the authors clarify the rationale behind this choice? Specifically, which type of algorithm is more affected by drastic changes in data distribution, and why?\n3. Figure 4 and Figure 5 should be re-labelled such that Figure 4 should be Figure 5 and Figure 5 should be Figure 4 since in the current version, Figure 5 appears before Figure 4. \n4. I find Figure 5 hard to interpret. Could the authors provide more detail on what the curves represent? Additionally, enhancing the clarity of the caption might help. \n5. Line 86: The authors mention a change of objective. However, it seems the objective function remains consistent throughout. Could the authors clarify if they are actually referring to changes in data distribution rather than the objective itself?\n6. Line 259: The authors mention 'matching the trend.' Do you mean matching the overall trend rather than the specific shape or pattern?\n7. Line 261:  Could the authors clarify what you mean by 'overshooting' and 'undershooting'? The current usage is confusing.\n8. Line 283: The use of 'However' at the beginning of Line 283 immediately follows another sentence starting with 'However' in Line 280. Could these be rephrased to improve the flow of the text?", "limitations": "None. The authors have presented the concerned limitations of their work in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is generally clear, particularly benefiting from a detailed explanation of the Adam optimizer’s mechanics, which enhances accessibility for those who may be unfamiliar with the specifics. The simplicity of the proposed modification—only resetting the time step—is a significant advantage, allowing easy implementation and thus potentially reducing errors in implementation. The experiments are thoughtfully designed, aligning closely with the research questions and conducted in the Atari and Craftax environments, which are known for their complexity. Additionally, the selection of baselines, including the standard and a modified version of Adam, demonstrates the proposed approach’s effectiveness against established methods.", "weaknesses": "The research question addressed by the authors, focusing solely on large gradient updates due to changes in the target network, seems somewhat narrow. This focus may overlook the broader applicability and efficacy of the proposed solution across various reinforcement learning scenarios. Notably, many algorithms have achieved better stability using softer target updates, like Polyak averaging, which the authors acknowledge [1,2,3]. It would be beneficial to see how the proposed optimizer modification compares in environments where such techniques are traditionally employed.\n\nMoreover, the approach of resetting parameters in an optimizer, though straightforward, could be considered a drastic and potentially inelegant solution. This method might introduce other issues, such as affecting the convergence behavior of the optimizer. While I currently lack an alternative suggestion, exploring methods that adjust the scale of parameter updates dynamically, rather than resetting them, might offer a more refined solution.\n\n[1] Lillicrap, Timothy P., et al. \"Continuous control with deep reinforcement learning.\" arXiv preprint arXiv:1509.02971(2015).\n\n[2] Kaplanis, Christos, Murray Shanahan, and Claudia Clopath. \"Continual reinforcement learning with complex synapses.\" International Conference on Machine Learning. PMLR, 2018.\n\n[3] Schwarzer, Max, et al. \"Bigger, better, faster: Human-level atari with human-level efficiency.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We were pleased with your extensive praise of our paper, finding the writing \"generally clear, particularly benefitting from a detailed explanation of the Adam optimizer's mechanics\", the \"simplicity of the method a significant advantage\", the experiments \"thoughtfully designed, aligning closely with the research directions and conducted in...environments, which are known for their complexity\", and that the baselines \"demonstrate the proposed approach's effectiveness against other methods\".\n\nYour first criticism of our paper is that the setting seems \"somewhat narrow\". We strongly disagree with this characterization. We demonstrate results in the most studied on-policy and off-policy RL algorithms. No other reviewer has raised this as a concern, with all other reviewers praising the applicability of our approach.\n\nSecondly, you criticize our solution as \"drastic and potentially inelegant\", further stating that it \"might introduce other issues, such as affecting the convergence behavior of the optimizer\", and finally suggesting we try to find a more \"refined\" solution. Firstly, this criticism contradicts *all reviewers* who praise our approach's simplicity. Secondly, the claim that our approach \"might introduce other issues\", without a clear explanation as to how or why those issues might occur is entirely generic. Such generic claims are unfair and invalid criticism. \n\nThirdly, while we could always do more theoretical analysis on the convergence properties of our approach, we provide extensive empirical results that demonstrate our approach's ability to find higher return policies than relevant baselines across on-policy and off-policy RL. Finally, we find your generic criticism of our work using imprecise terms such as \"drastic\" and \"inelegant\" entirely unfair.\n\nWe respond to your questions one by one below. \n\n1. We do not consider changes in data distribution in this work. While we think this is an interesting area of research, we believe that methods to address such problems are likely to look different to our approach and are best studied in future work. We believe our current contribution stands well enough alone without such additions.\n2. As discussed above, we do not address data distribution shifts in this work. For target nonstationarity, we would expect that value-based methods are more affected by this given their long optimisation times with fixed target networks. Our method's stronger performance in DQN hints at this, but more research is likely needed to verify this hypothesis.\n3. Thank you for pointing this out. We have fixed it in our copy and will include this in the camera ready.\n4. In PPO, you collect new data and optimize a stationary objective, before repeating the process with your new actor and critic. Each optimization is thus over a stationary objective. In Figure 5, we average over these stationary objectives throughout training and analyze the gradient (i.e. before the optimizer) and update (i.e. after the optimizer) properties. This rebuttal is too short for a detailed discussion of the findings, for which we refer you to the paper.\n5. In the section following line 86, we clearly and precisely explain our setting, clarifying on line 90 that the $\\phi$ used in $L(\\theta, \\phi)$ does not include the data distribution. Would you be able to clarify your misunderstanding here? We clearly state in that section we do not investigate the data distribution.\n6. Yes exactly, we mean that the overall shape matches.\n7. By overshooting and undershooting, we mean increasing (or decreasing respectively) over a threshold. We will amend this section in a camera-ready version to clarify this.\n8. Thank you for pointing this out, we will amend it in a camera-ready version.\n\nThank you again for taking the time to review our paper. We were surprised that, given your extensive praise of our paper and your listed weaknesses, your final rating was a reject. We hope that our reasoning here encourages you to raise your score to a rating that reflects all points made in your review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T3RqGzdM2A", "reviewer": "Reviewer_37TD", "summary": "One of the main challenges of reinforcement learning is its inherent nonstationary nature. Such non-stationarity can cause learning difficulties. The tools currently available for deep reinforcement learning are largely borrowed from deep learning, such as the Adam optimizer, which this paper focuses on. The authors show that Adam, under nonstationarity, can have large updates leading to learning difficulties. Thus, they propose a simple modification to Adam in which they reset the time parameter at every epoch in PPO or every target update in DQN. This slight modification seems to significantly improve performance, agreeing with the observations by several works that proposed similar modifications to Adam [1,2,3] and showed gains in performance.\n\\\n\\\n[1] Emmanuel Bengio, Joelle Pineau, and Doina Precup. Correcting momentum in temporal difference learning. arXiv preprint arXiv:2106.03955, 2021.\n\\\n[2] Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep rl: An empirical study. arXiv preprint arXiv:2306.17833, 2023.\n\\\n[3] Shibhansh Dohare, Qingfeng Lan, and A Rupam Mahmood. Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop on Reinforcement Learning, 2023.", "questions": "- One of the main assumptions in the theoretical work is that the gradients become near zero at convergence, and then we get sudden large updates once the gradients start to be large. This might be the case for PPO, but it doesn’t seem to be the case for DQN. Dohare et al. (2023) showed that setting beta1 to equal beta improves PPO performance but has a smaller impact on DQN. Adam-Rel shows improvement in both PPO and DQN, but DQN's motivation is not clear. Can the authors explain why they expect Adam-Rel to improve DQN?\n- I’m puzzled by Adam-MR's poor performance. Your results seem to contradict (as you mentioned) two previous works. Would it be possible that you used an incorrect implementation for Adam-MR?\n- Do the authors have any intuition as to why Adam-MR performs poorly? In particular, do they have a similar theoretical analysis for Adam-MR showing that its updates can be large or even larger than Adam?", "limitations": "The authors adequately discussed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors present a novel and simple approach to solving a big problem. This is especially promising since it will allow quick wide adoption in the future. The authors introduced the problem carefully and talked about it in such a clear way that guides the reader step by step, and they also provide some theoretical justification for why their method would work from first principles. The evaluation benchmarks seem extensive, suggesting that the methods can be applicable to a wide range of problems.", "weaknesses": "- The main weakness of the paper is its limited comparison to other methods, particularly the methods that are closely related to Adam-Rel (e.g. [1,3]). Additionally, the only competitor is Adam-MR [2] which almost always gives worse performance, contradicting previous works. There is no similar theoretical analysis on Adam-MR explaining the poor performance or some empirical evidence convincing the reader why it fails.\n- Writing can be improved, especially when explaining parts of Adam. This can be done using more proper mathematical terms such as “estimators” or “correcting the biasedness of an estimator,” etc.\n- The title is written in such a way that suggests your method is applicable to a wide range of RL methods, but it seems that this is not the case, and you only focus on PPO and DQN. I suggest a change that reflects your contributions without overstating them. \n- No pseudocode is given for Adam-Rel with DQN. It’s inferred from the context that a time parameter reset is done before each target function update, but the pseudocode needs to be there to confirm this for the reader.\n- Dohare et al. (2023) showed that policy collapse in PPO with MuJoCo is more pronounced than DQN in Atari. It might be because the observations are bounded in Atari but not in MuJoCo, which increases the non-stationarity. The authors didn’t consider any environments other than pixel-based ones with bounded observation ranges. I think it’s necessary also to show some results on MuJoCo environemnts (or similar environments) to confirm that the results are qualitatively similar in those settings.\n\\\n\\\n\\\n**Minor issues:**\n- “It uses a learned critic trained by a TD loss to estimate the value function, and a clipped actor update of the form.” -> This is not accurate. PPO doesn’t use a learned critic. The critic is learned as well.\n- “over which the above update is calculated” -> You wrote an objective in Eq. 1, not an update\n- “Adam, the most popular optimizer that  uses momentum” -> It doesn’t use the momentum you defined in the background. You might want to clear this up for the reader.\n\\\n\\\n[1] Emmanuel Bengio, Joelle Pineau, and Doina Precup. Correcting momentum in temporal difference learning. arXiv preprint arXiv:2106.03955, 2021.\n\\\n[2] Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep rl: An empirical study. arXiv preprint arXiv:2306.17833, 2023.\n\\\n[3] Shibhansh Dohare, Qingfeng Lan, and A Rupam Mahmood. Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop on Reinforcement Learning, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper. We're glad you find our method to be \"novel and simple\" and capable of \"wide adoption in the future\". We also appreciate your comments about the clarity of our writing and theoretical justification, in addition to our evaluation being \"extensive\". We respond to each of your weaknesses in order below:\n\n\n\n* Regarding your request for further baseline methods, we note that this topic is relatively underexplored and lacks significant prior work. However, despite the work being from a workshop and not yet published at an archival conference, we have now added the method decribed in Dohare et al. [3] to our evaluation on Craftax, which shows that it does not significantly improve performance over Adam and underperforms our method. We will add this baseline to the remaining experiments in the camera-ready copy of this paper.\nRegarding your suggestion that our Adam-MR scores contradict prior work, we refer you to Appendix B of our paper, which explains this discrepancy in detail. In summary, we replicate Adam-MR's original scores in Atari, but evaluate against a significantly stronger baseline. As you state, our experiments are extensive and should be sufficient to convince the reader that Adam-MR fails in the settings studied in our work.\n* We will endeavour to be increasingly formal in future revisions of this work, however we note that our current explanation of Adam is already extensive at two pages and that most reviewers praise the paper's writing and clarity.\n* We evaluate on DQN and PPO, the two predominant algorithms in on-policy and off-policy RL, which you agree suggests the method will be \"applicable to a wide range of problems\". Therefore, we believe that our results already demonstrate that Adam-Rel will be applicable to a wide range of RL methods and are unsure which additional methods you believe would enable us to make this claim.\n* We are confident that readers can infer the function of Adam-Rel on DQN without complete pseudocode due to the extreme simplicity of our method, as was agreed by all reviewers. You correctly infer the method in your review, however, we would be willing to add this to the appendix if you believe there is ambiguity as to where $t$ is reset.\n* Firstly, regarding your suggestion that we consider only pixel-based environments, we note that Craftax is symbolic and not pixel-based. Secondly, your suggestion that unbounded observations cause policy collapse is interesting, however, it is one of many valid hypotheses for this result. It is equally possible, if not more likely, that differences in the algorithms or other variations between the two environments cause this observation. Since this hypothesis does not appear to have strong support or context, we do not believe it is fair to require its investigation as a barrier to acceptance for this work.\n\nThank you for suggesting edits in the minor issues. We believe these are inferable from context -- the critic is \"learned\" as part of the algorithm, the update is generally the derivative of the objective, and we state that we provide only the \"typical\" formulation of momentum -- so we hope that these do not influence your score. However, we will take care to find less ambiguous wording in the next revision.\n\nFinally, we respond to each of your questions below:\n* We believe this phenomenon would be equally, if not more, pronounced in DQN as the value function undergoes more gradient steps before it is updated. However, verifying this is would provide an interesting area for analysis in future work.\n* We again refer you to Appendix B of our paper, which explains this discrepancy in detail. In summary, we replicate Adam-MR's original scores in Atari, but evaluate against a significantly stronger baseline. We also investigate Adam-MR on an entirely new algorithm (PPO) in two settings, where the performance drop is most pronounced.\n* We believe the Adam-MR results are unsuprising for PPO, as the algorithm has a small number of updates before the momentum is reset. This prevents a reasonable momentum value from being estimated by Adam-MR before it is reset, which predictably hurts performance. Given we demonstrate that resetting $t$ bounds update size, we believe a more pertinent question is why it would be necessary to reset the momentum at all?\n\nThank you again for taking the time to review our work. To summarise the key rebuttal: we have added an experiment with [3] but do not believe there is other significant prior work; we and most reviewers find our writing clear; our experiments already demonstrate wide applicability by covering the predominant algorithms in on-policy and off-policy RL; we and all reviewers find our method to be simple to understand; we clarified our environments and do not believe that unbounded observations are a necessary hypothesis to investigate for acceptance.\n\nGiven this, we as unsure of what further steps we can take to improve the paper based on your feedback. In light of our reasoning, your praise of the method, and your strong scores for Soundness (4), Presentation (3), and Contribution (3), we hope you will consider raising your score to an acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gmqfWVJTj7", "reviewer": "Reviewer_SmBt", "summary": "Due to problems with momentum and a non-stationary target, the authors propose resetting the value of ‘t’ in Adam, which is used to determine bias correction on the momentum terms, when updating the target. This approach is validated on PPO and DQN in Atari and Craftax.", "questions": "It would be interesting to see more analysis/insight either empirically or theoretically with regards to what happens to the gradient. For example, do momentum estimates correlate with the error terms over new iterations? While resetting seems like a brute force approach to mitigating problems with momentum and non-stationarity, it’s unclear that maintaining the momentum estimate is necessarily better.", "limitations": "Satisfactory.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Performance benefits over two key algorithms (PPO and DQN) when used with Adam.\n- Simple to implement idea.", "weaknesses": "- In terms of new insight, this paper offers very little that hasn’t been said in prior work (Bengio et al., Asadi et al.). Similarly, analysis in the paper is based around the update size. However, simply resetting would also bound the update size, so I feel like there is some lacking analysis or evidence to explain why maintaining momentum estimates is a reasonable approach. \n- Since the idea is not analyzed in depth, I would say there is limited empirical evidence (in terms of evaluations over environments and algorithms), especially compared to prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\nRegarding your comment on new insight, we ask where Adam's $t$ parameter is studied in either of the papers your shared, or any prior work? Furthermore, is there any suggestion that resetting this parameter can theoretically bound the update size, or have the empirical impact on performance presented here? If not, what further insight would go beyond the \"very little\" you suggest we present?\n\nRegarding your suggestion that Adam-MR also bounds the update size, we ask why it is more reasonable to *also* reset the momentum when resetting only $t$ is sufficient to bound the update size? Our evalution thoroughly demonstrates the empirical benefit of this and provides strong evidence for maintaining momentum estimates.\n\nRegarding our method not being analysed in depth, we provide a detailed background on Adam, analyse the update size with and without timestep resetting, and provide a formal proof analysing the impact of sudden changes in update size. Is there a specific element of this analysis you believe to be missing? If not, we do not believe that the analysis being not \"in depth\" is a well-founded criticism.\n\nRegarding our empirical evidence being limited compared to prior work, we note that, compared to prior work, we evaluate on an additional class of algorithms (on-policy) and a new environment (Craftax), and that all other reviewers praise the thoroughness of our empirical evaluations. Without a concrete suggestion for further experiments or what empirical evidence is missing, this appears to be a generic criticism that could be made on any paper and we ask that you reconsider this point.\n\nIn response to your question, we again ask why the momentum estimate should be reset given our demonstration that timestep resetting bounds the update size and our extensive results demonstrating that resetting momentum harms performance.\n\nWe hope you will be able to provide further clarification regarding the points raised above. If not, given our reasoning and your praise for our paper, we hope that you will either raise your score or explain the barriers preventing acceptance?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We summarise the strengths and weaknesses from all reviews below.

## Strengths

**Writing** Reviewer **JkaJ** praises the paper as "generally well-written", whilst Reviewers **uFqs** and **37TD** praise the paper's clarity, respectively commenting that the paper "benefits from a detailed explanation of the Adam optimizer's mechanics" and is presented in "a clear way that guides the reader step-by-step".

**Importance** Reviewers **JkaJ**, **uFqs**, and **37TD** praise the importance of the problem setting, respectively describing it as "an important direction of research", "commonly encountered with Adam optimizers in deep learning", and "a big problem...it will allow quick wide adpotion in the future", whilst Reviewer **SmBt** notes the method's "performance benefits over two key algorithms".

**Method** All reviewers (**JkaJ**, **uFqs**, **37TD**, **SmBt**) praise the simplicity and ease of implementation of the method.

**Results** Reviewers **JkaJ**, **uFqs**, and **37TD** praise the design and scope of our evaluation, respectively describing it as "studying [Adam with non-stationarity] in much more detail that any prior work", "thoughtfully designed...conducted in [environments] which are known for their complexity", and "extensive, suggesting that [Adam-Rel is] applicable to a wide range of problems".

## Weaknesses


**Writing** Reviewer **37TD** suggests that the writing "can be improved, especially when explaining parts of Adam." Given the current explaination is over two pages, we would struggle to fit further detail in the main body, but will endeavour to do so in the appendix. We also note that Reviewer **uFqs** praises the "detailed explanation of the Adam optimizer's mechanics".

**Importance** Reviewer **uFqs** complains that our problem setting is "somewhat narrow" because we only investigate instant target changes. We provide experiments that demonstrate its effectiveness in DQN and PPO, the two most studied deep RL methods encompassing both on-policy and off-policy RL. Additionally, no other reviewer raised this as a concern, with all other reviewers the applicability of our approach.



**Method** Reviewer **uFqs** also complains that our method could be considered a "drastic and potentially inelegant solution", suggesting that it "might cause issues with the convergence of the optimizer" and that we should find "a more refined solution". This is an unscientific criticism for which we provide no response. Suggesting an unspecified negative impact and using undefined terms such as "inelegant", "drastic" and "refined" to assess the quality of work has no place in peer review.
        
**Evaluation** Reviewer **37TD** suggests that Adam-MR's poor performance contradicts previous works, and asks whether we have an incorrect implementation. However, we explain this discrepancy in detail in Appendix B. To summarise this, we replicate the scores of Asadi et al. [1] in Atari, but evaluate against a significantly stronger baseline.

Reviewer **37TD** also cites Dohare et al.'s result that DQN in Atari has less performance collapse than PPO in MuJoCo, and suggests that we should evaluate in a setting that is not pixel based and without bounded observations. Firstly, the claim that we only evaluate in pixel-based settings is incorrect as Craftax has a symbolic observation space. Secondly, whilst bounded observations are one possible hypothesis for this result, there are a plethora of alternative explanations. This previous result could equally be explained by PPO's learning rate schedule, or other differences between these algorithms. Therefore, it is unscientific to suggest that these experiments be a barrier to acceptance given their lack of grounding.

Finally, Reviewer **37TD** criticises our title's claim of applying to a wide range of RL methods. We evaluate on DQN and PPO, the two predominant algorithms in on-policy and off-policy RL, and the same reviewer states that our method will have "quick wide adoption in the future" and be "applicable to a wide range of problems". As the reviewer suggests no additional algorithm that might offer further insight, it is unclear what might allow us to make this claim.

Reviewer **SmBt** claims that there is "limited emprical evidence...particularly compared to prior work". Unfortunately, they provide no suggestion as to what future experiments could be added, making the comment a generic criticism. Compared to prior work, we evaluate on an additional class of algorithms (on-policy) and a new environment (Craftax), and note that all other reviewers praise the thoroughness of our empirical evaluations.

**Prior Work** Reviewers **JkaJ** and **37TD** request comparison with other prior work, in particular Dohare et al. [1], an unpublished workshop paper. While it is not possible given the limited time to rerun all experiments with this baseline, **we add an evaluation of [1] in Craftax to the PDF**, which shows that it does not significantly improve performance over Adam and underperforms our method. In a camera-ready copy of the paper, we will add this baseline to all experiments. 

Reviewer **SmBt** claims that our work "offers very little that hasn't been said in prior work", in particular highlighting that resetting the optimizer would also bound the update size. This baseline (resetting the optimizer) is precisely the baseline that we compare to throughout the work, Adam-MR. Unfortunately, we believe this also misses the primary message of our paper: if resetting only $t$ is sufficient to bound the update size, why is it necessary to *also* reset the momentum?

[1] Dohare, Shibhansh, Qingfeng Lan, and A. Rupam Mahmood. "Overcoming policy collapse in deep reinforcement learning." Sixteenth European Workshop on Reinforcement Learning. 2023.

Meta-review: This paper proposes Adam-Rel, a small modification to Adam, that is motivated by the goal of reducing the impact of non-stationarity on policy optimization due to a sudden increase in update norms.

Reviews for this paper are mixed. The authors had concerns regarding the clarity/quality of initial reviews, but the reviewers clarified their opinions in discussion, and also engaged further in closed discussion about the strengths and weaknesses of this paper. The final discussion focused on two key issues: whether large update norms really are the problem, and whether the experimental evidence sufficient to say that the proposed approach is a good general solution.  I think there are reasonable reasons for doubting this paper on both issues. 

The authors demonstrate that at least on the problems they tested, update norms do increase at expected times with Adam, and Adam-Rel alleviates this. But this doesn't necessarily imply that fixing the update norms will always improve optimization. It is entirely possible that since the update norms do not increase without bounds in practice, Adam with suitable hyperparameters can adapt well to them. Similarly, it is possible to come up with scenarios where momentum reset (Adam-MR, a competing approach) might be more beneficial than Adam-Rel --- a reviewer suggests a scenario where the direction of the gradient changes. Could it be the case that Adam-MR is better on some problems (that were not tested) while Adam-Rel is better on other problems? Finally, famously tricky-to-get-right implementation details of PPO make it difficult to trust results when they are on a single benchmark (Crafter). 

Clearly, we can't be sure without much more experimentation. Moreover, there is an issue in deep RL research that algorithms from many papers can't be trusted due to insufficient rigor in designing experiments, and they turn out to not be useful in carefully controlled larger scale studies. This is the reason reviewers are being extra cautious.

Nevertheless, after careful consideration, I believe this paper should be accepted as it meets the bar for the results to be interesting enough to warrant further investigation by the broader community. My decision is based on the following:
- The authors have made effort to adopt a cautious tone in the paper (e.g. results "suggest" instead of "show"), instead of overly confident claims.
- The limitations of idealized theoretical analysis are made clear and then examined experimentally, again in the spirit of presenting and analyzing evidence.
- Many limitations of proposed approach are clearly discussed, and I expect that the authors would not object to discussing more.

I believe the only thing missing from this paper is an explicit admission that although the paper has spent considerable resources on experiments, RL environments and algorithms are so varied that it is quite possible that the results don't generalize broadly, and further investigation is warranted. I strongly encourage the authors to add a caveat like this to complete the limitations section, in line with their cautious approach throughout the paper.
Decision: Accept (poster)