Resetting the Optimizer in Deep RL: An Empirical Study
Authors: Kavosh Asadi
Rasool Fakoor
Shoham Sabach
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Reinforcement Learning
Rainbow Algorithm
Atari benchmark
Adam Optimizer
TL;DR: We show that simply resetting the optimizer after synchronizing the online and the target network can significantly improve the performance of the Rainbow algorithm on the Atari benchmark.

Abstract: We focus on the task of approximating the optimal value function in deep reinforcement learning. This iterative process is comprised of solving a sequence of optimization problems where the loss function changes per iteration. The common approach to solving this sequence of problems is to employ modern variants of the stochastic gradient descent algorithm such as Adam. These optimizers maintain their own internal parameters such as estimates of the first-order and the second-order moments of the gradient, and update them over time. Therefore, information obtained in previous iterations is used to solve the optimization problem in the current iteration. We demonstrate that this can contaminate the moment estimates because the optimization landscape can change arbitrarily from one iteration to the next one. To hedge against this negative effect, a simple idea is to reset the internal parameters of the optimizer when starting a new iteration. We empirically investigate this resetting idea by employing various optimizers in conjunction with the Rainbow algorithm. We demonstrate that this simple modification significantly improves the performance of deep RL on the Atari benchmark.

Resetting the Optimizer in Deep RL: An Empirical Study
Kavosh Asadi Amazon
Rasool Fakoor
Shoham Sabach Amazon & Techhion
Amazon
Abstract
We focus the task of approximating the optimal value function in deep rein- forcement learning. This iterative process is comprised of solving a sequence of optimization problems where the loss function changes per iteration. The common approach to solving this sequence of problems is to employ modern variants of the stochastic gradient descent algorithm such as Adam. These optimizers maintain their own internal parameters such as estimates of the first-order and the second- order moments of the gradient, and update them over time. Therefore, information obtained in previous iterations is used to solve the optimization problem in the current iteration. We demonstrate that this can contaminate the moment estimates because the optimization landscape can change arbitrarily from one iteration to the next one. To hedge against this negative effect, a simple idea to reset the internal parameters of the optimizer when starting new iteration. We empirically inves- tigate this resetting idea by employing various optimizers in conjunction with the Rainbow algorithm. We demonstrate that this simple modification significantly improves the performance of deep RL on the Atari benchmark.
Introduction
Value-function optimization lies at the epicenter of large-scale deep reinforcement learning (RL). In this context, the deep RL agent is equipped with parameterized neural network that, in conjunction with large-scale optimization, is employed to find an approximation of the optimal value function. The standard practice is to initialize the optimizer only once and at the beginning of training [1]. With each step of gradient computation, the optimizer updates its internal parameters, such as the gradient's moment estimates [2], and uses these internal parameters update the network.
A unique ingredient in value-function optimization is a technique known as forward bootstrap- ping [3]. In typical regression problems, the loss function is a measure of the discrepancy between fixed target and the agent's current approximation. This stands in contrast mization with forward bootstrapping, In this setting, the goal is to minimize the discrepancy between the agent's current approximation and a second approximation obtained by performing one or mul- tiple steps of look ahead. The regression target thus depends on the RL agent's own approximation which changes continually over the span of learning.
falue-function opti-
We first argue that in the presence of forward bootstrapping the value-function optimization pro- cess can best thought of solving sequence of optimization problems where the loss function changes per iteration. We demonstrate that in this case the loss function being minimized is com- prised of two inputs: target parameters that remain fixed during each iteration, and optimization (or online) parameters that are adjusted to minimize the loss function during each iteration. In this context, updating the target parameters changes the loss function being minimized in the next iter- ation. By extension, the first-order and the second-order moments of the gradient can also change arbitrarily because they depend on the landscape of the loss function.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Realizing that the RL agent faces a sequence of optimization problems, a natural practice would be reset the optimizer at the beginning of each new optimization problem. However, in deep RL, modern optimization algorithms are employed often without ever resetting their internal parameters. We question this standard practice, and in particular, similar to Bengio et al. [4] we ask if the internal parameters accumulated by the optimizer in the previous iterations can still be useful in updating the parameters of the neural network in the current iteration. Could it be the case that in most cases relying gradient computations pertaining to the previous iterations is just contaminating the internal parameters? And ultimately, can this negatively affect the performance the RL agent? We answer these questions through various experiments.
We then propose simple modification existing RL agents where we reset the internal parameters of their optimizer at the beginning of each iteration. Using the deep RL terminology, each time we update the target network, we also reset the internal parameters of the optimizer. We show that this remarkably simple augmentation significantly improves the performance of the competitive Rainbow agent [5] when used in conjunction with various optimizers. Most notably, under the standard Adam optimizer [6], we observe that resetting unleashes the true power of Adam and ultimately results in much better reward performance, suggesting that resetting the optimizer is a more promising choice in the context of value-function optimization.
RL as a Sequence of Optimization Problems
We argue that many deep RL algorithms can be viewed as iterative optimization algorithms where the loss function being minimized changes per iteration. To this end, we first recall that the DQN algorithm [7] decouples the learning parameters into two sets of parameters: the target parameters 8, and the optimization (or online) parameters w that are adjusted at each step. DQN updates these two parameters in iterations. More specifically, each iteration is comprised of an initial step, wt.0 05, performing multiple updates with a fixed 0%:
k + a(r + max g(ss "'''''''''' d': - g(s, a; w²th)) Voq(s, w w 1++ + -w fik
(1)
The agent then synchronizes the two learning parameters, g < w prior to moving to the next iteration t + 1 where updates are performed using d++1, Here K is is commonly set to 8000 for DQN and its successors [5].
whose value
Observe that the effect of changing the optimization parameters w max q(s', 0²) is ignored during gradient computation despite the fact that an implicit dependence exists due to synchroniza- tion. In fact, the update cannot be written as the gradient of any loss function that only takes single parameter as input [8]. However, this update can be written as the gradient of a function with two input parameters. To illustrate this, we now define:
H(0, w) =
Σ (8,0,
(r maxmaa's'' d':0) w(s) a
(2)
2
where B is the experience replay buffer containing the agent's environmental interactions. Observe that the update (1) could be thought of as performing gradient descent using V.. on single sample (s, a, r, 8') where v., is the partial gradient of H with respect to w. Therefore, by viewing the K gradient steps as rough approximation of exactly minimizing H with respect to the optimization parameters w, we can write this iterative process as:
0++1 Rs arg min H HO',, w) arg min H,(w).
(3)
In this optimization perspective [9], what is typically referred to the online parameter is updated incrementally because the objective function (2) does not generally lend itself into a closed-form solution. Notice also that in practice rather than the quadratic loss, DQN uses the slightly different Huber loss [7]. Moreover, follow-up versions of DQN use different loss functions, such as the Quantile loss [10], sample experience tuples from the buffer non-uniformly [1]], use multi-step updates [12, 13], or make additional modifications [5, 14). That said, the general structure of the algorithm follows the same trend in that it proceeds in iterations, and that the loss function being minimized changes per iteration.

---

3 Revisiting Adam in RL Optimization
So far we have shown that popular deep RI. algorithms could be thought of as a sequence of op- timization problems where we approximately solve each iteration using first-order optimization al- gorithms. The most primitive optimizer is the stochastic gradient descent (SGD) algorithm, which simply estimates and follows the descent direction. However, using vanilla SGD is ineffective in the context of deep RL. In contrast, more sophisticated successors SGD have been used successfully. For example, the original DQN paper [7] used the RMSProp optimizer [15]. Similarly, Rainbow [5] used the Adam optimizer [6], which could roughly be thought of as the momentum cementum-bbsed [16] version of RMSProp. More formally, suppose that our goal is to minimize certain loss function Jww). Then, starting from w", after computing a stochastic gradient g' z [ww), Adam proceeds by computing running average of the first-order and the second-order moments of the gradient as follows:
and where (2 applies the square function to g¹ element-wise. Notice that Adam we initialize m O and y 0. and the estimates are biased towards 0 in the first few steps. Define the function power(æ, xr y) = x.. To remove this bias, a debiasing step is performed in Adam as follows:
m t- 3-mm + (1-1111111 (1-
o' 320 + -2)
m - mammm/me m2/(11
(o)
and
E E v² /(1/11111 (1-oower((a (S2.e( ()),
before finally updating the network parameters: w' < w - cm/(Vv +) with small hyper- parameter e that prevents division by zero. The hyper-parameter a the learning rate.
As explained before, in deep RL we often minimize a sequence of loss functions, unlike the standard usecase of Adam above where minimize fixed loss. We now present the pseudocode of DQN with Adam. Notice that in this pseudocode we do not present the pieces related to the RL agent's s environmental interactions to primarily highlight pieces pertaining to how 0 and use t to denote the sequence of loss functions (outer iteration), and we use & to denote each gradient step (inner iteration) in fixed outer iteration One can present the code with these indices, but to simplify the presentation, we use a third index that is incremented with each gradient step.
updated. We
Algorithm Pseudocode for DQN with (resetting) Adam
Input: 0- T, K
Input: 3 $2, 0,
Set Adam's hyper-parameters
0, m 0,00
0
> Initialize Adam's 's internal parameters
for t = to T
1 do
w x ft
i = 0, m 0,
=0 0
b Reset Adam's 's internal parameters
for k to K
do g¹ + VH, (H.
+- i+1
m² +- 31m- + (1 - "B)gg' and o² +- B2ot-1 + +( (1-)2)(gg))
m+ (1-p//ee1ee i)) and +- v0/(11 /1 (1-powwrr(22.)) whit+ & wiik am/ /omm(VVom e)
D Adam's debiasing step
end for gc+1_ w
end for Return 0T-1
From the pscudocode above, first notice that the index is not reset, then the debiasing quantities power( (31, i) and power( (8,, i) quickly go to and so the debiasing steps will have minimal effect on the overall update, if at all. In absence of resetting, the optimization strategy could then be thought of as initializing the first-order (m) and the second-order (+) moment estimates each iteration by whatever their values were at the end of the previous iteration t 1. This seems like an arbitrary choice, that deviates from design decisions that were made by Adam [6].
This choice makes some sense if the optimization landscape in the previous iterations is similar to that of the current iteration, but it is not clear that this will always be the case in deep RL. In cases
3

---

Amidar
BeamRicer
Breakout
1000
CrazyCliber
DemonAttack
Gopher
Hero
11000
100
1300
Kangaroo
Phoenix
Seaquea!
Zacoon
1000
2003
ZIergg spticde ACOP
1900
133
503
Traning Praves (Wiloo)
Figure Performance of Rainbow with and without resetting the Adam optimizer and with the default value of K = 8000 on 12 randomly-chosen Atari games. All results are averaged over 10 random seeds. Resetting the optimizer often improves the agent's performance.
where this is not the case, the agent can waste many gradient updates just to "unlearn" the effects of the previous iterations on the internal parameters m and This is a contamination effect that plagues RL optimization as we later demonstrate, Fortunately, there is a remarkably easy fix. Note that the Adam optimizer is fully equipped to deal with resetting the moment estimates to 0, despite the bias it introduces, because the bias is dealt with adequately by performing the debiasing step. Finally, note that this is an inexpensive and convenient fix in that adds no computational cost to the baseline algorithm, nor does it add any new typer-parameter
Experiments
We chose the standard Atari benchmark [10] to perform our study, and also chose the popular Rain- bow agent [5], which fruitfully combined a couple of important techniques in learning the value function. This combination resulted in Rainbow being the state-of-the-an agent, one that remains competitive baseline to this day. We used the most popular implementation of Rainbow, namely the one in the Dopamine framework [1], and followed Dopamine's experimental protocol.
Note that other than the popular Dopamine baseline [1], we checked the second most popular imple- mentations of DQN and Rainbow on Github, namely Github. wwcobb.co//aaaa aiihhin/aanbbo,, and found that resetting is absent in these implementations as well. Lack of resetting is thus not an oversight of the Dopamine implementation, but the standard practice in numerous Deep RL implementations.
and
4.1 Rainbow with Resetting Adam
Here our desire is to investigate the effect of resetting the optimizer on the behavior of the Rainbow agent with its default Adam optimizer. For all of our ablation studies, including this experiment, we worked with the following 12 games: Amidar, Asterix, BeamRider, Breakout, CrazyClimber, DemonAttack, Gopher, Hero, Kangaroo, Phoenix Seaquest, and Zaxxon. Note that we will present comprehensive results the full set of 55 Atari games later. Limiting our experiments these 12 games allowed us to run multiple seeds per agent-environmen pair, and therefore, obtain statistically significant results.
From Algorithm 1, note that a key hyper- aper-parameter in the implementation of DQN (and by extension Rainbow) is the number of gradient updates (K) per iteration whose default value is 8000 in most deep RL papers 17, 17, 5]. We are interested to see the impact of this hyper-parameter on the performance of Rainbow with and without resetting. We first present the results for K 8000 Figure! 1. We can see that with K 8000 resetting the optimizer often results in improved performance.

---

We now change the value of K to understand the impact of changing K on this comparison. Low- ering K provides a smaller budget to the optimizer for solving each iteration. Notice that regardless of the value of K we performed the same number of overall gradient updates to the optimization (online) network across the entire training. Stated differently, a smaller value of K will correspond to a larger value T in Algorithm 1 because for the sake of fair comparison we always keep the multiplication of K x T fixed across all values of K in Rainbow with or without resetting.
We now repeat this experiment for multiple other values of K. Moreover, akin to the standard practice in the literature [18], rather than looking at individual learning curves per game, hereafter we look at the human-normallnn normalized performance the agents on all games, namely: ScoreAge Seote
Scoresturas Setorg To compare the performance of the two agents across all 12 games, we compute the median this number across the games and present these results for each value of K in Figure 2.
Ki 1000
K-2000
K-4000
K-6000
K-6000
Rainbow
Rainbow reset
Rainbow random reset
Traniig Fraines
Figure 2: A comparison between Rainbow with and without resetting on the 12 Atari games for different values of K. The Y-axis the human-normalized median. Observe that for all but K it is clearly better to reset the Adam optimizer. Notice, also that the best performance with resetting obtained by values of K that are much smaller than the default 8000 used in numerous papers.
From Figure 2 we see very clearly that Rainbow with resetting is dominating the original Rain- bow agent for all values K but K 1. Notice that in the extreme case of K 1 the Adam opti- mizer is in effect not accumulating any internal parameters because we reset after each step. There- fore, in essence, when K = the update would be more akin to using the RProp optimizer [19], which is not effective in our setting. Another interesting trend is that in fact when we reset the opti- mizer, smaller values of K than the default 8000 become the most competitive. This is in contrast to Rainbow without reset where the value of K does not influence the final performance. This is because resetting removes the initial bias in the moment estimates that would otherwise be present when no resetting is performed. The resetting agent does not have first "unlearn" this bias, and thus can reasonably solve each iteration with smaller K and move to the next iteration faster having accurately solved the previous iteration.
To further situate our results, we add another baseline where we reset the optimizer stochastically, meaning with some probability after each update the optimization network. Whereas in the orig- inal resetting case (red), we reset the optimize right at the beginning of each iteration, in this case just reset the optimizer after each update to the optimization parameter w, and the value of K determines the probability (K) with which we reset the Adam optimizer. With this choice of prob- ability, on expectation have 1 reset per iteration, but now resetting is stochastically performed any given step. We present this result in Figure where this additional baseline is indicated in blue.
We see that even randomly resetting the Adam optimizer provides some benefits relative to not resetting. However, we obtain the higher performance improvement by resetting the optimizer at the beginning of each iteration. Please see the Appendix for individual learning curves.
We further distill the above results by computing the area under the curves in Figure 2. Observe from Figure 3 that in the case of resetting (red), an inverted-U shape manifests itself with the best performance achieved by intermediate values of K. Also, to our surprise, the case of no-resetting (which is indicated in black and corresponds to the standard Rainbow agent), notice that perfor- mance is basically flat as a function of K. Even taken to the extreme with K = 1, we observe no performance degradation in Rainbow. This means that freezing the target network for 8000 steps,
5

---

3.5
under curve
Figure 3: Effect of resetting the Adam optimizer in Rainbow. Resetting the optimizer the beginning of each iteration yields the best performance. We also observe moderate improvements when resetting with probability 1/K after each update of the optimization (online) network w (blue). The original Rainbow agent without resetting (black) is dominated by its resetting counterparts.
2.0
Rainbow
1.0
Rainbow reset Rainbow - random reset
0.0
2000
4000
6000
8000
K
while harmless, provides no meaningful improvement the performance Rainbow. It is not clear to us, then, why the choice of K 8000 was made in the original Rainbow paper. One possible explanation that this choice is just a legacy from the previous papers. As we show later, in DQN it is important that large value of K is chosen, otherwise there will be extreme performance degra- dation, so it is possible that an assumption about the need for a large K in Rainbow was made, and the validity of this assumption
not tested thoroughly.
To demonstrate the contamination effect, and better understand where the benefit of resetting is coming from, we compute the cosine similarity between Adam's 's first-order moment estimate and the gradient estimate just before and after the update the target network. Viewed as measure of direction-wis wise similarity, the cosine of two vectors and v is defined by Cosine(u, u, v) = u //([[u]]]l Ieel)). general, Cosine(u, u, v) > if the two vectors are similar, and Cosine (u, v) Rs if there no similarity. In the context of Algorithm 1, this corresponds with computing the cosine similarity between g and m² twice per each iteration namely when = K (before target update) and when k = 0 (after target update). We now present this result in the following table.
Amidar
Asterix
BeamRider
Breakout averaged over 12 games
before
0.399
0.393
0.38
0.367
0.389
after
-0.001
-0.002
-0.002
0.003
-0.002
Table 1: Cosine similarity between Adam's first-order moment estimate and the gradient estimate just before and after updating the target network gt+1 + w. For each game, we compute the two quantities for all iterations f, and then average across all T iterations. We show the results for the first 4 games, and also report the average over all 12 games in the last column. Observe that the similarity near zero after the update, meaning that direction-wise the initial moment estimate has no similarity to the gradient estimates early in the iteration. Thus, in the absence of resetting, the initial moment estimate just contaminates Adam's 's moment accumulation procedure.
4.2 Experiments with Other Agent-Optimizer Combinations
We far demonstrated that we can improve value-function optimization in Rainbow using the idea of resetting Adam's moments. this effect specific to the Rainbow-Adam combination or does generalize to settings where we alternative optimization algorithms and even different agents?
To answer this question, we first investigate the setting where we use the RMSProp optimizer. First presented in lecture note by Hinton [15], RMSProp can be thought of as reduction of Adam where we only maintain a running average the second-order moment, but not the first. other words, RMSProp corresponds to Adam with 31 0 and also skipping the debiasing step for the second-order moment estimate. Similar to Figure 3, we ran Rainbow and RMSProp with and with- out resetting the same 12 games, computed the human-normalizee median, and then distilled the learning curves by computing their area under the curve for each value of K. Our result are presented in Figure 4 (top left). Notice that while overall we performance degradation when we move from Adam to RMSProp, we still observe that resetting the optimizer after updating the target network can positively affect the agent's score.
We next consider successor (rather than predecessor) of Adam, namely the Rectified Adam optimizer introduced by Liu et al. [20]. This recent variant of Adam is especially interesting because

---

Rainbow - RMSProp
Rainbow Rectified Adam
Rainbow RMSProp - reset
Rainbow Rectified Adam reset
3.5
area under the curve
0.8
8rea under the curve
3.0
2.0
0.2
1.5
500
2000
4000
6000
8000
K
DQN - Adam
RainbowPro - Adam
DQN Adam reset
RainbowPro - Adam reset
area under the curve
0.8
area under the curve
3.0
0.6
2.5
0.2
1.5
K
K
Figure 4: The positive effect of resetting in the context of Rainbow with the RMSProp optimizer (top left), Rainbow with the Rectified Adam optimizer (top right), DQN with the Adam optimizer (bottom left), and RainbowPro with the Adam optimizer (bottom right).
it controls the variance of the Adam optimizer early in training by employing step-size up in conjunction with a rectifying technique. With resetting, the optimization algorithm basically starts from scratch many times, and the rectification technique of Liu et al. [20] can be crucial in further improving performance. We present this results in Figure (top right). Observe that equipping the agent with resetting is having positive impact akin what showed in the case of R.Mpprop.
Pivoting into different agents, we present the next experiment where we used the Adam optimizer in conjunction with the DQN algorithm [7]. This agent is more primitive version of Rainbow, but we again can observe that a performance improvement manifests itself when we reset the optimizer per iteration (Figure 4 bottom left).
We next look at the RainbowPro agent introduced by Asadi al. [21], where they endowed Rain- bow with proximal updates. This ensures that at each iteration the optimization parameter gravitates towards the previous target parameter. Again, we employ resetting the context of their introduced agent. The result, which again shows the conducive impact of resetting, is presented in Figure (bottom right). Overall we can see the earlier result with Rainbow and the Adam optimizer gener- alizes to a couple of reasonable alternatives, thus supporting the hypothesis that our findings may remain valid for much broader set agent-optimizer combinations.
4.3 Resetting Individual Moments
So far we have shown that resetting optimizer moments positively impacts RL agents. Notice that in the specific case of Adam (and its variants),
detailed in Section 3, two disjoint moment estimates are being computed so a natural follow up question how important is to reset each individual moment. Our goal is to answer this question now.
To this end, we again performed the previous experiment on the same set of games. This time, we added two other resetting baselines, a first baseline in which we only reset Adam's 's first-order
7

---

Rainbow
Rainbow reset both moments
Rainbow reset 1st moment
Figure 5: A comparison between different choices of resetting in Adam's moments under the Rain- bow agent. The optimizer separately accumulates two moment estimates. this context, only re- setting the first-order moment (while not resetting the second-order moment) is not effective. In con- trast, only resetting the second-order moment is quite effective even when we do not reset the first- order moment. The best results are obtained when resetting both the first-order and the second-order moment.
Rainbow reset 2nd moment
3.5
area under the curve
3.0
2.0
500
2000
4000
6000
8000
K
moment estimate per iteration while not resetting the second-order moment, and the converse case where we only reset the second-order moment estimate. In Figure 5, compare the performance of these two baselines relative to the two earlier cases where we either never reset any moment, or reset both moments.
important ingredient here the default choice 31 and 33 in popular deep-learning libraries (such as Tensorflow and PyTorch), and by extension in the Dompamine framework and in our paper. In these settings, we have 31 0.9 and 32 = 0.999. This means that the second-order moment estimate is heavily leaning to the past gradient estimates, and rally arise. This is consistent with the result here that it is quite beneficial even when we only reset the second-order moment. More generally, the importance of resetting will be somewhat dependent on the specific 3 values being used. Note that one may think of our simple resetting strategy as an example of using adaptive 3 values, where use 31 B2 = 0 when moving into new iteration (k 0) and then falling back into the original 31 = 0.9 and 32 0.999 for k > 0. Here by adaptive mean that 3 values can depend on (or even 1) in arbitrary ways as opposed being constant across training. We may use other adaptive strategies that can potentially improve upon our simple resetting idea, but we leave this exciting direction for future work.
contamination will natu-
4.4 Comprehensive Experiments 55 Atari Games
So far focused on performing smaller ablation studies the subset of Atari games. We now desire to perform a more comprehensive experiment on the full 55 Atari games.
We now evaluate 3 different agent-optimizer combinations. As our benchmark, we have the original Rainbow algorithm with the Adam optimizer, without resetting, and the original value of = 8000. We refer to this as the standard Rainbow agent. Our next agent is Rainbow with the Adam optimizer and resetting. light of our results from Figure 3 we chose the value of K = 1000 for this agent. Finally, we have Rainbow with Rectified Adam and resetting. Similarly, and in light of Figure 4, we chose the value of K = 4000 for this agent. In Figure 6, we present the mean and median performance of the agents on 55 games.
we can see that resetting the Adam optimizer clearly boosting the performance of the Rainbow agent in terms of both the mean and the median performance of the agent across games. Moreover, we present another comparison in terms of the per-game asymptotic improvements the two resetting agents against the Rainbow agent in Figure 7.
From Figure
4.5 Continuous Control
For completeness we also conducted experiments in continuous action environments using MuJoCo physics simulator [22]. Specifically, we used Soft Actor-Critic (SAC) [23], an off-policy actor-critic algorithm with stochastic policy. It is worth noting that SAC and Rainbow/DQN differ in several aspects. Chief among these differences is that SAC utilizes soft target updates (also known Polyak updates) to update the target parameter, while Rainbow/DQN uses hard target updates (see Section

---

Rainbow
Rainbow
Rainbow Adam reset
Rainbow - Adam -- reset
Rainbow Rectified Adam reset
Rainbow - Rectified Adam - reset
human-normalzee median
2.5
human-normalized mean
15
2.0
1.5
10
1.0
0.5
30
120
30
60
90
120
Training Frames (Million)
Training Frames (Million)
Figure 6: A comparison between Rainbow without reset against two resetting agents, namely Rain- bow with restting Adam and Rainbow with resetting Rectified Adam. Results are averaged over 10 random seeds, where human-normalize the results and take their median (left) and mean (right) over 55 games. Resetting the optimizer is clearly favorable.
Soaes
Axar
Adartc
Denestiacc
Sdate-
UphDown
UsNDovn
Arraad
Arbald
Gocher
Naraehhe
KungF
Baroer
Time
Berzurk
Sweeridww
Porsan
Dopyr
Pong
Adlanida
A
Frodb by
Azarcita
Ventae
Aka
Jarsaband
Entum
Torsooon
Zheon
Dodng
Enduo-
ReadFlunne
Steekowl
Boang
Abort
Reamad
Cartell
Falivabelty
Ober
Aud
Satretone
FattegDerty
Kangans
Rhwaaad
Agreabond
Targlowerge
Ander
www.ron
flwiing
Band-at
Gowler
fowingg
Promseye
Comranad
ChopowCsmennss
60%
10%
0%
10%
00%
200%
50%
10%%
0%
10%
30%
2005
Figure 7: Asymptotic performance improvement of the resetting agents against Rainbow without resetting. In particular, we show improvements of Rainbow with resetting Adam against Rainbow (left), and Rainbow with resetting Rectified Adam against Rainbow (right).

---

2 for more details). SAC also uses the value of K = 1 so clearly it is not ideal to reset the optimizer after each step. Thus, it not clear when to reset the optimizer. We show some our results in the Appendix, where the Adam optimizer was reset for both the critic and actor networks every 5000 steps.
It is evident that resetting the optimizer is not as helpful here to the extent it was with hard target updates, but there is still small positive effect. Contamination is still present [4], but is not clear how to best address it in this case, and we may require more advanced techniques than our simple resetting idea. We leave the investigation of this to future work.
Related Work
In this paper, we showed that common approaches to function optimization could be viewed as a sequence of optimization problems. This view is occasionally discussed in previous work [24] but in different contexts, perhaps earliest in Fitted Value Iteration [25. 26]. Fitted Q Iteration [27, 28], and related approximate
algorithms [29, 30, 31]. More recently, Dabney et al. [32] presented the concept of the value improvement path where they argued that even in the single-task setting with stationary environments, the RL agent is still faced with sequence of problems, and that the representation-leane -earninn process can benefit from looking at the entire sequence of problems. This view was further explored to show that ignoring this non-stationary aspect can lead to capacity [33] and plasticity loss [34]. This capacity loss problem can be mitigated in various ways, such as by regularization [33, 35, 36), introducing new parameters [34), by using activation functions that are more conducive to learning a sequence of targets [37]. None of these works, however, considered the effect of RL non-stationarity the optimizer. Notice that this kind of non-stationarity is due to the solution not the environment, and stands in contrast the non-stationary setting that arises when the MDP reward and transitions change, a setting that is well-explored [38, 39, 40, 41, 42, 43].
the internal parameters of
To hedge against the contamination effect, we proposed to reset the optimizer per iteration. Re- setting the optimizer is a well-established idea in the optimization literature, and can be found for example in paper by Nesterov [44]. Adaptive versions of resetting
also common and perform well in practice [45, 46). Restarting the step-size (learning rate) also has some precedence in deep learning [47, 48, 49]. See also the standard book on large-scale optimization [50] well as a recent review of this topic in the context of optimization [51].
Resetting RL ingredients other than the optimizer is another line of work that is related to our paper. Earliest work due to Anderson [52], which restarts some network parameters when the magnitude of error is unusually large. A more recent example is Nikishin et al. [53] who proposed to reset some of the network weights, and the corresponding Adam statistics, to maintain plasticity. In contrast, motivation is resetting Adam statistics is to ensure we do not contaminate the moment estimation process when we move to a new iteration. Nikishin et al. conducted their experiments in the simpler continuous-control setting with policy-gradiem approaches where Polyak-base used and K = 1, while here we focus on resetting in the context of hard target updates with large values of K. Perhaps the most similar work to our paper is that of Bengio et al. [4] who noticed the harmful effect of contamination, and proposed a solution based on computing the Hessian and Taylor expansion of the loss, one that is unfortunately too expensive be applied in conjunction with deep networks owing to their large number of weights.
updates are often
Conclusion
In this paper, we argued that value-function optimization could best be thought of as solving a se- quence of optimization problems where the loss function changes per iteration. Having adopted this view, we argued that it is quite natural to reset the optimizer, which can combat the contamination effect plaguing deep RL in the absence of resetting.
more general conclusion of our work is that we must obtain a deeper understanding of the internal process of optimization algorithms in order to unleash their true power in the context of deep RL. Standard results from optimization in supervised learning may not always readily transfer to RL. It important to keep in mind that these optimization techniques were often designed for setting that are meaningfully different than RL, and thus some of the pitfalls of RL training can plague them when applied without proper consideration.
10

---

References
[1] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Belle- mare. Dopamine: A Research Framework for Deep Reinforcement Learning. arXiv, 2018.
[2] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv, 2016.
[3] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[4] Emmanuel Bengio, Joelle Pineau, and Doina Precup. Correcting momentum in temporal dif- ference learning, 2021.
[5] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining im- provements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
[6] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- fional Conference Learning Representations, 2015.
[7] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 2015.
[8] Hamid Reza Maei. Gradient lemporal-differene learning algorithms. PhD thesis, University of Alberta, 2011.
[9] Kavosh Asadi, Shoham Sabach, Yao Liu, Omer Gottesman, and Rasool Fakoor. TD conver- gence: An optimization perspective. Advances in Neural Information Processing Systems, 2023.
[10] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International conference on machine learning, 2017.
[11] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXivv, 2015.
[12] Kristopher De Asis, J Hernandez-Garvin G Holland, and Richard Sutton. Multi-step reinforce- ment learning: A unifying algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[13] Yunhao Tang, Remi Munos, Mark Rowland, Bernardo Avila Pires, Will Dabney, and Marc Bellemare. The nature of temporal difference errors multi-step distributional reinforcement learning, Advances in Neural Information Processing Systems, 2022.
[14] Kavosh Asadi, Neev Parikh, Ronald E Parr, George D Konidaris, and Michael L Littman. Deep radial-basis value functions for continuous control. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[15] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 2012.
[16] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 1964.
[17] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double g-leamnig. In Proceedings of the AAAI conference on artificial intelligence, 2016.
[18] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Ducling network architectures for deep reinforcement learning. In International conference on machine learning, 2016.
[19] Martin Riedmiller and Heinrich Braun Braun. A direct adaptive method for faster backprop- agation learning: The rprop algorithm. In IEEE international conference on neural networks, 1993.
11

---

[20]
Liyuan Liu, Haoming Jiang, Pengcheng He. Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv,, 2019.
[21]
Kavosh Asadi, Rasool Fakoor, Omer Gottesman, Taesup Kim, Michael Littman, and Alexan- der Smola. Faster deep reinforcement learning with slower online network. Advances in Neural Information Processing Systems, 2022.
[22] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 0026-5033. IEEE, 2012.
[23] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2018
[24] Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learning. In Learning for Dynamics and Control, 2020.
[25] Geoffrey J Gordon. Stable function approximation in dynamic programming. In Machine learning proceedings 1995. pages 261-268. Elsevier, 1995.
[26] Damien Ernst Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
[27] Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural rein- forcement learning method. In Machine Learning: ECML 2005: 16th European Conference on Machine Learning, 2005.
[28] András Antos, Csaba Szepesvári, and Rémi Munos. Fitted q-iteration in continuous action- space mdps. Advances neural information processing systems, 2007.
[29] Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
[30] Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 2003.
[31] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Rein- forcement learning: State-of-the-arl, 2012.
[32] Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G Bellemare, and David Silver. The ialoe-improvement path: Towards better representations for reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[33] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv, 2022.
[34] Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and Andre Barreto. Deep reinforcement learning with plasticity injection. In Workshop on Reincarnating Reinforcement Learning at ICLR, 2023.
[35] Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regen- erative regularization. arXiv preprint arXiv:2308.1 1958, 2023.
[36] Shibhansh Dohare, Juan Hermandez-Garcit Parash Rahman, Richard Sutton, and A Rupam Mahmood, Loss of plasticity in deep continual learning. arXiv preprint, 2023.
[37] Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv, 2023.
[38] Sindhu Padakandla, Prabuchandran KJ, and Shalabh Bhatnagar. Reinforcement learning algo- rithm for non-stationary environments. Applied Intelligence, 2020.
[39] Erwan Lecarpentier, David Abel, Kavosh Asadi, Yuu Jinnai, Emmanuel Rachelson, and Michael I. Littman. Lipschitz lifelong reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, 2021.
12

---

[40] Yash Chandak. Reinforcement Learning for Non-stationary problems. PhD thesis, University of Massachusetts Amherst, 2022.
[41] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual rein- forcement learning: review and perspectives. Journal of Artificial Intelligence Research, 2022.
[42] Jelena Luketina, Sebastian Flennerhag. Yannick Schroecker, David Abel, Tom Zahavy, and Satinder Singh. Meta-gradients in non-stationary environments. In Conference on Lifelong Learning Agents, 2022.
[43] David Abel, André Barreto, Benjamin Van Roy, Doina Precup, Hado van Hasselt, and Satinder Singh. A definition of continual reinforcement learning. arXiv, 2023.
[44] Yurii Nesterov. Gradient methods for minimizing composite functions. Mathematical Pro- gramming, 2012.
[45] Brendan O'Donoghue and Emmanuel J. Candès. Adaptive restart for accelerated gradient schemes. Foundations of Computational Mathematics, 2012.
[46] Bao Wang, Tan M. Nguyen, Andrea L. Bertozzi, Richard G. Baraniuk, and Stanley J. Osher. Scheduled restart momentum for accelerated stochastic gradient descent, 2020.
[47] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. ArXiv, 2017.
[48] Nya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. International Conference on Learning Representations, 2017.
[49] Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv, 2018.
[50] Amir Beck. First-order methods in optimization. SIAM, 2017.
[51] Sebastian Pokutta. Restarting algorithms: Sometimes there is free lunch. Integration of Al and OR Techniques in Constraini Programming, 2020.
[52] Charles Anderson. Q-learning with hidden-unit restarting. Advances in Neural Information
Processing Systems, 5, 1992.
[53] Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference Machine Learning, 2022.
13

---

7 Appendix
7.1 Hyperparameters
We report the hyper-parameters used in experiments.
Rainbow-Adam hyper-parameters (shared)
Replay buffer size
200000
Target update period
variable (default 8000)
Max steps per episode
27000
Batch size
64
Update period
4
Number of frame skip
Update horizon
3
6-greedy (training time)
0.01
e-greedy (evaluation time)
0.001
e-greedy decay period
250000
Burn-in period Min replay size
20000
Discount factor (7)
0.99
Adam learning rate
6.25 x 10-5
Adam
0.00015
Adam 31
0.9
Adam 32
0,999
Rainbow-RMSProp
RMSProp P RMSProp
0.9
10-7
Rainbow-Rectifen Adam
warm up proportion
0.1
DQN-Adam typer-parameters
Adam learning rate
2 x 10
SainbowPro-Adan hyper-parameter
proximal parameter (c)
0.05
Table 2: Hyper-parameters used in our experiments.
14

---

7.2 Complete Results from Section 4.1 We show individual learning curves for Rainbow-Adan with and without resetting for different values of K.
Amidar
Asterit
BeamRicer
Breakout
CrazyClibber
DemonAttack
Gopher
Haro
11000
0200
1000
4:03
100
120
K03
Kangeroo
Phoerix
Seaqueoi
Zacoon
2000
avargge opticd SOOR
1000
120
1000
Rarbow
Rarbow
Trarep (Wiket
Figure 8: Learning curves for Rainbow-Adan without resetting, resetting after each target-network update, and with random resetting. In this case, K = 8000.
usc
CrazyClimber
DemanAllack
Gopher
Haro
1100
VMc
3000
100
2003
Kangaroo
Seequist
Zacxon
1509
Za rbow
Ranbow reset
50
Ranboow rendomm
Figure 9: K = 6000.
15

---

Amidar
Adtarix
BeamRiker
Braakout
2000
CrazyClibber
DemonAttack
Gophor
Hero
2000
0000
11000
1900
4003
Dx0
1000
300
2003
X000
Kangaron
Phoerix
Seaqueot
Zacoon
Traniig Franee (Wile))
Figure 10: K 4000.
Amidar
Asterix
BeamRicer
Breakout
1200
1000
CrazyClmber
DemonAttack
Gopher
Hero
1000
200C
1,500
1,000
1.00
Kangeroo
Phoenix
Sasquia!
Zacoon
3000
eoeTg SpRUCe SOON
1200
290
Trereg (Wiler)
Figure 11: K 2000.
16

---

Amidar
Astarix
BeamRiker
Breakout
1000
CrazyClibber
DemonAttack
Goophor
Herr
2000
2000
1200
1200
Kangaron
Phoerix
Seaqueot
Zucoon
2000
KHROOO MSROCE SCOP
1000
1200
Traning Franer (Wile))
Figure 12: K 1000.
Amidar
Asterix
BeamRicer
Breakout
woc
CrazyClmber
DemonAttack
Gopher
Hero
1.00
1.00
Kangaroo
Phoenix
Saaquia!
Zacoon
2014
2000
1009
1200
Trereg (Wilrr)
Figure 13: K 500.
17

---

Amidar
Asterix
DeamRicer
Breakout
CrazyClinber
DemonAttack
1.00
1200
100
Phoenix
Seaquest
Zacoon
Wuregg wotocde POOTE
Traning Praves (WSloO)
Figure 14: K 1. With this value of K, two of the agents (Rainbow - reset and Rainbow - random reset) become identical.
18

---

7.3 Complete Results from Section 4.2
We now show complete results from Section 4.2 starting with Rainbow RMSProp.
Amidar
Astenix
DeamRicer
Breakout
300
200
Crazylimber
DemonAltack
Gopher
Hero
1300
1300
Kangeroo
Phoenix
MMC
Seaquest
Zacoon
SHOTOE EOROR SCOP
Traning Franee (Wilre)
Figure 15: Performance of Rainbow with and without resetting the RMSProp optimizer and with fixed value of K = 8000 on 12 randomly-chosen Atari games.
CruzyClinber
DemonAllack
Gopher
Haro
450
1000
Kangaroo
Seequis!
Zaccon
Hamrocw H1M22
Rairscer RMSPPro rooer
Figure 16: K 6000.
19

---

Anidar
CrazyClimber
Gopher
130
1000
200
Kangaroo
Seaquest
390
201
Thanoow HMMtSPopp
Htaroow HEMESProp neoert
Traning Fraver (10800))
Figure 17: K = 4000.
Amidar
CrazyClinber
Demon&llack
Gopher
Haro
1.00
130
340
Kangaroo
Seequis!
Zacxoo
200
Rairscar RMEPPHO
Figure 18: K = 2000.
20

---

09
CrazyClibber
Gopher
11000
1.000
1000
200
Kangaroo
Seaquest
Hagnntuu CIMC Toanoow TIMS
Traning Fraver (U0lio))
Figure 19: K = 1000.
Amidar
Acterix
DeamRicer
Breakout
20
CrazyGlmber
DemonAttack
Gopher
Hero
Kangaroo
Phoerix
Seaquest
Zacxon
200
201
Rugarta
Traniig Franes (VSBor)
Figure 20: 500.
We now take the human- normalized median on 12 games and present them for each value of K.
21

---

K=500
K=1000
K=2000
C2
04
K=4000
K=6000
K=8000
PLSTION TOTTOETTT Bedd medan
04
Rambbw RMSPrOP
Rainbow RMSProp reset
Taining (Milion)
Figure 21: A comparison between Rainbow with and without resetting RMSProp on the 12 Atari games for different values of K.
22

---

We now look at Rainbow with the Rectified Adam optimizer.
Amidar
Asterix
DeamRicer
Breakout
100
CracyClmber
DemonAttack
Gopher
Hero
2000
0200
Kangaroo
Phoenix
Seaques!
Zacooo
SHOTT SRRCO SCOM
2003
1002
Traning Franee (Wilor)
Figure 22: Performance of Rainbow with and without resetting the Rectified Adam optimizer and with fixed value of K = 8000 on 12 randomly-chosen Atari games.
vec
CrazyClimber
DemanAllack
Gopher
Haro
po0
2015
vis
1000
1200
Kanganoo
Seequiis!
Zaccoo
Now Reriified Allam
bow
Reciifeed Adamm teeet
Figure 23: K = 6000.
23

---

Amidar
CrazyClimber
Gopher
0003
11:00
3000
5.000
WMc
2003
Kangaroo
Seaquest
Zaxxon
X000
3003
fow -Deciiied and hoam
bow --eciiee hoam reaed
Traning Fraver (10800))
Figure 24: K = 4000.
Amidar
BeamRicer
voe
CrazyClimber
DemanAllack
Gopher
Haro
1200
1000
400
100
Kangaroo
Seequis!
Zaccoon
4000
Woog BOOSS
1200
fow
Rectified Alamm
bow Reciifed Atam teeet
Figure 25: K = 2000.
24

---

Amidar
2000
CrazyClibber
Gopher
3000
2013
4033
Kangaroo
Seaquest
Zacxon
10:00
1Mc
1000
100
NoW Teeciliee hoam
bow
--eciteee hoam FRaed
Traning Fooiee (108in))
Figure 26: K = 1000.
Amidar
Acterix
DeamRicer
Breakout
usc
CrazyGlmber
DemonAttack
Gopher
Hero
4000
1.000
2000
2000
1000
100
Kangaroo
Phoerix
Seaquest
Zacoon
and Acam
Tranning Pranus (VSBlo))
Figure 27: 500.
We now take the human- normalized median on 12 games and present them for each value of K.
25

---

K=500
1000
K=2000
K=4000
K=6000
K=8000
PummO -OTOOIOOT Bodd median
Rankow Rectfled
Rambow Recified Adam resel
Taining Framos (Milion)
Figure 28: A comparison between Rainbow with and without resetting Rectified Adam on the 12 Atari games for different values of K.
26

---

We now move to the case of DdN-Adam.
Amidar
Aster'
BeamRicer
Breakout
20
CrazyClibber
DemonAttack
Gopher
Hero
11000
100
1000
Kangaroo
Phoerix
Seequest
Zacaon
Traning Prames (Ville))
Figure 29: Performance of DQN with and without resetting the Adam optimizer and with fixed value of K = 8000 on 12 randomly-chosen Atari games.
Amidar
CrazyClimber
Deman&ltack
Gopher
Haro
1200
100
Kangaroo
Seequii!
Zacxon
wurge OOTCUE SCORS
DON Alam
200
DOK Adam
Figure 30: K 6000.
27

---

Amidar
CrazyClimber
Gopher
5300
wX0
300
Kangaroo
Seaquest
OCN hcart
Traning Friver (U0lio)
Figure 31: K = 4000.
Amidar
CrazyClinber
Deman&llack
Gopher
Haro
120
3.00
Kangaroo
Seequist
Zacxon
1009
Figure 32: K = 2000.
28

---

CrazyClimber
Gopher
2000
100
Kangaroo
Seaquest
Zacxnn
1200
Traning Fraver (U0l0e))
Figure 33: K = 1000.
Amidar
Asterix
DeamRicer
Breakout
20
CrazyCliber
DemonAttack
Gopher
Hero
Kangaroo
Phoerix
Seaquest
Zacion
1200
Training Pranus (VBBor)
Figure 34: 500.
We now take the human-normalinae median on 12 games and present them for each value of K.
29

---

K=500
K=1000
K=2000
01
K=4000
K=6000
K=8000
1.0
PUATO Hor----ooaaaa Bond medan
DON Adam resel
Tainiig Framos (Milion)
Figure 35: A comparison between DQN with and without resetting Adam on the 12 Atari games for different values of K.
30

---

We now move to the Rainbow Pro agent with Adam.
Amidar
Astenix
DeamRicer
Breakout
1000
CrazyClmber
DemonAttack
Gopher
Hero
2.00
1000
11000
4000
5.00
0209
Kangaroo
Phoenix
Seaques!
Zacoon
MM
SHETTg SARCO SCOP
1000
2000
200
1200
2003
Traning Franeet (Vilor)
Figure 36: Performance of Rainbow Pro with and without resetting the Adam optimizer and with fixed value of K = 8000 on 12 randomly-chose Atari games.
110
CruzyClimbbe
DemanAllack
Gopher
Haro
1.00
0.00
Kangaroo
Seequii!
Zaccoon
4000
Pai PartocPPu Alam
Rai-tcaPto- Adam meet
Figure 37: K 6000.
31

---

Amidar
BeamRicer
CrazyClimber
Gopher
1000
1200
Kangaroo
Seaquest
Zaxxon
1000
Toe Acam
Paentoofii Adam readd
Traning Friver (10lio))
Figure 38: K = 4000.
Amidar
20w
20000
CruzyClimber
DamanAllack
Gopher
Haro
2,00
4000
1000
500
0209
Kangaroo
Seequis!
Zacxon
2000
1000
RairtcaPro- Adam meet
PartcePPu- Alam
Figure 39: K = 2000.
32

---

Amidarr
CrazyClimber
Gopher
2.00
2006
4003
1200
Kangaroo
Seaquest
Zaxoon
1000
12000
Toe Acam
Paentooffie Adam reaed
Traning Friver (10lio))
Figure 40: K = 1000.
Amidar
Acterix
DeamRicer
Breakout
vow
CrazyGlmber
DemonAttack
Gopher
Hero
2.00
1000
4000
1,000
400
Kangaroo
Phoerix
Seaquest
Zacoon
2000
1000
bowttro Acam boitbooPh-----
Traniig Franus (VSBlo)
Figure 41: 500.
We now take the numan-normalin median on 12 games and present them for each value of K.
33

---

K=500
1000
K=2000
K=4000
K=6000
K=8000
NONTTTTT fodd median
RainbowPro
RainbowPro Adam resel
Taining (Milion)
Figure 42: A comparison between Rainbow Pro with and without resetting Adam on the 12 Atari games for different values of K.
34

---

7.4 Complete Results from Section 4.3
We now show the full learning curves pertaining to Section 4.3 where we studied resetting individual moments.
Anidarr
10000
CruzyClibber
DemonAltack
Gopher
Hero
2010
2000
5100
0200
Kangaroo
Phoerix
Seaqueot
Ziecoon
1000
1000
Sannew both momoir
Ruroce lesut momeent
Raitie lopul momennt
Figure 43: Learning curves for Rainbow-Adam without resetting, resetting after each target-network update, and with random resetting. this case, 8000.
Touts
CrazyClimber
DemanAllack
Gopher
Haro
1100
2000
20m
Kangaroo
Seequiss
Zacxon
2000
2000
100
1.00
bennow seeet both mamans
Rarrice reset momentt
Rairocer seeet 2nd monnent
Figure 44: K = 6000.
35

---

Amidar
1000
CrazyClimber
Gopher
2000
2000
Bangaaoo
Seaquest
1000
1000
wnocer both moroir
tamocer neant tat moment
Hamooce neant 2ndd moment
Traning Foaiee (U9lio))
Figure 45: K = 4000.
Amidaar
1oe
CrimeClimber
DemanAllack
Gopher
Haro
020
2000
3000
10m
1200
Kangaroo
Seequiis
Zaccoon
5000
2009
Anncer seeet both momans
Rarocc reset moment
Rairocer seset 2nd monnent
Figure 46: K = 2000.
36

---

Amidar
BeamRicer
CrazyClimber
Gopher
11200
1.00
1200
Kangaroo
Seaquest
Zaxoon
1000
1200
MMC
Wnoc neaet both mortint
Tanocer neent tol momend
Iaaroce neant 2ndd moment
Traning Fraiee (198lo))
Figure 47: K = 1000.
Amidar
Actorix
DeamRicer
Breakout
CrazyGlibber
DemonAttack
Gopher
Hero
1100
1000
10W
100
Kangaroo
Phoerix
Seaquest
Zacoon
bith
Marocw reast
morrart
Traniig Pranus (VSllo)
Figure 48: 500.
We now take the numan-normalin median on 12 games and present them for each value of K.
37

---

K=500
1000
K=2000
K=4000
K=6000
K=8000
POrIA comm BEO0 median
Rainer
Rambb sw reset both moments
Rainbww
Rainbow reset 2nd moment
Taining (Milion)
Figure 49; A comparison between no resetting, resetting both moments, and resetting individual
moments in Rainbow-Adam.
38

---

7.5 Complete Results From Section 4.4
We now show full learning curves for all 55 Atari games and over 10 random seeds. We benchmark three agents: the default Rainbow agent from the Dopamine (no reset), Rainbow with resetting the Adam optimizer, and Rainbow with resetting the rectified Adam optimizer.
Alien
Amidar
Assault
5900
3000
1900
+900
2012
000
Asto ftx
Aeterodd
Atlentis
Bankffeist
1300
BattleZone
Beam
Berzerk
Bowling
1000
2000
2002
2000
30
Boxing
Breakout
Carnival
Chopper Command
1000
2012
2000
100
CrazyClimber
DemonAttack
Endura
FishingDerby
1000
1900
2014
0000
Freeway
Frostblle
Gapher
Graviar
2000
2000
64
Hore
IceHockey
Jamesbond
JournayEscape
4000
900
-15
400
0
30
120
Figure 50: Learning curves for 55 games (Part 1).
39

---

Kangaroo
Krul
KungFuMMaster
MonlezumaiReeengg
1000
Name Thistamme
Phoenix
200000
128
Pong
Ponyan
PrivatEEe
Qbert
2008
+00
-10
Robotank
Riverraid
RoadRunner
Seaqueest
1000
1000
400
90
Solaris
Spacelnvadere
StarGunnar
TimePioot
1900
10000
2010
7000
500
8008
590
200
Tutankham
UpNDown
Venture
VideoPinball
250
4900
WizardOWWor
YarsRevenge
Zaxison
WeeTER aptande SHCTO
24100
2000
Adoe
Traniig Franso pallon
Figure 51: Learning curves for 55 games (Part II).
40

---

7.6 Complete Results From Section 4.5
finally present results on continuous control task with soft actor critic (SAC) and the Adam optimizer, where reset the optimizers every 5000 steps. Note that, in contrsat Atari and Rainbow, the target parameter is updated using the Polyak strategy, so it is less clear when to reset the optimizer. Thus chose the simple strategy of resetting the optimizer every 5000 steps. We leave further exploration of resetting with Polyak updates to future work.
x1101
HalfCheetah-v3
X10'
Walker2d-v3
Average Return
1.75
Average Return
1.25
1.00
0.50
reset
resel
0.00
Time
Time
Hopper-v3
x10
Humanoid-v3
Average Return
Average Return +
3
with
SAC renet SAC
SAC
Time
x10
Time
X10
Ant-v3
Average Return
BAC with reset BAC
6
Time
Figure 52: A comparison between Soft Actor-Critie (SAC) with and without resetting Adam on the standard MuJoCo tasks. this study, both the actor and critic optimizers are reset every 5000 steps. The results averaged over 10 different seeds.
41


Reviews:
[{"review_id": "QJXT14vw1f", "reviewer": "Reviewer_Ks5C", "summary": "The paper studies optimization in value-based deep reinforcement learning. The key insight is that when using target networks for action-value function training, changes in the target parameters yield a change in the optimization problem the online parameters are solving. Because of that, the authors argue that preserving the adaptive optimizer statistics (e.g. of Adam) might or might not be desirable. The paper then studies the effect of resetting the optimizer state after (hard) target updates mostly using the Rainbow algorithm on Atari games as a testbed yielding a slight positive aggregate improvement.", "questions": "Target network parameter updates indeed change the loss landscape that the online parameters are navigating. In addition to that, updating the replay buffer changes the distribution of inputs and hence the optimization problem for online parameters. Do you have ideas on how an optimizer could be changed to adapt to the input shifts?", "limitations": "“We hypothesize that this can contaminate the internal parameters of the employed optimizer in situations where the optimization landscape of the previous iterations is quite different from the current iteration.” (L9) The reviewer didn’t find an empirical verification of this assumption.\n\nMany deep RL algorithms use moving average target updates after each step instead of periodic hard updates. The authors demonstrate preliminary evidence that in soft actor-critic that uses such a practice, the optimizer resets do not improve the performance. Having said that, the reviewer appreciates the transparency about the negative results.\n\nAgain, one of the limitations is that in some environments resetting the optimizer yields negative results. It implies that a better alternative could be triggering the optimizer reset using a criterion (e.g. based on a measure of the loss landscape change / by performing a lookahead and assessing whether the reset was helpful)", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength of the paper is the simplicity of the contribution; the paper is well-written and easy to follow, and the method is motivated and described well. The experimental protocol is solid: it uses the full set of 55 Atari games and a standard Rainbow implementation.", "weaknesses": "The main weakness of the paper is the mixed empirical results. Granted, the median human-normalized performance improves from ~1.75 to ~2.25, however, per-game effects from resetting the optimizer are highly heterogeneous, yielding performance deterioration in ~14 environments. The soundness of the paper could have been higher if, at least, an explanation (supported by evidence) for the negative effects was given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent carefully reviewing the paper and for appreciating our work. Please find below some clarification regarding your questions.\n\n- \"Granted, the median human-normalized performance improves from 1.75 to 2.25, however, per-game effects from resetting the optimizer are highly heterogeneous, yielding performance deterioration in 14 environments.\"\n\nWe ask the reviewer to consider the fact that resetting the optimizer is an extremely simple modification of the original algorithm in that it is adding no new hyper-parameter and/or computation to the baselines. We believe that due to the simplicity of resetting and the fact that it is a very natural thing to do, the bar should not be set very high to the point where we expect dramatic improvement on all games. In order words, simplicity of the approach is in our view a strength of the approach and not a bug, and so the improvements should be celebrated more given this simplicity.\n\nAlso, by looking at the original Rainbow paper, we observe that the positive impact due to each individual component of Rainbow is not that high, and so in light of that, we are actually positively surprised that the simple resetting idea can be so effective. We did not explore adaptive versions of resetting that could have been even more dominant relative to not resetting, but we think the fact that the simple resetting is so effective can instigate future work that can come up with even more performant resetting strategies.\n\n- In addition to that, updating the replay buffer changes the distribution of inputs and hence the optimization problem for online parameters. Do you have ideas on how an optimizer could be changed to adapt to the input shifts?\n\nThis is a very good point, and we indeed agree that the online RL problem (in contrast to offline RL) is non-stationary even from one time-step to the other. We did not explore approaches that account for shifts in inputs as they were beyond the scope of this result. That said, the mere fact that the reviewer is, akin to us, interested in this question is a testament that our current results are insightful, thought-provoking, and would instigate further research in this direction.\n\n- ``We hypothesize that this can contaminate the internal parameters of the employed optimizer in situations where the optimization landscape of the previous iterations is quite different from the current iteration.'' (L9) The reviewer didn’t find an empirical verification of this assumption.\n\nPlease take a look at the new cosine similarity experiment in the attached pdf. We see very little cosine similarity between the gradient and Adam's first moment estimate immediately after moving into a new iteration, which empirically verifies the contamination effect."}, {"review_id": "1Jd7Bp4thj", "reviewer": "Reviewer_5d1P", "summary": "This paper questions the standard use of Adam-type optimizers in deep RL. The paper argues that solution methods in deep RL are best thought of as solving a sequence of optimization problems. And that the standard use of optimizers leads to \"contamination\" of the optimizer's internal parameters. The paper then proposes to reset the optimizer's internal parameters to fix this \"contamination.\" Finally, the experiments on Atari show that resetting the optimizer's internal parameters leads to significant performance improvement.", "questions": "What would be a good definition of \"contamination\"?", "limitations": "No confidence intervals are reported for any experiment in the paper. I recommend the authors report the 95% bootstrapped confidence intervals for their results. [2] and [3] provide good guidelines for properly reporting experimental results in deep RL.\n\n\n[2] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.\n[3] Patterson, A., Neumann, S., White, M., & White, A. (2023). Empirical Design in Reinforcement Learning. arXiv preprint arXiv:2304.01315.\n\nEDIT:\n\nI have updated my score based on the new results provided by the authors.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper is that the key idea of the paper, resetting optimizer parameters at the beginning of each iteration, is simple and effective. I liked the general theme of the paper, i.e., we need to understand better the tools we borrow from other fields. The paper is well-written and easy to understand, making it accessible to a wide audience. The experiments in section 4.3 are performed on 55 Atarti games with ten seeds each, which might mean the results are statistically significant. Resetting seems to be beneficial for multiple optimizers like RMSprop, Adam and Rectified Adam. ", "weaknesses": "The paper has two major weaknesses:\n1. The paper claims at many points that a \"contamination\" effect plagues RL (for example, lines 107-109) and that many updates are wasted to unlearn the effects of the previous iteration. However, the paper does not describe what exactly this \"contamination\" means, and neither does it show the presence of any \"contamination.\" All the paper shows is that there is a performance boost when we reset the optimizer's internal parameters. This performance boost is not direct evidence of contamination from iteration to iteration.\n\nHowever, this weakness can be easily overcome. The paper first needs to contain a definition of \"contamination,\" maybe the authors mean that the internal parameters($m$ and $v$) are too far away from their true values at the beginning of each iteration. One way to measure this difference could be to measure the cosine similarity between the current value of $m$ and the true value of $m$. The true value can be measured by taking the gradient of all the samples in the buffer and taking steps using that gradient. A large difference between the true and current value of $m$ would mean contamination. The paper also suggests that this contamination is particularly large at the beginning of each iteration compared to a random time in the learning process. Again, this can be easily shown by showing that the difference in true $m$ and current $m$ is larger at the beginning of the iteration compared to any random time in the learning process. \n\n2. None of the results in the paper except the ones in section 4.3 are statistically significant. The paper only shows results for a single random seed. I should note that the authors are aware of this weakness (line 138). The best way to look at the current experiments in Sections 4.1 and 4.2 is that they are used to tune hyper-parameters for the experiments in Section 4.3. The authors might be limited in their computational resources, but in that case, it is better to present statistically significant results in smaller environments like MinAtar[1] than unreplicable results in a big environment. \n\nOther than these two main problems, there are a few other minor issues in the paper.\n1. The update equations for Adam in Section 3 are wrong. Instead of using $m$ and $v$ for the final update, new variables $\\hat{m}$ and $\\hat{v}$ are used. See to the original Adam paper for the correct equations.  \n2. Line 176 says that $K=1$ corresponds to vanilla gradient descent. But, that is not true. For $K=1$, the update is similar to the Rprop optimizer, not SGD. For $K=1$, the update only takes into account the sign of the partial derivative but not its magnitude. \n3. The value of $K$ is not properly tuned. In Figure 4, the difference between $K=1000$ and $K=500$ is insignificant. So, the optimal value of $K$ could be smaller. I suggest the authors also try smaller values of K, like 250, 125, etc. \n\nI like the ideas presented in the paper. However, I can not recommend accepting the paper in its current form in light of these weaknesses. \n\n[1] Young, K., & Tian, T. (2019). Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer appreciating our idea. In what follows, we address the particular weaknesses and questions raised.\n\n- The paper does not describe what exactly this contamination means, and neither does it show the presence of any contamination.\n\nPlease take a look at the new cosine similarity experiment in the attached pdf. We see very little cosine similarity between the gradient and Adam's moment estimate immediately after moving into a new iteration. If the moment estimate was not stale, you would expect them to form a low angle with the gradient, and therefore give us high cosine similarity, but this is demonstrably not the case for all games we tested. \n\nThanks for your suggestion in conducting the cosine similarity experiment, though notice that computing the true value of m like you suggested is extremely expensive, and so inspired by your comment, we conducted a similar experiment where we measured the cosine similarity between the Adam's first moment estimates and the stochastic gradient computed immediately after moving to the next iteration.\n\n- None of the results in the paper except the ones in section 4.3 are statistically significant. \n\nThat is a fair criticism, and in light of your concern, we updated all of our results to include 10 random seeds. We hope that the reviewer considers increasing their score since they stated using 1 random seed as one of the main weaknesses of our work. To be fully transparent, we were limited by compute at the time of submission and made the strategical decision to use most compute for the experiment with all 55 Atari games.\n\n- The update equations for Adam in Section 3 are wrong. \n\nYou are correct, but we also want to reassure the reviewer that we know the Adam update. We just miss $\\hat m$ and $\\hat v$. This was simply a typo which we have addressed in the main text. Thanks for your comment nonetheless.\n\n- Line 176 says that  corresponds to vanilla gradient descent. \n\nYou are correct, and we will fix that.\n- No confidence intervals are reported for any experiment in the paper. \n\nPlease see the attached pdf for the updated results. We now report confidence intervals so we hope the reviewer reconsiders their evaluation.\n\n-  I suggest the authors also try smaller values of K, like 250, 125, etc.\n\nThis is a good suggestion and we are happy to add this in the final paper if accepted. For the rebuttal we prioritized making the existing experiments statistically significant. We really hope the reviewer considers increasing their score and help us get the paper to the finish line and we will be happy to then use our time to try values of K between 500 and 1.\n\n- What would be a good definition of ``contamination''? \n\nA definition we have adopted is small cosine similarity between the first moment estimate and the gradient vector immediately after updating the target network. If the two vectors are very rarely aligned, the moment estimate is in fact just misleading the agent, and so we call this a contamination effect.\n\n"}, {"review_id": "dA3l8MysPx", "reviewer": "Reviewer_59Jp", "summary": "The paper addresses the issue of using modern optimizers, such as Adam, which maintain internal parameters that are updated over time, potentially contaminating the optimization process. To mitigate this effect, the paper proposes a simple strategy of resetting the internal parameters of the optimizer at the start of each iteration. Empirical investigations using different optimizers and the Rainbow algorithm show that this modification enhances the performance of deep reinforcement learning on the Atari benchmark.", "questions": "- How does this method compare to other resetting approaches in terms of effectiveness?\n- What is the level of statistical significance observed in the results?\n- Why is resetting not effective for continuous control tasks?\n- Are there any experiments demonstrating the impact of contamination on the tasks discussed in this paper?\n- What are the consequences of reducing the frequency of optimizer resets beyond K=8000?\n- How can an optimal value for $K$ be determined?\n- Which ADAM/optimizer parameters are relevant when performing resets, i.e. have an effect when reset?\n- Are the observed effects still present when modifying the ADAM/optimizer hyperparameters?\n- Do different loss functions used in various DQN versions (e.g., MSE, Huber, Quantile) exhibit similar behaviors?", "limitations": "The authors have made some effort to address the limitations; however, it is crucial for them to conduct a more comprehensive investigation into these limitations, as mentioned in the \"Weakness\" section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "### Writing\nThe authors effectively communicate their ideas and concepts, ensuring clarity and coherence throughout the paper. The logical structure and well-reasoned arguments contribute to the overall quality of the essay. The article excels in providing the reader with a clear understanding of the problem's context and significance. By effectively conveying the goals and challenges of the study, the authors enhance the reader's comprehension of the subsequent experiments. Overall, the writing is of high quality, facilitating a smooth and engaging reading experience.\n\n### Method\nThe paper presents an easy-to-use approach by introducing a method that is not only easy to implement, but also easy to apply, which enhances the potential adoption and practicality of the proposed approach.This user-friendly feature makes the method highly accessible and beneficial to researchers and practitioners in various fields.\nThe used code bases and hyperparameters are provided, allowing the results to be reproduced.", "weaknesses": "While I appreciate the proposed method's ease of use, I believe that the authors could have conducted a more comprehensive and statistically rigorous analysis of their approach, considering its simplicity.\nOne notable limitation of the paper is the absence of confidence interval plots and statistical analysis, which could have been derived from [1], to enhance the clarity and precision of the findings. Incorporating these elements would have allowed readers to better understand the level of uncertainty associated with the reported results, thus bolstering the overall robustness of the study.\nFurthermore, the authors only rely on a single seed for the initial analysis, without providing a compelling rationale for this choice. Although using a single seed can streamline the experimental process, it diminishes the validity of the findings by disregarding potential result variations arising from multiple seeds. A more thorough explanation or a comparison of outcomes based on different seeds would have added value to the introduction, ensuring a more comprehensive analysis.\nI appreciate that the authors included continuous control tasks in their study; however, these tasks are not thoroughly explored. While the authors provide hypotheses to explain the unexpected results, a deeper analysis would have been expected.\n\nIn line 293, the authors reference a follow-up paper on resetting approaches but fail to cite the original work [2], which states in the section \"What and how to reset\" that resetting the optimizer has almost no significant impact due to quick updates of the moments. This contradicts the findings of this work.\nWhich brings me to the conclusion that I believe that the paper shows promise and the authors have taken a positive direction. However, in its current form, the paper falls short of being acceptable. It is essential to include comparisons to other baselines, such as [2], to provide a more thorough understanding of the opportunities and limitations, and to gain a clearer understanding of the internal effects in order to explain the aforementioned points.\n\n### Minor\n- The protocol for the random resets is not easy to understand and should be specified more clearly \n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.\n[2] Nikishin, Evgenii, et al. \"The primacy bias in deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first would like to thank the reviewer for mentioning that our paper is high quality, facilitating a smooth and engaging reading experience. We also appreciate that the reviewer provided a detailed review.\n\n- While I appreciate the proposed method's ease of use, I believe that the authors could have conducted a more comprehensive and statistically rigorous analysis of their approach, considering its simplicity\n\nWe agree that we should have ensured we get statistically significant results not just for the experiment with all 55 Atari games, but also with the ablation studies. Please see the attached pdf where we have added 9 new seeds to all ablation studies.\n\n- In line 293, the authors reference a follow-up paper on resetting approaches but fail to cite the original work [2], which states in the section \"What and how to reset\" that resetting the optimizer has almost no significant impact due to quick updates of the moments. This contradicts the findings of this work.\n\nWe actually reference the paper [2] in our paper. It is our reference [47]. We are not sure why the reviewer is assessing our work negatively given that we actually cite this paper. We are happy to add more discussion about situating our work relative to this innovative paper.\n\nTo summarize, notice that the motivation of their work and the techniques used for resetting are quite different. Nikishin et al. study the primacy bias phenomenon whereby the RL agent usually overfits to the training examples it encounters early during training, and thus can lose its plasticity. This is quite orthogonal to our motivation where we argue that an RL agent is facing a sequence of optimization problems and therefore we should reset the optimizer at the beginning of each new problem. Nikishin et al., reset parts of the agent's weights to maintain plasticity, whereas we discuss resetting the optimizer to ensure we do not contaminate the gradient steps taken by the optimizer. Finally, Nikishin et al. is conducting their experiments in the simpler continuous control setting with policy-gradient approaches where Polyak-based updates are often used for the target network. This is another significant deviation from our setting where we focus on the DQN style of algorithms where hard target-network updates are often preferred to the Polyak update. Overall, we have different motivations, techniques, and settings relative to Nikishin et al. We are happy to further clarify this in the text.\n\n- Why is resetting not effective for continuous control tasks?\n\nIn the continuous control task the value of $K$ used in algorithms such as SAC and TD3 is pretty small (often as small as $K=1$). Therefore, resetting the optimizer after each outer iteration corresponds to not giving any time to the optimizer to compute the moment estimates. Clearly suppressing the optimizer in terms of not allowing it to compute moments is not a good idea.\n\n- Are there any experiments demonstrating the impact of contamination on the tasks discussed in this paper?\n\nPlease see the attached pdf where we verified the contamination effect empirically.\n\n- What are the consequences of reducing the frequency of optimizer resets beyond $K=8000$?\n\nWe are not sure what the reviewer is asking when they say ``reducing beyond $8000$''. If the question is about increasing this value beyond $8000$, in our experience, spending too much time solving each iteration is not a good idea since after each step of optimization, by standard convention, we also take a step in the environment. Therefore, it is usually better to crudely solve each iteration (smaller $K$) and move to the next one. Resetting allows us to move to the next iteration faster, which is why for resetting a smaller than $K=8000$ is usually a better value.\n\n- How can an optimal value for $K$ be determined?\n\nWe would like to remind that we did not introduce this parameter. Therefore, we do not think it is incumbent upon us to answer this question in our paper as this is simply beyond the scope of our work. Also, not knowing the optimal $K$ is not at all reflective of a weakness of the resetting strategy, because we also need to tune $K$ when not resetting the optimizer. That said, this is a good empirical question, one that we would be super excited to focus on after publishing this paper.\n\n- Which ADAM/optimizer parameters are relevant when performing resets, i.e., have an effect when reset?\n\nWe have noticed that we need to reset both Adam's first and second estimates to achieve the best performance. Only resetting the first or the second moment is not effective. We are happy to add this experiment to the paper.\n\n- Are the observed effects still present when modifying the ADAM/optimizer hyperparameters?\n\nThe reviewer can find additional experiments in the original submission where we changed the Adam optimizer to RMSProp and Rectified Adam, and we observed that resetting is useful in those contexts too. Moreover, for the rebuttal we also added experiments on DQN and IQN, where again we showed that resetting is helpful. See the attached pdf for the new results.\n\n- Do different loss functions used in various DQN versions (e.g., MSE, Huber, Quantile) exhibit similar behaviors?\n\nAgain, see our answer above. Rainbow, Rainbow Pro, IQN, and DQN are all using different loss functions. Nonetheless resetting is helpful in all cases above.\n\n"}, {"review_id": "SBjWMjTlEf", "reviewer": "Reviewer_aUSe", "summary": "The authors argue that Adam's internal parameters should be reset with each iteration. The authors demonstrate the effectiveness of this approach in the Atari domain. ", "questions": "As mentioned in weaknesses:\n- Does this result generalize to other methods besides Rainbow? Such as DQN or more modern deep RL methods.\n- Can the authors show that initializing Adam's parameters with 0 is better than using the parameters from the previous iteration in a more concrete way? Such as examining the behavior of the actual values. \n", "limitations": "No concerns. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Results are convincing for Rainbow.\n- Novelty is very low, but potential impact is high, if the result generalizes, there is little reason not to use this method in every DQN-style RL algorithm.  ", "weaknesses": "- As mentioned by the authors, novelty is low compared to Bengio et al. \n- Results are only shown on Rainbow and do not appear to work for SAC (with reason) -- but does raise the question if the method is effective for other RL methods. \n- There is limited insight. Can the authors show that initializing Adam's parameters with 0 is better than using the parameters from the previous iteration in a more concrete way? Such as examining the behavior of the actual values. The fact that not resetting is seemingly better at low values of K suggests that not resetting can provide a reasonable initialization for the parameters of Adam.\n\nMinor\n- The y-axis is unlabelled in several figures. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's reading our paper as well as the overall quite positive assessment. Thanks for pointing out that the paper's potential impact is high.\n\n- As mentioned by the authors, novelty is low compared to Bengio et al.\n\nNote that Bengio et al. proposed an approach to account for the staleness of the moment estimates in RL. Their approach is one that requires computing the Hessian, and so, it is not the most practical approach. Also, they did not conduct any experiment on the Atari domain with various algorithms and optimizers. We did not say that our novelty is low compared to them, merely that they also identified the same issue with using momentum-based optimizers in deep RL.\n\n- Results are only shown on Rainbow and do not appear to work for SAC (with reason) -- but does raise the question if the method is effective for other RL methods.\n\nAs we have shown in the original experiments, resetting is useful for 1) Rainbow with Adam, 2) Rainbow with RMSProp, 3) Rainbow Pro with Adam 4) Rainbow with Rectified Adam. Additionally we added new results for 5) DQN with Adam and 6) IQN with Adam. Overall the resetting approach has been useful in all variants of DQN-like algorithms we tested. As for approaches such as SAC, notice that in the continuous control tasks the value of $K$ used in algorithms such as SAC is pretty small (often as small as $1$). Therefore, resetting the optimizer after each outer iteration corresponds to not giving any time to the optimizer to compute the moment estimates. We think that the positive impact of resetting in all DQN-like algorithms is strong enough to warrant publication.\n\n- There is limited insight. Can the authors show that initializing Adam's parameters with 0 is better than using the parameters from the previous iteration in a more concrete way?\n\nWe examined the behavior of gradient estimates and their cosine similarity relative to the Adam's first moment estimates. There is no meaningful correlation between the two, suggesting that not resetting corresponds with arbitrarily initializing Adam estimates. Please see the attached pdf for the new results.\n\n- The y-axis is unlabelled in several figures.\n\nThanks for catching this issue. We have labeled the axis.\n\n- Does this result generalize to other methods besides Rainbow? Such as DQN or more modern deep RL methods.\n\nYes, they indeed do, based on our old and new experiments where we add results for DQN (and IQN) per your suggestion. Please see the pdf, and we hope the reviewer takes into account the added result in their final evaluation."}]

Author rebuttal:
We appreciate our reviewers for their thoughtful feedback. Despite their constructive criticism, we believe that all reviewers see positive aspects in our work. In particular, reviewer Ks5C agrees that our approach is motivated and well-described and that we ran our main experiments on all 55 Atari games with 10 seeds. Reviewer 5d1P agrees with the main message we liked to convey in this paper, namely that we need to understand better the tools we borrow from other fields, in this case deep-learning optimization. Reviewer 59Jp states that the paper presents an easy-to-use approach by introducing a method that is not only easy to implement, but also easy to apply, which enhances the potential adoption and practicality of the proposed approach, and finally Reviewer aUSe mentions that the impact of our work could be potentially high given the wide-spread use of modern optimizers in deep RL.

That said, Reviewers 5d1P and 59Jp raised the issue pertaining to using only one seed in some of our experiments. We first reiterate that for our main experiment on all 55 Atari games, we used 10 random seeds which is more than or at least on par with mainstream deep RL papers. Because of the volume of our ablation studies (already more than 1000 experiments) we were not able to use more than 1 seed in our ablations when submitting the paper. That said, we understand your concern, and agree that with the ablation studies, too, we need to have made sure that our results are statistically significant akin to what we did with the full experiment with all 55 Artari games.

To this end, we added 9 more seeds to the single seed we originally used for the ablation experiments, so in total we currently have 10 seeds for our ablation and main experiments. We updated Figures 1, 2, and 3 in the paper and we added error bars as requested by two of the reviewers. Please see the attached pdf for updated results. We are happy to add more seeds to all experiments in the final camera-ready version of the paper. We are under the impression that reviewers 5d1P and 59Jp overall liked the ideas presented in this work, but gave us a low score in large part due to the seed issue, so we really hope that they reconsider their score in light of now using 10 seeds in all experiments.

Moreover, reviewers 5d1P and 59Jp asked us to elaborate further on the contamination effect that we claim to have plagued modern optimizers in deep RL. To recap, we showed that TD/Rainbow/DQN/etc could be thought of as RL algorithms that solve a sequence of optimization problems:

$\theta^{t+1}\leftarrow \arg\min_{w} H(\theta^t,w)\ ,$

where we use Adam (without resetting) to approximately solve all iterations. Here, by contamination we mean that the Adam moment estimates computed in the previous iterations $(t-1, t-2, ...)$ correlate weakly (if at all) with the gradient of the objective function at iteration $t$. To support our claim, we conducted a new experiment where we measured the cosine similarity between Adam's first moment and the gradient we compute immediately after we update the target network. If there is no contamination effect then surely the cosine similarity (which is between -1 and +1) should usually be positive. 

However, as shown in the attached pdf, we see that while it is true that very early in training the cosine similarity is positive (meaning that moment estimates are not stale), the cosine similarity quickly converges to around zero, indicating that there is no meaningful similarity between Adam's moment estimate and the gradient. This observation manifests itself in all games we tested. This is what we refer to as the contamination effect: without resetting, at each iteration we are in effect poorly initializing Adam, and so the optimizer needs to waste some gradient steps just to unlearn this poor initialization. Resetting is a simple and effective strategy to hedge against this contamination.

Finally, Reviewers aUSe and 59Jp asked us to look at effects of resetting on other standard deep RL algorithms, with DQN as their primary suggestion. In light of their feedback, we ran DQN and IQN with and without resetting and repeated the experiment presented in Figure 4 of the paper. In this case, we benchmarked DQN and IQN for different values of $K$ with and without resetting Adam. We see that a similar result manifests itself in which our resetting algorithms perform better than the standard non-resetting algorithms.

With these new results added, overall we have shown that resetting enhances the performance of 1) Rainbow with Adam, 2) IQN with Adam, 3) DQN with Adam, 4) Rainbow Pro with Adam, 5) Rainbow with RMSProp, and 6) Rainbow with Rectified Adam. We really believe that this is a comprehensive set of results, and that it is reasonable to expect that this phenomenon would generalize across other successors of DQN and Rainbow. As mentioned by reviewer aUSe, this strengthens the significance of our work, so we hope Reviewers aUSe and 59Jp consider these additional experiments in their final evaluation of our paper. 

Please take a look at the attached pdf for 1) experiments with added seeds, 2) experiments measuring the contamination effect, and 3) experiments with DQN and IQN.


Meta-review: There were no major concerns for this paper after the rebuttal. The remaining weaknesses are:
* Lowish novelty
* Applicable only to DQN-like algorithms with hard-updates

Overall, the paper is sufficiently interesting and done well-enough to merit acceptance. Given that the proposed change is small and easy to implement, it is likely to have a significant impact. The tangential Rainbow results w/o a target network also quite interesting. I recommend acceptance.
Decision: Accept (poster)