Title: Resetting the Optimizer in Deep RL: An Empirical Study
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Reinforcement Learning

Rainbow Algorithm

Atari benchmark

Adam Optimizer
Abstract: We focus on the task of approximating the optimal value function in deep reinforcement learning. This iterative process is comprised of solving a sequence of optimization problems where the loss function changes per iteration. The common approach to solving this sequence of problems is to employ modern variants of the stochastic gradient descent algorithm such as Adam. These optimizers maintain their own internal parameters such as estimates of the first-order and the second-order moments of the gradient, and update them over time. Therefore, information obtained in previous iterations is used to solve the optimization problem in the current iteration. We demonstrate that this can contaminate the moment estimates because the optimization landscape can change arbitrarily from one iteration to the next one. To hedge against this negative effect, a simple idea is to reset the internal parameters of the optimizer when starting a new iteration. We empirically investigate this resetting idea by employing various optimizers in conjunction with the Rainbow algorithm. We demonstrate that this simple modification significantly improves the performance of deep RL on the Atari benchmark.

Resetting the Optimizer in Deep RL: An Empirical Study

Abstract

We focus on the task of approximating the optimal value function in deep reinforcement learning. This iterative process is comprised of solving a sequence of optimization problems where the loss function changes per iteration. The common approach to solving this sequence of problems is to employ modern variants of the stochastic gradient descent algorithm such as Adam. These optimizers maintain their own internal parameters such as estimates of the first-order and the secondorder moments of the gradient, and update them over time. Therefore, information obtained in previous iterations is used to solve the optimization problem in the current iteration. We demonstrate that this can contaminate the moment estimates because the optimization landscape can change arbitrarily from one iteration to the next one. To hedge against this negative effect, a simple idea is to reset the internal parameters of the optimizer when starting a new iteration. We empirically investigate this resetting idea by employing various optimizers in conjunction with the Rainbow algorithm. We demonstrate that this simple modification significantly improves the performance of deep RL on the Atari benchmark.

1 Introduction

Value-function optimization lies at the epicenter of large-scale deep reinforcement learning (RL). In this context, the deep RL agent is equipped with a parameterized neural network that, in conjunction with large-scale optimization, is employed to find an approximation of the optimal value function. The standard practice is to initialize the optimizer only once and at the beginning of training [1]. With each step of gradient computation, the optimizer updates its internal parameters, such as the gradient's moment estimates [2], and uses these internal parameters to update the network.

A unique ingredient in value-function optimization is a technique known as forward bootstrapping [3]. In typical regression problems, the loss function is a measure of the discrepancy between a fixed target and the agent's current approximation. This stands in contrast to value-function optimization with forward bootstrapping. In this setting, the goal is to minimize the discrepancy between the agent's current approximation and a second approximation obtained by performing one or multiple steps of look ahead. The regression target thus depends on the RL agent's own approximation which changes continually over the span of learning.

We first argue that in the presence of forward bootstrapping the value-function optimization process can best be thought of as solving a sequence of optimization problems where the loss function changes per iteration. We demonstrate that in this case the loss function being minimized is comprised of two inputs: target parameters that remain fixed during each iteration, and optimization (or online) parameters that are adjusted to minimize the loss function during each iteration. In this context, updating the target parameters changes the loss function being minimized in the next iteration. By extension, the first-order and the second-order moments of the gradient can also change arbitrarily because they depend on the landscape of the loss function.

Shoham Sabach Amazon & Technion

Realizing that the RL agent faces a sequence of optimization problems, a natural practice would be to reset the optimizer at the beginning of each new optimization problem. However, in deep RL, modern optimization algorithms are employed often without ever resetting their internal parameters. Wequestion this standard practice, and in particular, similar to Bengio et al. [4] we ask if the internal parameters accumulated by the optimizer in the previous iterations can still be useful in updating the parameters of the neural network in the current iteration. Could it be the case that in most cases relying on gradient computations pertaining to the previous iterations is just contaminating the internal parameters? And ultimately, can this negatively affect the performance of the RL agent? We answer these questions through various experiments.

We then propose a simple modification to existing RL agents where we reset the internal parameters of their optimizer at the beginning of each iteration. Using the deep RL terminology, each time we update the target network, we also reset the internal parameters of the optimizer. We show that this remarkably simple augmentation significantly improves the performance of the competitive Rainbow agent [5] when used in conjunction with various optimizers. Most notably, under the standard Adam optimizer [6], we observe that resetting unleashes the true power of Adam and ultimately results in much better reward performance, suggesting that resetting the optimizer is a more promising choice in the context of value-function optimization.

2 RL as a Sequence of Optimization Problems

We argue that many deep RL algorithms can be viewed as iterative optimization algorithms where the loss function being minimized changes per iteration. To this end, we first recall that the DQN algorithm [7] decouples the learning parameters into two sets of parameters: the target parameters θ , and the optimization (or online) parameters w that are adjusted at each step. DQN updates these two parameters in iterations. More specifically, each iteration is comprised of an initial step, w t, 0 = θ t , performing multiple updates with a fixed θ t :

The agent then synchronizes the two learning parameters, θ t +1 ← w t,K , prior to moving to the next iteration t +1 where updates are performed using θ t +1 . Here K is a hyper-parameter whose value is commonly set to 8000 for DQN and its successors [5].

Observe that the effect of changing the optimization parameters w on max a ′ q ( s ′ , a ′ ; θ t ) is ignored during gradient computation despite the fact that an implicit dependence exists due to synchronization. In fact, the update cannot be written as the gradient of any loss function that only takes a single parameter as input [8]. However, this update can be written as the gradient of a function with two input parameters. To illustrate this, we now define:

where B is the experience replay buffer containing the agent's environmental interactions. Observe that the update (1) could be thought of as performing gradient descent using ∇ w H on a single sample ⟨ s, a, r, s ′ ⟩ where ∇ w H is the partial gradient of H with respect to w . Therefore, by viewing the K gradient steps as a rough approximation of exactly minimizing H with respect to the optimization parameters w , we can write this iterative process as:

In this optimization perspective [9], what is typically referred to as the online parameter is updated incrementally because the objective function (2) does not generally lend itself into a closed-form solution. Notice also that in practice rather than the quadratic loss, DQN uses the slightly different Huber loss [7]. Moreover, follow-up versions of DQN use different loss functions, such as the Quantile loss [10], sample experience tuples from the buffer non-uniformly [11], use multi-step updates [12, 13], or make additional modifications [5, 14]. That said, the general structure of the algorithm follows the same trend in that it proceeds in iterations, and that the loss function being minimized changes per iteration.

3 Revisiting Adam in RL Optimization

So far we have shown that popular deep RL algorithms could be thought of as a sequence of optimization problems where we approximately solve each iteration using first-order optimization algorithms. The most primitive optimizer is the stochastic gradient descent (SGD) algorithm, which simply estimates and follows the descent direction. However, using vanilla SGD is ineffective in the context of deep RL. In contrast, more sophisticated successors of SGD have been used successfully. For example, the original DQN paper [7] used the RMSProp optimizer [15]. Similarly, Rainbow [5] used the Adam optimizer [6], which could roughly be thought of as the momentum-based [16] version of RMSProp. More formally, suppose that our goal is to minimize a certain loss function J ( w ) . Then, starting from w 0 , after computing a stochastic gradient g i ≈ J ( w i ) , Adam proceeds by computing a running average of the first-order and the second-order moments of the gradient as follows:

where ( g i ) 2 applies the square function to g i element-wise. Notice that in Adam we initialize m 0 = 0 and v 0 = 0 , and so the estimates are biased towards 0 in the first few steps. Define the function power ( x, y ) = x y . To remove this bias, a debiasing step is performed in Adam as follows:

before finally updating the network parameters: w i ← w i -1 -α ̂ m/ ( ̂ v + ϵ ) with a small hyperparameter ϵ that prevents division by zero. The hyper-parameter α is the learning rate.

√

As explained before, in deep RL we often minimize a sequence of loss functions, unlike the standard usecase of Adam above where we minimize a fixed loss. We now present the pseudocode of DQN with Adam. Notice that in this pseudocode we do not present the pieces related to the RL agent's environmental interactions to primarily highlight pieces pertaining to how θ and w are updated. We use t to denote the sequence of loss functions (outer iteration), and we use k to denote each gradient step (inner iteration) in a fixed outer iteration t . One can present the code with these two indices, but to simplify the presentation, we use a third index i that is incremented with each gradient step.

From the pseudocode above, first notice that if the index i is not reset, then the debiasing quantities 1 -power ( β 1 , i ) and 1 -power ( β 2 , i ) quickly go to 1 and so the debiasing steps will have minimal effect on the overall update, if at all. In absence of resetting, the optimization strategy could then be thought of as initializing the first-order ( m ) and the second-order ( v ) moment estimates at each iteration t by whatever their values were at the end of the previous iteration t -1 . This seems like an arbitrary choice, one that deviates from design decisions that were made by Adam [6].

This choice makes some sense if the optimization landscape in the previous iterations is similar to that of the current iteration, but it is not clear that this will always be the case in deep RL. In cases

average episode score

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

20000

40000

60000

80000

Asterix

0

30

60

90

120

2500

5000

7500

10000

12500

15000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

50000

75000

100000

125000

150000

175000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

DemonAttack

0

30

60

90

120

5000

10000

15000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

5000

10000

15000

20000

25000

Phoenix

0

30

60

90

120

0

5000

10000

15000

20000

Seaquest

0

30

60

90

120

Training Frames (Million)

5000

10000

15000

20000

Zaxxon

Rainbow

Rainbow -- reset

where this is not the case, the agent can waste many gradient updates just to 'unlearn' the effects of the previous iterations on the internal parameters m and v . This is a contamination effect that plagues RL optimization as we later demonstrate. Fortunately, there is a remarkably easy fix. Note that the Adam optimizer is fully equipped to deal with resetting the moment estimates to 0 , despite the bias it introduces, because the bias is dealt with adequately by performing the debiasing step. Finally, note that this is an inexpensive and convenient fix in that it adds no computational cost to the baseline algorithm, nor does it add any new hyper-parameter.

4 Experiments

We chose the standard Atari benchmark [10] to perform our study, and also chose the popular Rainbow agent [5], which fruitfully combined a couple of important techniques in learning the value function. This combination resulted in Rainbow being the state-of-the-art agent, one that remains a competitive baseline to this day. We used the most popular implementation of Rainbow, namely the one in the Dopamine framework [1], and followed Dopamine's experimental protocol.

Note that other than the popular Dopamine baseline [1], we checked the second most popular implementations of DQN and Rainbow on Github, namely Github.com/devsisters/DQN-tensorflow and Github.com/Kaixhin/Rainbow, and found that resetting is absent in these implementations as well. Lack of resetting is thus not an oversight of the Dopamine implementation, but the standard practice in numerous Deep RL implementations.

4.1 Rainbow with Resetting Adam

Here our desire is to investigate the effect of resetting the optimizer on the behavior of the Rainbow agent with its default Adam optimizer. For all of our ablation studies, including this experiment, we worked with the following 12 games: Amidar , Asterix , BeamRider , Breakout , CrazyClimber , DemonAttack , Gopher , Hero , Kangaroo , Phoenix , Seaquest , and Zaxxon . Note that we will present comprehensive results on the full set of 55 Atari games later. Limiting our experiments to these 12 games allowed us to run multiple seeds per agent-environment pair, and therefore, obtain statistically significant results.

From Algorithm 1, note that a key hyper-parameter in the implementation of DQN (and by extension Rainbow) is the number of gradient updates ( K ) per iteration whose default value is 8000 in most deep RL papers [7, 17, 5]. We are interested to see the impact of this hyper-parameter on the performance of Rainbow with and without resetting. We first present the results for K = 8000 in Figure 1. We can see that with K = 8000 resetting the optimizer often results in improved performance.

We now change the value of K to understand the impact of changing K on this comparison. Lowering K provides a smaller budget to the optimizer for solving each iteration. Notice that regardless of the value of K we performed the same number of overall gradient updates to the optimization (online) network across the entire training. Stated differently, a smaller value of K will correspond to a larger value T in Algorithm 1 because for the sake of a fair comparison we always keep the multiplication of K × T fixed across all values of K in Rainbow with or without resetting.

We now repeat this experiment for multiple other values of K . Moreover, akin to the standard practice in the literature [18], rather than looking at individual learning curves per game, hereafter we look at the human-normalized performance of the agents on all games, namely: ScoreAgent -ScoreRandom ScoreHuman -ScoreRandom . To compare the performance of the two agents across all 12 games, we compute the median of this number across the games and present these results for each value of K in Figure 2.

0

30

60

90

120

0

1

2

3

4

5

K=1

0

30

60

90

120

0

1

2

3

4

5

K=500

0

30

60

90

120

0

1

2

3

4

5

0

30

60

90

120

0

1

2

3

4

5

K=2000

0

30

60

90

120

0

1

2

3

4

5

K=4000

0

30

60

90

120

0

1

2

3

4

5

K=6000

0

30

60

90

120

Training Frames (Million)

0

1

2

3

4

5

human-normalized median

K=8000

0

30

60

90

120

Training Frames (Million)

0

1

2

3

4

5

human-normalized median

K=8000

Rainbow

Rainbow -- reset

Rainbow -- random reset

From Figure 2 we can see very clearly that Rainbow with resetting is dominating the original Rainbow agent for all values of K but K = 1 . Notice that in the extreme case of K = 1 the Adam optimizer is in effect not accumulating any internal parameters because we reset after each step. Therefore, in essence, when K = 1 the update would be more akin to using the RProp optimizer [19], which is not effective in our setting. Another interesting trend is that in fact when we reset the optimizer, smaller values of K than the default 8000 become the most competitive. This is in contrast to Rainbow without reset where the value of K does not influence the final performance. This is because resetting removes the initial bias in the moment estimates that would otherwise be present when no resetting is performed. The resetting agent does not have to first 'unlearn' this bias, and thus can reasonably solve each iteration with a smaller K and move to the next iteration faster having accurately solved the previous iteration.

To further situate our results, we add another baseline where we reset the optimizer stochastically, meaning with some probability after each update to the optimization network. Whereas in the original resetting case (red), we reset the optimize right at the beginning of each iteration, in this case we just reset the optimizer after each update to the optimization parameter w , and the value of K determines the probability ( 1 /K ) with which we reset the Adam optimizer. With this choice of probability, on expectation we have 1 reset per iteration, but now resetting is stochastically performed at any given step. We present this result in Figure 3 where this additional baseline is indicated in blue.

We see that even randomly resetting the Adam optimizer provides some benefits relative to not resetting. However, we obtain the higher performance improvement by resetting the optimizer at the beginning of each iteration. Please see the Appendix for individual learning curves.

We further distill the above results by computing the area under the curves in Figure 2. Observe from Figure 3 that in the case of resetting (red), an inverted-U shape manifests itself with the best performance achieved by intermediate values of K . Also, to our surprise, in the case of no-resetting (which is indicated in black and corresponds to the standard Rainbow agent), notice that performance is basically flat as a function of K . Even taken to the extreme with K = 1 , we observe no performance degradation in Rainbow. This means that freezing the target network for 8000 steps,

1

2000

4000

6000

8000

K

0.0

1.0

2.0

3.5

area under the curve

Rainbow

Rainbow -- reset

Rainbow -- random reset

while harmless, provides no meaningful improvement to the performance of Rainbow. It is not clear to us, then, why the choice of K = 8000 was made in the original Rainbow paper. One possible explanation is that this choice is just a legacy from the previous papers. As we show later, in DQN it is important that a large value of K is chosen, otherwise there will be extreme performance degradation, so it is possible that an assumption about the need for a large K in Rainbow was made, and the validity of this assumption was not tested thoroughly.

To demonstrate the contamination effect, and to better understand where the benefit of resetting is coming from, we compute the cosine similarity between Adam's first-order moment estimate and the gradient estimate just before and after the update to the target network. Viewed as a measure of direction-wise similarity, the cosine of two vectors u and v is defined by Cosine ( u, v ) = u ⊤ v/ ( || u ||· || v || ) . In general, Cosine ( u, v ) ≫ 0 if the two vectors are similar, and Cosine ( u, v ) ≈ 0 if there is no similarity. In the context of Algorithm 1, this corresponds with computing the cosine similarity between g i and m i twice per each iteration t , namely when k = K -1 (before target update) and when k = 0 (after target update). We now present this result in the following table.

4.2 Experiments with Other Agent-Optimizer Combinations

We so far demonstrated that we can improve value-function optimization in Rainbow using the idea of resetting Adam's moments. Is this effect specific to the Rainbow-Adam combination or does it generalize to settings where we use alternative optimization algorithms and even different agents?

To answer this question, we first investigate the setting where we use the RMSProp optimizer. First presented in a lecture note by Hinton [15], RMSProp can be thought of as a reduction of Adam where we only maintain a running average of the second-order moment, but not the first. In other words, RMSProp corresponds to Adam with β 1 = 0 and also skipping the debiasing step for the second-order moment estimate. Similar to Figure 3, we ran Rainbow and RMSProp with and without resetting on the same 12 games, computed the human-normalized median, and then distilled the learning curves by computing their area under the curve for each value of K . Our result are presented in Figure 4 (top left). Notice that while overall we see a performance degradation when we move from Adam to RMSProp, we still observe that resetting the optimizer after updating the target network can positively affect the agent's score.

We next consider a successor (rather than a predecessor) of Adam, namely the Rectified Adam optimizer introduced by Liu et al. [20]. This recent variant of Adam is especially interesting because

500

2000

4000

6000

8000

K

0.0

0.2

0.4

0.6

0.8

1.0

area under the curve

Rainbow -- RMSProp

Rainbow -- RMSProp -- reset

500

2000

4000

6000

8000

K

1.0

1.5

2.0

2.5

3.0

3.5

area under the curve

Rainbow -- Rectified Adam

Rainbow -- Rectified Adam -- reset

500

2000

4000

6000

8000

K

0.0

0.2

0.4

0.6

0.8

1.0

area under the curve

DQN -- Adam

DQN -- Adam -- reset

500

2000

4000

6000

8000

K

1.0

1.5

2.0

2.5

3.0

3.5

area under the curve

RainbowPro -- Adam

RainbowPro -- Adam -- reset

it controls the variance of the Adam optimizer early in training by employing step-size warm up in conjunction with a rectifying technique. With resetting, the optimization algorithm basically starts from scratch many times, and so the rectification technique of Liu et al. [20] can be crucial in further improving performance. We present this results in Figure 4 (top right). Observe that equipping the agent with resetting is having a positive impact akin to what we showed in the case of RMSprop.

Pivoting into different agents, we present the next experiment where we used the Adam optimizer in conjunction with the DQN algorithm [7]. This agent is a more primitive version of Rainbow, but we again can observe that a performance improvement manifests itself when we reset the optimizer per iteration (Figure 4 bottom left).

We next look at the RainbowPro agent introduced by Asadi et al. [21], where they endowed Rainbow with proximal updates. This ensures that at each iteration the optimization parameter gravitates towards the previous target parameter. Again, we employ resetting in the context of their introduced agent. The result, which again shows the conducive impact of resetting, is presented in Figure 4 (bottom right). Overall we can see the earlier result with Rainbow and the Adam optimizer generalizes to a couple of reasonable alternatives, thus supporting the hypothesis that our findings may remain valid for a much broader set of agent-optimizer combinations.

4.3 Resetting Individual Moments

So far we have shown that resetting optimizer moments positively impacts RL agents. Notice that in the specific case of Adam (and its variants), as we detailed in Section 3, two disjoint moment estimates are being computed so a natural follow up question is how important it is to reset each individual moment. Our goal is to answer this question now.

To this end, we again performed the previous experiment on the same set of 12 games. This time, we added two other resetting baselines, a first baseline in which we only reset Adam's first-order

500

2000

4000

6000

8000

K

2.0

2.5

3.0

3.5

area under the curve

Rainbow

Rainbow -- reset both moments

Rainbow -- reset 1st moment

Rainbow -- reset 2nd moment

moment estimate per iteration while not resetting the second-order moment, and the converse case where we only reset the second-order moment estimate. In Figure 5, we compare the performance of these two baselines relative to the two earlier cases where we either never reset any moment, or reset both moments.

An important ingredient here is the default choice of β 1 and β 2 in popular deep-learning libraries (such as Tensorflow and PyTorch), and by extension in the Dompamine framework and in our paper. In these settings, we have β 1 = 0 . 9 and β 2 = 0 . 999 . This means that the second-order moment estimate is heavily leaning on to the past gradient estimates, and so more contamination will naturally arise. This is consistent with the result here that it is quite beneficial even when we only reset the second-order moment. More generally, the importance of resetting will be somewhat dependent on the specific β values being used. Note that one may think of our simple resetting strategy as an example of using adaptive β values, where we use β 1 = β 2 = 0 when moving into a new iteration ( k = 0 ) and then falling back into the original β 1 = 0 . 9 and β 2 = 0 . 999 for k > 0 . Here by adaptive we mean that β values can depend on k (or even t ) in arbitrary ways as opposed to being constant across training. We may use other adaptive strategies that can potentially improve upon our simple resetting idea, but we leave this exciting direction for future work.

4.4 Comprehensive Experiments on 55 Atari Games

So far we focused on performing smaller ablation studies on the subset of Atari games. We now desire to perform a more comprehensive experiment on the full 55 Atari games.

We now evaluate 3 different agent-optimizer combinations. As our benchmark, we have the original Rainbow algorithm with the Adam optimizer, without resetting, and the original value of K = 8000 . We refer to this as the standard Rainbow agent. Our next agent is Rainbow with the Adam optimizer and resetting. In light of our results from Figure 3 we chose the value of K = 1000 for this agent. Finally, we have Rainbow with Rectified Adam and resetting. Similarly, and in light of Figure 4, we chose the value of K = 4000 for this agent. In Figure 6, we present the mean and median performance of the 3 agents on 55 games.

From Figure 6, we can see that resetting the Adam optimizer is clearly boosting the performance of the Rainbow agent in terms of both the mean and the median performance of the agent across games. Moreover, we present another comparison in terms of the per-game asymptotic improvements of the two resetting agents against the Rainbow agent in Figure 7.

4.5 Continuous Control

For completeness we also conducted experiments in continuous-action environments using MuJoCo physics simulator [22]. Specifically, we used Soft Actor-Critic (SAC) [23], an off-policy actor-critic algorithm with a stochastic policy. It is worth noting that SAC and Rainbow/DQN differ in several aspects. Chief among these differences is that SAC utilizes soft target updates (also known as Polyak updates) to update the target parameter, while Rainbow/DQN uses hard target updates (see Section

0

30

60

90

120

Training Frames (Million)

0.5

1.0

1.5

2.0

2.5

human-normalized median

Rainbow

Rainbow -- Adam -- reset

Rainbow -- Rectified Adam -- reset

0

30

60

90

120

Training Frames (Million)

0

5

10

15

human-normalized mean

Rainbow

Rainbow -- Adam -- reset

Rainbow -- Rectified Adam -- reset

- 50%

- 10%

0%

10%

50%

200%

ChopperCommand

PrivateEye

Gravitar

BankHeist

Bowling

Amidar

MontezumaRevenge

YarsRevenge

Jamesbond

Hero

Krull

Kangaroo

BattleZone

RoadRunner

Riverraid

Carnival

Robotank

Atlantis

Breakout

VideoPinball

Pitfall

Boxing

Seaquest

Pong

Zaxxon

Enduro

MsPacman

JourneyEscape

Alien

Assault

Tutankham

Venture

Frostbite

Asteroids

Freeway

CrazyClimber

BeamRider

Pooyan

StarGunner

TimePilot

KungFuMaster

Gopher

NameThisGame

AirRaid

IceHockey

UpNDown

Solaris

Asterix

DemonAttack

WizardOfWor

SpaceInvaders

Phoenix

- 50%

- 10%

0%

10%

50% 200%

ChopperCommand

PrivateEye

Bowling

Gravitar

BankHeist

MontezumaRevenge

Hero

Amidar

Pitfall

Riverraid

YarsRevenge

Krull

Kangaroo

JourneyEscape

Carnival

Freeway

Breakout

Atlantis

Boxing

Robotank

RoadRunner

Enduro

VideoPinball

MsPacman

Alien

BattleZone

Frostbite

Jamesbond

Seaquest

Venture

Tutankham

Asteroids

Zaxxon

Assault

CrazyClimber

Pong

Pooyan

TimePilot

BeamRider

StarGunner

KungFuMaster

NameThisGame

Gopher

AirRaid

IceHockey

Solaris

UpNDown

DemonAttack

WizardOfWor

SpaceInvaders

Asterix

Phoenix

2 for more details). SAC also uses the value of K = 1 so clearly it is not ideal to reset the optimizer after each step. Thus, it is not clear when to reset the optimizer. We show some our results in the Appendix, where the Adam optimizer was reset for both the critic and actor networks every 5000 steps.

It is evident that resetting the optimizer is not as helpful here to the extent it was with hard target updates, but there is still a small positive effect. Contamination is still present [4], but it is not clear how to best address it in this case, and we may require more advanced techniques than our simple resetting idea. We leave the investigation of this to future work.

5 Related Work

In this paper, we showed that common approaches to value-function optimization could be viewed as a sequence of optimization problems. This view is occasionally discussed in previous work [24] but in different contexts, perhaps earliest in Fitted Value Iteration [25, 26], Fitted Q Iteration [27, 28], and related approximate dynamic-programming algorithms [29, 30, 31]. More recently, Dabney et al. [32] presented the concept of the value improvement path where they argued that even in the single-task setting with stationary environments, the RL agent is still faced with a sequence of problems, and that the representation-learning process can benefit from looking at the entire sequence of problems. This view was further explored to show that ignoring this non-stationary aspect can lead to capacity [33] and plasticity loss [34]. This capacity loss problem can be mitigated in various ways, such as by regularization [33, 35, 36], introducing new parameters [34], or by using activation functions that are more conducive to learning a sequence of targets [37]. None of these works, however, considered the effect of RL non-stationarity on the internal parameters of the optimizer. Notice that this kind of non-stationarity is due to the solution not the environment, and stands in contrast to the non-stationary setting that arises when the MDP reward and transitions change, a setting that is well-explored [38, 39, 40, 41, 42, 43].

To hedge against the contamination effect, we proposed to reset the optimizer per iteration. Resetting the optimizer is a well-established idea in the optimization literature, and can be found for example in a paper by Nesterov [44]. Adaptive versions of resetting are also common and perform well in practice [45, 46]. Restarting the step-size (learning rate) also has some precedence in deep learning [47, 48, 49]. See also the standard book on large-scale optimization [50] as well as a recent review of this topic in the context of optimization [51].

Resetting RL ingredients other than the optimizer is another line of work that is related to our paper. Earliest work is due to Anderson [52], which restarts some network parameters when the magnitude of error is unusually large. A more recent example is Nikishin et al. [53] who proposed to reset some of the network weights, and the corresponding Adam statistics, to maintain plasticity. In contrast, our motivation is resetting Adam statistics is to ensure we do not contaminate the moment estimation process when we move to a new iteration. Nikishin et al. conducted their experiments in the simpler continuous-control setting with policy-gradient approaches where Polyak-based updates are often used and K = 1 , while here we focus on resetting in the context of hard target updates with large values of K . Perhaps the most similar work to our paper is that of Bengio et al. [4] who noticed the harmful effect of contamination, and proposed a solution based on computing the Hessian and a Taylor expansion of the loss, one that is unfortunately too expensive to be applied in conjunction with deep networks owing to their large number of weights.

6 Conclusion

In this paper, we argued that value-function optimization could best be thought of as solving a sequence of optimization problems where the loss function changes per iteration. Having adopted this view, we argued that it is quite natural to reset the optimizer, which can combat the contamination effect plaguing deep RL in the absence of resetting.

Amore general conclusion of our work is that we must obtain a deeper understanding of the internal process of optimization algorithms in order to unleash their true power in the context of deep RL. Standard results from optimization in supervised learning may not always readily transfer to RL. It is important to keep in mind that these optimization techniques were often designed for setting that are meaningfully different than RL, and thus some of the pitfalls of RL training can plague them when applied without proper consideration.

References

Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. arXiv , 2018.

Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv , 2016.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Emmanuel Bengio, Joelle Pineau, and Doina Precup. Correcting momentum in temporal difference learning, 2021.

Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence , 2018.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 2015.

Hamid Reza Maei. Gradient temporal-difference learning algorithms . PhD thesis, University of Alberta, 2011.

Kavosh Asadi, Shoham Sabach, Yao Liu, Omer Gottesman, and Rasool Fakoor. TD convergence: An optimization perspective. Advances in Neural Information Processing Systems , 2023.

Marc G Bellemare, Will Dabney, and R´ emi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning , 2017.

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv , 2015.

Kristopher De Asis, J Hernandez-Garcia, G Holland, and Richard Sutton. Multi-step reinforcement learning: A unifying algorithm. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Yunhao Tang, Remi Munos, Mark Rowland, Bernardo Avila Pires, Will Dabney, and Marc Bellemare. The nature of temporal difference errors in multi-step distributional reinforcement learning. Advances in Neural Information Processing Systems , 2022.

Kavosh Asadi, Neev Parikh, Ronald E Parr, George D Konidaris, and Michael L Littman. Deep radial-basis value functions for continuous control. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 2012.

Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics , 1964.

Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence , 2016.

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning , 2016.

Martin Riedmiller and Heinrich Braun Braun. A direct adaptive method for faster backpropagation learning: The rprop algorithm. In IEEE international conference on neural networks , 1993.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv , 2019.

Kavosh Asadi, Rasool Fakoor, Omer Gottesman, Taesup Kim, Michael Littman, and Alexander J Smola. Faster deep reinforcement learning with slower online network. Advances in Neural Information Processing Systems , 2022.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning , Proceedings of Machine Learning Research, 2018.

Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learning. In Learning for Dynamics and Control , 2020.

Geoffrey J Gordon. Stable function approximation in dynamic programming. In Machine learning proceedings 1995 , pages 261-268. Elsevier, 1995.

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6, 2005.

Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML 2005: 16th European Conference on Machine Learning , 2005.

Andr´ as Antos, Csaba Szepesv´ ari, and R´ emi Munos. Fitted q-iteration in continuous actionspace mdps. Advances in neural information processing systems , 2007.

Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming . Athena Scientific, 1996.

Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Research , 2003.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforcement learning: State-of-the-art , 2012.

Will Dabney, Andr´ e Barreto, Mark Rowland, Robert Dadashi, John Quan, Marc G Bellemare, and David Silver. The value-improvement path: Towards better representations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv , 2022.

Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and Andre Barreto. Deep reinforcement learning with plasticity injection. In Workshop on Reincarnating Reinforcement Learning at ICLR , 2023.

Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308.11958 , 2023.

Shibhansh Dohare, Juan Hernandez-Garcia, Parash Rahman, Richard Sutton, and A Rupam Mahmood. Loss of plasticity in deep continual learning. arXiv preprint , 2023.

Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv , 2023.

Sindhu Padakandla, Prabuchandran KJ, and Shalabh Bhatnagar. Reinforcement learning algorithm for non-stationary environments. Applied Intelligence , 2020.

Erwan Lecarpentier, David Abel, Kavosh Asadi, Yuu Jinnai, Emmanuel Rachelson, and Michael L Littman. Lipschitz lifelong reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

Yash Chandak. Reinforcement Learning for Non-stationary problems . PhD thesis, University of Massachusetts Amherst, 2022.

Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research , 2022.

Jelena Luketina, Sebastian Flennerhag, Yannick Schroecker, David Abel, Tom Zahavy, and Satinder Singh. Meta-gradients in non-stationary environments. In Conference on Lifelong Learning Agents , 2022.

David Abel, Andr´ e Barreto, Benjamin Van Roy, Doina Precup, Hado van Hasselt, and Satinder Singh. A definition of continual reinforcement learning. arXiv , 2023.

Yurii Nesterov. Gradient methods for minimizing composite functions. Mathematical Programming , 2012.

Brendan O'Donoghue and Emmanuel J. Cand` es. Adaptive restart for accelerated gradient schemes. Foundations of Computational Mathematics , 2012.

Bao Wang, Tan M. Nguyen, Andrea L. Bertozzi, Richard G. Baraniuk, and Stanley J. Osher. Scheduled restart momentum for accelerated stochastic gradient descent, 2020.

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. ArXiv , 2017.

Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. International Conference on Learning Representations , 2017.

Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv , 2018.

Amir Beck. First-order methods in optimization . SIAM, 2017.

Sebastian Pokutta. Restarting algorithms: Sometimes there is free lunch. In Integration of AI and OR Techniques in Constraint Programming , 2020.

Charles Anderson. Q-learning with hidden-unit restarting. Advances in Neural Information Processing Systems , 5, 1992.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning , 2022.

7 Appendix

7.1 Hyperparameters

We report the hyper-parameters used in our experiments.

7.2 Complete Results from Section 4.1

We show individual learning curves for Rainbow-Adam with and without resetting for different values of K .

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

20000

40000

60000

80000

Asterix

0

30

60

90

120

0

5000

10000

15000

20000

25000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

50000

75000

100000

125000

150000

175000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

DemonAttack

0

30

60

90

120

5000

10000

15000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

5000

10000

15000

20000

25000

Phoenix

0

30

60

90

120

0

5000

10000

15000

20000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow

Rainbow -- reset

Rainbow -- random reset

average episode score

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

20000

40000

60000

80000

100000

Asterix

0

30

60

90

120

2500

5000

7500

10000

12500

15000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

50000

75000

100000

125000

150000

175000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

100000

DemonAttack

0

30

60

90

120

5000

10000

15000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

5000

10000

15000

20000

25000

Phoenix

0

30

60

90

120

0

5000

10000

15000

20000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

Zaxxon

Rainbow

Rainbow -- reset

Rainbow -- random reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

20000

40000

60000

80000

100000

Asterix

0

30

60

90

120

5000

10000

15000

20000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

50000

75000

100000

125000

150000

175000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

10000

20000

30000

Phoenix

0

30

60

90

120

0

10000

20000

30000

Seaquest

0

30

60

90

120

Training Frames (Million)

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow

Rainbow -- reset

Rainbow -- random reset

0

30

60

90

120

0

1000

2000

3000

4000

Amidar

0

30

60

90

120

0

25000

50000

75000

100000

Asterix

0

30

60

90

120

5000

10000

15000

20000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

100000

120000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

20000

40000

60000

80000

Phoenix

0

30

60

90

120

0

5000

10000

15000

20000

25000

Seaquest

0

30

60

90

120

Training Frames (Million)

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow

Rainbow -- reset

Rainbow -- random reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

50000

100000

150000

Asterix

0

30

60

90

120

5000

10000

15000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

100000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

20000

40000

60000

80000

100000

Phoenix

0

30

60

90

120

0

10000

20000

30000

Seaquest

0

30

60

90

120

Training Frames (Million)

5000

10000

15000

20000

25000

average episode score

Zaxxon

Rainbow

Rainbow -- reset

Rainbow -- random reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

25000

50000

75000

100000

125000

Asterix

0

30

60

90

120

5000

10000

15000

20000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

Gopher

0

30

60

90

120

20000

40000

60000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

25000

50000

75000

100000

125000

Phoenix

0

30

60

90

120

0

5000

10000

15000

20000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow

Rainbow -- reset

Rainbow -- random reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

20000

40000

60000

Asterix

0

30

60

90

120

0

2500

5000

7500

10000

12500

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

0

50000

100000

150000

CrazyClimber

0

30

60

90

120

0

10000

20000

30000

40000

DemonAttack

0

30

60

90

120

0

5000

10000

15000

Gopher

0

30

60

90

120

0

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

5000

10000

15000

Phoenix

0

30

60

90

120

0

2000

4000

6000

8000

10000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

average episode score

Zaxxon

Rainbow

Rainbow -- reset

7.3 Complete Results from Section 4.2

We now show complete results from Section 4.2 starting with Rainbow RMSProp.

0

30

60

90

120

50

100

150

200

Amidar

0

30

60

90

120

2000

3000

4000

Asterix

0

30

60

90

120

2000

3000

4000

5000

BeamRider

0

30

60

90

120

0

20

40

60

Breakout

0

30

60

90

120

40000

60000

80000

100000

120000

CrazyClimber

0

30

60

90

120

2000

4000

6000

8000

DemonAttack

0

30

60

90

120

2000

3000

4000

5000

Gopher

0

30

60

90

120

5000

10000

15000

20000

Hero

0

30

60

90

120

2000

4000

6000

8000

Kangaroo

0

30

60

90

120

2000

3000

4000

5000

6000

7000

Phoenix

0

30

60

90

120

500

1000

1500

2000

2500

3000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

2000

4000

6000

average episode score

Zaxxon

Rainbow -- RMSProp

Rainbow -- RMSProp -- reset

0

30

60

90

120

50

100

150

Amidar

0

30

60

90

120

1000

2000

3000

Asterix

0

30

60

90

120

2000

3000

4000

5000

6000

BeamRider

0

30

60

90

120

0

20

40

60

Breakout

0

30

60

90

120

20000

40000

60000

80000

CrazyClimber

0

30

60

90

120

2000

4000

6000

8000

DemonAttack

0

30

60

90

120

2000

3000

4000

5000

Gopher

0

30

60

90

120

5000

10000

15000

Hero

0

30

60

90

120

2000

4000

6000

Kangaroo

0

30

60

90

120

2000

3000

4000

5000

Phoenix

0

30

60

90

120

500

1000

1500

2000

2500

3000

Seaquest

0

30

60

90

120

Training Frames (Million)

2000

4000

6000

average episode score

Zaxxon

Rainbow -- RMSProp

Rainbow -- RMSProp -- reset

0

30

60

90

120

50

100

150

Amidar

0

30

60

90

120

1000

2000

3000

4000

Asterix

0

30

60

90

120

2000

3000

4000

5000

BeamRider

0

30

60

90

120

20

40

60

Breakout

0

30

60

90

120

40000

60000

80000

100000

CrazyClimber

0

30

60

90

120

2000

4000

6000

8000

DemonAttack

0

30

60

90

120

2000

3000

4000

5000

Gopher

0

30

60

90

120

5000

10000

15000

20000

Hero

0

30

60

90

120

0

2000

4000

6000

Kangaroo

0

30

60

90

120

1000

2000

3000

4000

5000

Phoenix

0

30

60

90

120

500

1000

1500

2000

2500

Seaquest

0

30

60

90

120

Training Frames (Million)

2000

4000

6000

average episode score

Zaxxon

Rainbow -- RMSProp

Rainbow -- RMSProp -- reset

0

30

60

90

120

50

100

150

Amidar

0

30

60

90

120

1000

2000

3000

4000

Asterix

0

30

60

90

120

3000

4000

5000

6000

BeamRider

0

30

60

90

120

10

20

30

40

50

60

Breakout

0

30

60

90

120

40000

60000

80000

100000

120000

CrazyClimber

0

30

60

90

120

2000

4000

6000

8000

DemonAttack

0

30

60

90

120

2000

3000

4000

5000

Gopher

0

30

60

90

120

5000

10000

15000

20000

25000

Hero

0

30

60

90

120

2000

4000

6000

8000

Kangaroo

0

30

60

90

120

2000

3000

4000

5000

Phoenix

0

30

60

90

120

500

1000

1500

2000

2500

3000

Seaquest

0

30

60

90

120

Training Frames (Million)

2000

4000

6000

8000

average episode score

Zaxxon

Rainbow -- RMSProp

Rainbow -- RMSProp -- reset

0

30

60

90

120

50

100

150

200

250

Amidar

0

30

60

90

120

2000

4000

6000

8000

10000

Asterix

0

30

60

90

120

2000

4000

6000

BeamRider

0

30

60

90

120

20

40

60

80

100

Breakout

0

30

60

90

120

40000

60000

80000

100000

120000

CrazyClimber

0

30

60

90

120

2500

5000

7500

10000

12500

DemonAttack

0

30

60

90

120

2000

3000

4000

5000

6000

Gopher

0

30

60

90

120

5000

10000

15000

20000

Hero

0

30

60

90

120

2500

5000

7500

10000

12500

Kangaroo

0

30

60

90

120

2000

4000

6000

8000

10000

12000

Phoenix

0

30

60

90

120

500

1000

1500

2000

2500

Seaquest

0

30

60

90

120

Training Frames (Million)

2000

4000

6000

8000

10000

average episode score

Zaxxon

Rainbow -- RMSProp

Rainbow -- RMSProp -- reset

0

30

60

90

120

0

200

400

600

800

Amidar

0

30

60

90

120

0

2000

4000

6000

8000

10000

Asterix

0

30

60

90

120

2000

4000

6000

BeamRider

0

30

60

90

120

0

100

200

300

Breakout

0

30

60

90

120

0

50000

100000

150000

CrazyClimber

0

30

60

90

120

0

5000

10000

15000

DemonAttack

0

30

60

90

120

0

5000

10000

15000

20000

Gopher

0

30

60

90

120

5000

10000

15000

20000

Hero

0

30

60

90

120

0

2000

4000

6000

Kangaroo

0

30

60

90

120

0

2000

4000

6000

Phoenix

0

30

60

90

120

0

500

1000

1500

2000

2500

Seaquest

0

30

60

90

120

Training Frames (Million)

0

2000

4000

6000

average episode score

Zaxxon

Rainbow -- RMSProp

Rainbow -- RMSProp -- reset

We now take the human-normalized median on 12 games and present them for each value of K .

0

30

60

90

120

0.0

0.2

0.4

0.6

0.8

K=500

0

30

60

90

120

0.2

0.4

0.6

0.8

1.0

0

30

60

90

120

0.3

0.4

0.5

0.6

0.7

K=2000

0

30

60

90

120

0.2

0.3

0.4

0.5

0.6

0.7

K=4000

0

30

60

90

120

0.2

0.3

0.4

0.5

0.6

K=6000

0

30

60

90

120

Training Frames (Million)

0.2

0.3

0.4

0.5

0.6

human-normalized median

K=8000

Rainbow -- RMSProp

Rainbow -- RMSProp -- reset

We now look at Rainbow with the Rectified Adam optimizer.

0

30

60

90

120

0

1000

2000

Amidar

0

30

60

90

120

0

50000

100000

150000

200000

250000

Asterix

0

30

60

90

120

5000

10000

15000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

CrazyClimber

0

30

60

90

120

25000

50000

75000

100000

DemonAttack

0

30

60

90

120

0

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

20000

40000

60000

Phoenix

0

30

60

90

120

0

10000

20000

30000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

25000

Zaxxon

Rainbow -- Rectified Adam

Rainbow -- Rectified Adam -- reset

average episode score

0

30

60

90

120

0

1000

2000

3000

4000

Amidar

0

30

60

90

120

0

50000

100000

150000

200000

250000

Asterix

0

30

60

90

120

5000

10000

15000

20000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

0

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

50000

100000

150000

200000

250000

Phoenix

0

30

60

90

120

0

2000

4000

6000

8000

10000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

10000

20000

average episode score

Zaxxon

Rainbow -- Rectified Adam

Rainbow -- Rectified Adam -- reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

100000

200000

300000

Asterix

0

30

60

90

120

5000

10000

15000

20000

BeamRider

0

30

60

90

120

0

100

200

300

400

500

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

0

10000

20000

30000

40000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

100000

200000

300000

Phoenix

0

30

60

90

120

0

20000

40000

60000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

25000

average episode score

Zaxxon

Rainbow -- Rectified Adam

Rainbow -- Rectified Adam -- reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

50000

100000

150000

Asterix

0

30

60

90

120

0

5000

10000

15000

20000

25000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

20000

40000

60000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

50000

100000

150000

200000

Phoenix

0

30

60

90

120

0

10000

20000

30000

40000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow -- Rectified Adam

Rainbow -- Rectified Adam -- reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

20000

40000

60000

Asterix

0

30

60

90

120

0

5000

10000

15000

20000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

20000

40000

60000

80000

100000

DemonAttack

0

30

60

90

120

0

10000

20000

30000

40000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

20000

40000

60000

80000

Phoenix

0

30

60

90

120

0

5000

10000

15000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow -- Rectified Adam

Rainbow -- Rectified Adam -- reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

20000

40000

60000

80000

100000

Asterix

0

30

60

90

120

0

2500

5000

7500

10000

12500

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

10000

20000

30000

40000

50000

DemonAttack

0

30

60

90

120

0

10000

20000

30000

40000

Gopher

0

30

60

90

120

0

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

10000

20000

30000

40000

50000

Phoenix

0

30

60

90

120

0

5000

10000

15000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow -- Rectified Adam

Rainbow -- Rectified Adam -- reset

We now take the human-normalized median on 12 games and present them for each value of K .

0

30

60

90

120

0

1

2

3

K=500

0

30

60

90

120

0

1

2

3

4

0

30

60

90

120

0

2

4

K=2000

0

30

60

90

120

0

2

4

6

K=4000

0

30

60

90

120

0

2

4

6

K=6000

0

30

60

90

120

Training Frames (Million)

0

1

2

3

4

5

human-normalized median

K=8000

Rainbow -- Rectified Adam

Rainbow -- Rectified Adam -- reset

We now move to the case of DQN-Adam.

0

30

60

90

120

100

200

300

400

500

600

Amidar

0

30

60

90

120

2000

4000

6000

8000

Asterix

0

30

60

90

120

3000

4000

5000

6000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

25000

50000

75000

100000

125000

CrazyClimber

0

30

60

90

120

2000

4000

6000

DemonAttack

0

30

60

90

120

2000

4000

6000

8000

Gopher

0

30

60

90

120

0

5000

10000

15000

Hero

0

30

60

90

120

0

2500

5000

7500

10000

12500

Kangaroo

0

30

60

90

120

4000

5000

6000

Phoenix

0

30

60

90

120

2000

4000

6000

8000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

2000

4000

6000

8000

average episode score

Zaxxon

DQN -- Adam

DQN -- Adam -- reset

0

30

60

90

120

100

200

300

400

500

600

Amidar

0

30

60

90

120

1000

2000

3000

4000

5000

6000

Asterix

0

30

60

90

120

2000

3000

4000

5000

6000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

25000

50000

75000

100000

125000

CrazyClimber

0

30

60

90

120

2000

4000

6000

DemonAttack

0

30

60

90

120

2000

4000

6000

8000

Gopher

0

30

60

90

120

0

5000

10000

15000

Hero

0

30

60

90

120

0

5000

10000

Kangaroo

0

30

60

90

120

3000

4000

5000

6000

Phoenix

0

30

60

90

120

0

2000

4000

6000

8000

10000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

2000

4000

6000

8000

average episode score

Zaxxon

DQN -- Adam

DQN -- Adam -- reset

0

30

60

90

120

100

200

300

400

500

Amidar

0

30

60

90

120

1000

2000

3000

4000

5000

Asterix

0

30

60

90

120

2000

3000

4000

5000

6000

BeamRider

0

30

60

90

120

50

100

150

200

250

300

Breakout

0

30

60

90

120

25000

50000

75000

100000

125000

CrazyClimber

0

30

60

90

120

2000

4000

6000

DemonAttack

0

30

60

90

120

2000

4000

6000

8000

Gopher

0

30

60

90

120

0

2500

5000

7500

10000

12500

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

3000

4000

5000

Phoenix

0

30

60

90

120

2000

4000

6000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

2000

4000

6000

average episode score

Zaxxon

DQN -- Adam

DQN -- Adam -- reset

0

30

60

90

120

0

200

400

600

Amidar

0

30

60

90

120

1000

2000

3000

4000

Asterix

0

30

60

90

120

2000

3000

4000

5000

6000

BeamRider

0

30

60

90

120

50

100

150

200

250

300

Breakout

0

30

60

90

120

20000

40000

60000

80000

CrazyClimber

0

30

60

90

120

2000

4000

6000

DemonAttack

0

30

60

90

120

2000

4000

6000

8000

Gopher

0

30

60

90

120

0

2000

4000

6000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

3000

4000

5000

Phoenix

0

30

60

90

120

0

2000

4000

6000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

2000

4000

6000

average episode score

Zaxxon

DQN -- Adam

DQN -- Adam -- reset

0

30

60

90

120

0

200

400

600

Amidar

0

30

60

90

120

0

1000

2000

3000

4000

5000

Asterix

0

30

60

90

120

2000

3000

4000

5000

6000

BeamRider

0

30

60

90

120

0

50

100

150

200

Breakout

0

30

60

90

120

0

5000

10000

15000

20000

25000

CrazyClimber

0

30

60

90

120

1000

2000

3000

4000

5000

6000

DemonAttack

0

30

60

90

120

2000

4000

6000

Gopher

0

30

60

90

120

0

1000

2000

3000

Hero

0

30

60

90

120

0

2500

5000

7500

10000

12500

Kangaroo

0

30

60

90

120

2000

3000

4000

5000

Phoenix

0

30

60

90

120

0

2000

4000

6000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

1000

2000

3000

average episode score

Zaxxon

DQN -- Adam

DQN -- Adam -- reset

0

30

60

90

120

0

200

400

600

800

Amidar

0

30

60

90

120

0

1000

2000

3000

4000

5000

Asterix

0

30

60

90

120

2000

4000

6000

BeamRider

0

30

60

90

120

0

20

40

60

80

100

Breakout

0

30

60

90

120

0

1000

2000

3000

CrazyClimber

0

30

60

90

120

1000

2000

3000

4000

DemonAttack

0

30

60

90

120

0

1000

2000

3000

Gopher

0

30

60

90

120

0

200

400

600

800

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

1000

2000

3000

4000

5000

Phoenix

0

30

60

90

120

0

2000

4000

6000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

500

1000

average episode score

Zaxxon

DQN -- Adam

DQN -- Adam -- reset

We now take the human-normalized median on 12 games and present them for each value of K .

0

30

60

90

120

0.0

0.2

0.4

0.6

0.8

1.0

K=500

0

30

60

90

120

0.0

0.2

0.4

0.6

0.8

1.0

0

30

60

90

120

0.0

0.2

0.4

0.6

0.8

1.0

K=2000

0

30

60

90

120

0.0

0.2

0.4

0.6

0.8

1.0

K=4000

0

30

60

90

120

0.0

0.2

0.4

0.6

0.8

1.0

K=6000

0

30

60

90

120

Training Frames (Million)

0.0

0.2

0.4

0.6

0.8

1.0

human-normalized median

K=8000

DQN -- Adam

DQN -- Adam -- reset

We now move to the Rainbow Pro agent with Adam.

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

100000

200000

300000

Asterix

0

30

60

90

120

0

10000

20000

30000

BeamRider

0

30

60

90

120

0

100

200

300

400

500

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

50000

100000

DemonAttack

0

30

60

90

120

0

10000

20000

30000

40000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

100000

200000

Phoenix

0

30

60

90

120

0

5000

10000

15000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

10000

20000

30000

Zaxxon

RainbowPro -- Adam

RainbowPro -- Adam -- reset

average episode score

average episode score

0

30

60

90

120

0

500

1000

1500

2000

2500

Amidar

0

30

60

90

120

0

100000

200000

300000

Asterix

0

30

60

90

120

0

5000

10000

15000

20000

25000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

50000

100000

DemonAttack

0

30

60

90

120

0

10000

20000

30000

Gopher

0

30

60

90

120

20000

40000

60000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

100000

200000

300000

Phoenix

0

30

60

90

120

0

20000

40000

60000

80000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

25000

Zaxxon

RainbowPro -- Adam

RainbowPro -- Adam -- reset

0

30

60

90

120

0

1000

2000

3000

4000

Amidar

0

30

60

90

120

0

50000

100000

150000

200000

250000

Asterix

0

30

60

90

120

0

5000

10000

15000

20000

25000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

50000

100000

150000

200000

Phoenix

0

30

60

90

120

0

20000

40000

60000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

25000

average episode score

Zaxxon

RainbowPro -- Adam

RainbowPro -- Adam -- reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

100000

200000

300000

Asterix

0

30

60

90

120

5000

10000

15000

20000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

50000

100000

150000

200000

Phoenix

0

30

60

90

120

0

10000

20000

30000

40000

50000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

25000

average episode score

Zaxxon

RainbowPro -- Adam

RainbowPro -- Adam -- reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

100000

200000

300000

Asterix

0

30

60

90

120

5000

10000

15000

20000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

20000

40000

60000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

50000

100000

150000

200000

Phoenix

0

30

60

90

120

0

5000

10000

15000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

RainbowPro -- Adam

RainbowPro -- Adam -- reset

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

50000

100000

150000

200000

Asterix

0

30

60

90

120

0

5000

10000

15000

20000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

200000

CrazyClimber

0

30

60

90

120

0

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

20000

40000

60000

80000

100000

Phoenix

0

30

60

90

120

0

2500

5000

7500

10000

12500

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

RainbowPro -- Adam

RainbowPro -- Adam -- reset

We now take the human-normalized median on 12 games and present them for each value of K .

0

30

60

90

120

0

1

2

3

4

5

K=500

0

30

60

90

120

0

2

4

0

30

60

90

120

0

1

2

3

4

5

K=2000

0

30

60

90

120

0

2

4

6

K=4000

0

30

60

90

120

0

1

2

3

4

5

K=6000

0

30

60

90

120

Training Frames (Million)

0

2

4

6

human-normalized median

K=8000

RainbowPro -- Adam

RainbowPro -- Adam -- reset

7.4 Complete Results from Section 4.3

Wenowshow the full learning curves pertaining to Section 4.3 where we studied resetting individual moments.

average episode score

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

25000

50000

75000

100000

125000

Asterix

0

30

60

90

120

2500

5000

7500

10000

12500

15000

BeamRider

0

30

60

90

120

100

200

300

400

Breakout

0

30

60

90

120

50000

75000

100000

125000

150000

175000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

100000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

60000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

5000

10000

15000

20000

25000

30000

Phoenix

0

30

60

90

120

0

5000

10000

15000

20000

25000

Seaquest

0

30

60

90

120

0

5000

10000

15000

20000

Zaxxon

Rainbow

Rainbow -- reset both moments

Rainbow -- reset 1st moment

Rainbow -- reset 2nd moment

Training Frames (Million)

average episode score

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

25000

50000

75000

100000

Asterix

0

30

60

90

120

2500

5000

7500

10000

12500

15000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

75000

100000

125000

150000

175000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

100000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

10000

20000

30000

40000

Phoenix

0

30

60

90

120

0

5000

10000

15000

20000

25000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

Zaxxon

Rainbow

Rainbow -- reset both moments

Rainbow -- reset 1st moment

Rainbow -- reset 2nd moment

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

25000

50000

75000

100000

125000

Asterix

0

30

60

90

120

2500

5000

7500

10000

12500

15000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

75000

100000

125000

150000

175000

CrazyClimber

0

30

60

90

120

20000

40000

60000

80000

100000

DemonAttack

0

30

60

90

120

5000

10000

15000

20000

25000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

10000

20000

30000

Phoenix

0

30

60

90

120

0

10000

20000

30000

40000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow

Rainbow -- reset both moments

Rainbow -- reset 1st moment

Rainbow -- reset 2nd moment

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

50000

100000

150000

Asterix

0

30

60

90

120

5000

10000

15000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

75000

100000

125000

150000

175000

CrazyClimber

0

30

60

90

120

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

0

10000

20000

30000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

20000

40000

60000

80000

Phoenix

0

30

60

90

120

0

10000

20000

30000

40000

50000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

Zaxxon

Rainbow

Rainbow -- reset both moments

Rainbow -- reset 1st moment

Rainbow -- reset 2nd moment

average episode score

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

50000

100000

150000

200000

Asterix

0

30

60

90

120

5000

10000

15000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

CrazyClimber

0

30

60

90

120

0

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

0

10000

20000

30000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

20000

40000

60000

80000

100000

Phoenix

0

30

60

90

120

0

5000

10000

15000

Seaquest

0

30

60

90

120

Training Frames (Million)

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow

Rainbow -- reset both moments

Rainbow -- reset 1st moment

Rainbow -- reset 2nd moment

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

0

25000

50000

75000

100000

125000

Asterix

0

30

60

90

120

5000

10000

15000

BeamRider

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

50000

100000

150000

CrazyClimber

0

30

60

90

120

0

25000

50000

75000

100000

DemonAttack

0

30

60

90

120

0

10000

20000

30000

40000

Gopher

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

0

50000

100000

Phoenix

0

30

60

90

120

0

5000

10000

15000

20000

Seaquest

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow

Rainbow -- reset both moments

Rainbow -- reset 1st moment

Rainbow -- reset 2nd moment

We now take the human-normalized median on 12 games and present them for each value of K .

0

30

60

90

120

0

1

2

3

4

5

K=500

0

30

60

90

120

0

1

2

3

4

5

0

30

60

90

120

0

1

2

3

4

5

K=2000

0

30

60

90

120

0

1

2

3

4

K=4000

0

30

60

90

120

0

1

2

3

4

K=6000

0

30

60

90

120

Training Frames (Million)

0

1

2

3

4

human-normalized median

K=8000

Rainbow

Rainbow -- reset both moments

Rainbow -- reset 1st moment

Rainbow -- reset 2nd moment

7.5 Complete Results From Section 4.4

We now show full learning curves for all 55 Atari games and over 10 random seeds. We benchmark three agents: the default Rainbow agent from the Dopamine (no reset), Rainbow with resetting the Adam optimizer, and Rainbow with resetting the rectified Adam optimizer.

0

30

60

90

120

5000

10000

15000

20000

AirRaid

0

30

60

90

120

1000

2000

3000

4000

5000

Alien

0

30

60

90

120

0

1000

2000

3000

Amidar

0

30

60

90

120

5000

10000

15000

Assault

0

30

60

90

120

0

100000

200000

Asterix

0

30

60

90

120

500

1000

1500

2000

Asteroids

0

30

60

90

120

0.0

0.2

0.4

0.6

0.8

1.0

1e6

Atlantis

0

30

60

90

120

0

500

1000

1500

BankHeist

0

30

60

90

120

0

10000

20000

30000

40000

BattleZone

0

30

60

90

120

0

5000

10000

15000

20000

BeamRider

0

30

60

90

120

1000

2000

3000

0

30

60

90

120

10

20

30

40

50

Bowling

0

30

60

90

120

0

25

50

75

100

Boxing

0

30

60

90

120

0

100

200

300

400

Breakout

0

30

60

90

120

1000

2000

3000

4000

5000

Carnival

0

30

60

90

120

0

25000

50000

75000

100000

ChopperCommand

0

30

60

90

120

50000

100000

150000

CrazyClimber

0

30

60

90

120

0

25000

50000

75000

100000

125000

DemonAttack

0

30

60

90

120

500

1000

1500

2000

2500

Enduro

0

30

60

90

120

75

50

25

0

25

50

0

30

60

90

120

20

25

30

Freeway

0

30

60

90

120

2000

4000

6000

8000

Frostbite

0

30

60

90

120

0

10000

20000

30000

Gopher

0

30

60

90

120

0

1000

2000

3000

Gravitar

0

30

60

90

120

10000

20000

30000

40000

50000

Hero

0

30

60

90

120

15

10

5

0

5

IceHockey

0

30

60

90

120

0

5000

10000

15000

Jamesbond

0

30

60

90

120

8000

6000

4000

2000

0

JourneyEscape

0

30

60

90

120

0

5000

10000

15000

Kangaroo

0

30

60

90

120

6000

7000

8000

9000

Krull

0

30

60

90

120

10000

20000

30000

40000

KungFuMaster

0

30

60

90

120

0

500

1000

1500

MontezumaRevenge

0

30

60

90

120

1000

2000

3000

4000

5000

MsPacman

0

30

60

90

120

4000

6000

8000

10000

NameThisGame

0

30

60

90

120

0

50000

100000

150000

200000

250000

Phoenix

0

30

60

90

120

200

100

0

100

Pitfall

0

30

60

90

120

10

0

10

20

Pong

0

30

60

90

120

2000

4000

6000

8000

Pooyan

0

30

60

90

120

0

5000

10000

15000

PrivateEye

0

30

60

90

120

0

10000

20000

30000

0

30

60

90

120

5000

10000

15000

20000

25000

Riverraid

0

30

60

90

120

20000

40000

60000

RoadRunner

0

30

60

90

120

20

40

60

Robotank

0

30

60

90

120

0

5000

10000

15000

20000

25000

Seaquest

0

30

60

90

120

0

500

1000

1500

2000

2500

Solaris

0

30

60

90

120

0

2500

5000

7500

10000

12500

SpaceInvaders

0

30

60

90

120

0

25000

50000

75000

100000

125000

StarGunner

0

30

60

90

120

5000

10000

15000

TimePilot

0

30

60

90

120

100

150

200

250

Tutankham

0

30

60

90

120

20000

40000

60000

80000

UpNDown

0

30

60

90

120

0

200

400

600

800

1000

Venture

0

30

60

90

120

0

200000

400000

600000

VideoPinball

0

30

60

90

120

0

5000

10000

15000

20000

25000

WizardOfWor

0

30

60

90

120

20000

40000

60000

80000

YarsRevenge

0

30

60

90

120

Training Frames (Million)

0

5000

10000

15000

20000

average episode score

Zaxxon

Rainbow

Rainbow -- Adam -- reset

Rainbow -- Rectified Adam -- reset

7.6 Complete Results From Section 4.5

We finally present results on continuous control task with soft actor critic (SAC) and the Adam optimizer, where we reset the optimizers every 5000 steps. Note that, in contrsat to Atari and Rainbow, the target parameter θ is updated using the Polyak strategy, so it is less clear when to reset the optimizer. Thus we chose the simple strategy of resetting the optimizer every 5000 steps. We leave further exploration of resetting with Polyak updates to future work.

0

1

2

3

4

5

Time

×

10

6

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Average Return

×

10

4

HalfCheetah-v3

SAC with reset

SAC

0

1

2

3

4

5

Time

×

10

6

0

1

2

3

4

5

6

7

Average Return

×

10

3

SAC with reset

SAC

0

1

2

3

4

5

Time

×

10

6

0

1

2

3

4

Average Return

×

10

3

SAC with reset

SAC

0

1

2

3

4

5

Time

×

10

6

0

1

2

3

4

5

6

7

8

Average Return

×

10

3

Humanoid-v3

SAC with reset

SAC

0

1

2

3

4

5

Time

×

10

6

0

2

4

6

8

Average Return

×

10

3

SAC with reset

SAC

Peer reviews:
comments: SUMMARY:
The authors argue that Adam's internal parameters should be reset with each iteration. The authors demonstrate the effectiveness of this approach in the Atari domain.

STRENGTHS:
- Results are convincing for Rainbow.
- Novelty is very low, but potential impact is high, if the result generalizes, there is little reason not to use this method in every DQN-style RL algorithm.

WEAKNESSES:
- As mentioned by the authors, novelty is low compared to Bengio et al. 
- Results are only shown on Rainbow and do not appear to work for SAC (with reason) -- but does raise the question if the method is effective for other RL methods. 
- There is limited insight. Can the authors show that initializing Adam's parameters with 0 is better than using the parameters from the previous iteration in a more concrete way? Such as examining the behavior of the actual values. The fact that not resetting is seemingly better at low values of K suggests that not resetting can provide a reasonable initialization for the parameters of Adam.

Minor
- The y-axis is unlabelled in several figures.

QUESTIONS:
As mentioned in weaknesses:
- Does this result generalize to other methods besides Rainbow? Such as DQN or more modern deep RL methods.
- Can the authors show that initializing Adam's parameters with 0 is better than using the parameters from the previous iteration in a more concrete way? Such as examining the behavior of the actual values.

LIMITATIONS:
No concerns.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of using modern optimizers, such as Adam, which maintain internal parameters that are updated over time, potentially contaminating the optimization process. To mitigate this effect, the paper proposes a simple strategy of resetting the internal parameters of the optimizer at the start of each iteration. Empirical investigations using different optimizers and the Rainbow algorithm show that this modification enhances the performance of deep reinforcement learning on the Atari benchmark.\n\nSTRENGTHS:\n### Writing\nThe authors effectively communicate their ideas and concepts, ensuring clarity and coherence throughout the paper. The logical structure and well-reasoned arguments contribute to the overall quality of the essay. The article excels in providing the reader with a clear understanding of the problem's context and significance. By effectively conveying the goals and challenges of the study, the authors enhance the reader's comprehension of the subsequent experiments. Overall, the writing is of high quality, facilitating a smooth and engaging reading experience.\n\n### Method\nThe paper presents an easy-to-use approach by introducing a method that is not only easy to implement, but also easy to apply, which enhances the potential adoption and practicality of the proposed approach.This user-friendly feature makes the method highly accessible and beneficial to researchers and practitioners in various fields.\nThe used code bases and hyperparameters are provided, allowing the results to be reproduced.\n\nWEAKNESSES:\nWhile I appreciate the proposed method's ease of use, I believe that the authors could have conducted a more comprehensive and statistically rigorous analysis of their approach, considering its simplicity.\nOne notable limitation of the paper is the absence of confidence interval plots and statistical analysis, which could have been derived from [1], to enhance the clarity and precision of the findings. Incorporating these elements would have allowed readers to better understand the level of uncertainty associated with the reported results, thus bolstering the overall robustness of the study.\nFurthermore, the authors only rely on a single seed for the initial analysis, without providing a compelling rationale for this choice. Although using a single seed can streamline the experimental process, it diminishes the validity of the findings by disregarding potential result variations arising from multiple seeds. A more thorough explanation or a comparison of outcomes based on different seeds would have added value to the introduction, ensuring a more comprehensive analysis.\nI appreciate that the authors included continuous control tasks in their study; however, these tasks are not thoroughly explored. While the authors provide hypotheses to explain the unexpected results, a deeper analysis would have been expected.\n\nIn line 293, the authors reference a follow-up paper on resetting approaches but fail to cite the original work [2], which states in the section \"What and how to reset\" that resetting the optimizer has almost no significant impact due to quick updates of the moments. This contradicts the findings of this work.\nWhich brings me to the conclusion that I believe that the paper shows promise and the authors have taken a positive direction. However, in its current form, the paper falls short of being acceptable. It is essential to include comparisons to other baselines, such as [2], to provide a more thorough understanding of the opportunities and limitations, and to gain a clearer understanding of the internal effects in order to explain the aforementioned points.\n\n### Minor\n- The protocol for the random resets is not easy to understand and should be specified more clearly \n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.\n[2] Nikishin, Evgenii, et al. \"The primacy bias in deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\n- How does this method compare to other resetting approaches in terms of effectiveness?\n- What is the level of statistical significance observed in the results?\n- Why is resetting not effective for continuous control tasks?\n- Are there any experiments demonstrating the impact of contamination on the tasks discussed in this paper?\n- What are the consequences of reducing the frequency of optimizer resets beyond K=8000?\n- How can an optimal value for $K$ be determined?\n- Which ADAM/optimizer parameters are relevant when performing resets, i.e. have an effect when reset?\n- Are the observed effects still present when modifying the ADAM/optimizer hyperparameters?\n- Do different loss functions used in various DQN versions (e.g., MSE, Huber, Quantile) exhibit similar behaviors?\n\nLIMITATIONS:\nThe authors have made some effort to address the limitations; however, it is crucial for them to conduct a more comprehensive investigation into these limitations, as mentioned in the \"Weakness\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper questions the standard use of Adam-type optimizers in deep RL. The paper argues that solution methods in deep RL are best thought of as solving a sequence of optimization problems. And that the standard use of optimizers leads to \"contamination\" of the optimizer's internal parameters. The paper then proposes to reset the optimizer's internal parameters to fix this \"contamination.\" Finally, the experiments on Atari show that resetting the optimizer's internal parameters leads to significant performance improvement.\n\nSTRENGTHS:\nThe main strength of the paper is that the key idea of the paper, resetting optimizer parameters at the beginning of each iteration, is simple and effective. I liked the general theme of the paper, i.e., we need to understand better the tools we borrow from other fields. The paper is well-written and easy to understand, making it accessible to a wide audience. The experiments in section 4.3 are performed on 55 Atarti games with ten seeds each, which might mean the results are statistically significant. Resetting seems to be beneficial for multiple optimizers like RMSprop, Adam and Rectified Adam.\n\nWEAKNESSES:\nThe paper has two major weaknesses:\n1. The paper claims at many points that a \"contamination\" effect plagues RL (for example, lines 107-109) and that many updates are wasted to unlearn the effects of the previous iteration. However, the paper does not describe what exactly this \"contamination\" means, and neither does it show the presence of any \"contamination.\" All the paper shows is that there is a performance boost when we reset the optimizer's internal parameters. This performance boost is not direct evidence of contamination from iteration to iteration.\n\nHowever, this weakness can be easily overcome. The paper first needs to contain a definition of \"contamination,\" maybe the authors mean that the internal parameters($m$ and $v$) are too far away from their true values at the beginning of each iteration. One way to measure this difference could be to measure the cosine similarity between the current value of $m$ and the true value of $m$. The true value can be measured by taking the gradient of all the samples in the buffer and taking steps using that gradient. A large difference between the true and current value of $m$ would mean contamination. The paper also suggests that this contamination is particularly large at the beginning of each iteration compared to a random time in the learning process. Again, this can be easily shown by showing that the difference in true $m$ and current $m$ is larger at the beginning of the iteration compared to any random time in the learning process. \n\n2. None of the results in the paper except the ones in section 4.3 are statistically significant. The paper only shows results for a single random seed. I should note that the authors are aware of this weakness (line 138). The best way to look at the current experiments in Sections 4.1 and 4.2 is that they are used to tune hyper-parameters for the experiments in Section 4.3. The authors might be limited in their computational resources, but in that case, it is better to present statistically significant results in smaller environments like MinAtar[1] than unreplicable results in a big environment. \n\nOther than these two main problems, there are a few other minor issues in the paper.\n1. The update equations for Adam in Section 3 are wrong. Instead of using $m$ and $v$ for the final update, new variables $\\hat{m}$ and $\\hat{v}$ are used. See to the original Adam paper for the correct equations.  \n2. Line 176 says that $K=1$ corresponds to vanilla gradient descent. But, that is not true. For $K=1$, the update is similar to the Rprop optimizer, not SGD. For $K=1$, the update only takes into account the sign of the partial derivative but not its magnitude. \n3. The value of $K$ is not properly tuned. In Figure 4, the difference between $K=1000$ and $K=500$ is insignificant. So, the optimal value of $K$ could be smaller. I suggest the authors also try smaller values of K, like 250, 125, etc. \n\nI like the ideas presented in the paper. However, I can not recommend accepting the paper in its current form in light of these weaknesses. \n\n[1] Young, K., & Tian, T. (2019). Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176.\n\nQUESTIONS:\nWhat would be a good definition of \"contamination\"?\n\nLIMITATIONS:\nNo confidence intervals are reported for any experiment in the paper. I recommend the authors report the 95% bootstrapped confidence intervals for their results. [2] and [3] provide good guidelines for properly reporting experimental results in deep RL.\n\n\n[2] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.\n[3] Patterson, A., Neumann, S., White, M., & White, A. (2023). Empirical Design in Reinforcement Learning. arXiv preprint arXiv:2304.01315.\n\nEDIT:\n\nI have updated my score based on the new results provided by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies optimization in value-based deep reinforcement learning. The key insight is that when using target networks for action-value function training, changes in the target parameters yield a change in the optimization problem the online parameters are solving. Because of that, the authors argue that preserving the adaptive optimizer statistics (e.g. of Adam) might or might not be desirable. The paper then studies the effect of resetting the optimizer state after (hard) target updates mostly using the Rainbow algorithm on Atari games as a testbed yielding a slight positive aggregate improvement.\n\nSTRENGTHS:\nThe main strength of the paper is the simplicity of the contribution; the paper is well-written and easy to follow, and the method is motivated and described well. The experimental protocol is solid: it uses the full set of 55 Atari games and a standard Rainbow implementation.\n\nWEAKNESSES:\nThe main weakness of the paper is the mixed empirical results. Granted, the median human-normalized performance improves from ~1.75 to ~2.25, however, per-game effects from resetting the optimizer are highly heterogeneous, yielding performance deterioration in ~14 environments. The soundness of the paper could have been higher if, at least, an explanation (supported by evidence) for the negative effects was given.\n\nQUESTIONS:\nTarget network parameter updates indeed change the loss landscape that the online parameters are navigating. In addition to that, updating the replay buffer changes the distribution of inputs and hence the optimization problem for online parameters. Do you have ideas on how an optimizer could be changed to adapt to the input shifts?\n\nLIMITATIONS:\n“We hypothesize that this can contaminate the internal parameters of the employed optimizer in situations where the optimization landscape of the previous iterations is quite different from the current iteration.” (L9) The reviewer didn’t find an empirical verification of this assumption.\n\nMany deep RL algorithms use moving average target updates after each step instead of periodic hard updates. The authors demonstrate preliminary evidence that in soft actor-critic that uses such a practice, the optimizer resets do not improve the performance. Having said that, the reviewer appreciates the transparency about the negative results.\n\nAgain, one of the limitations is that in some environments resetting the optimizer yields negative results. It implies that a better alternative could be triggering the optimizer reset using a criterion (e.g. based on a measure of the loss landscape change / by performing a lookahead and assessing whether the reset was helpful)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}