Title: Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: lifelong reinforcement learning

parameter-free optimization

continual reinforcement learning

loss of plasticity
Abstract: A key challenge in lifelong reinforcement learning (RL) is the loss of plasticity, where previous learning progress hinders an agent's adaptation to new tasks. While regularization and resetting can help, they require precise hyperparameter selection at the outset and environment-dependent adjustments. Building on the principled theory of online convex optimization, we present a parameter-free optimizer for lifelong RL, called TRAC, which requires no tuning or prior knowledge about the distribution shifts. Extensive experiments on Procgen, Atari, and Gym Control environments show that TRAC works surprisingly well—mitigating loss of plasticity and rapidly adapting to challenging distribution shifts—despite the underlying optimization problem being nonconvex and nonstationary.

Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning

Abstract

A key challenge in lifelong reinforcement learning (RL) is the loss of plasticity, where previous learning progress hinders an agent's adaptation to new tasks. While regularization and resetting can help, they require precise hyperparameter selection at the outset and environment-dependent adjustments. Building on the principled theory of online convex optimization, we present a parameter-free optimizer for lifelong RL, called TRAC, which requires no tuning or prior knowledge about the distribution shifts. Extensive experiments on Procgen, Atari, and Gym Control environments show that TRAC works surprisingly well-mitigating loss of plasticity and rapidly adapting to challenging distribution shifts-despite the underlying optimization problem being nonconvex and nonstationary. Project website and code is available here.

1 Introduction

Spot, the agile robot dog, has been learning to walk confidently across soft, lush grass. But when Spot moves from the grassy field to a gravel surface, the small stones shift beneath her feet, causing her to stumble. When Spot tries to walk across a sandy beach or on ice, the challenges multiply, and her once-steady walk becomes erratic. Spot wants to adjust quickly to these new terrains, but the patterns she learned on grass are not suited to gravel, sand, or ice. Furthermore, she never knows when the terrain will change again and how different it will be, therefore must continually plan for the unknown while avoiding reliance on outdated experiences.

Spot's struggle exemplifies a well-known and extensively studied challenge in real-world decision making: lifelong reinforcement learning (lifelong RL) Abel et al. (2024); Nath et al. (2023); Mendez

0.8

1.0

1.2

1.4

1.6

1.8

2.0

2.2

2.4

Timesteps

1e7

0

5

10

15

20

25

30

35

Mean Episode Rewards

Starpilot

et al. (2020); Xie & Finn (2022). In lifelong RL, the learning agent must continually acquire new knowledge to adapt to the nonstationarity of the environment. At first glance, there appears to be

Zhiyu Zhang

an obvious solution: given a policy gradient oracle, the agent could just keep running gradient descent nonstop. However, recent experiments have demonstrated an intriguing behavior called loss of plasticity (Dohare et al., 2021; Lyle et al., 2022; Abbas et al., 2023; Sokar et al., 2023): despite persistent gradient steps, such an agent can gradually lose its responsiveness to incoming observations. There are even extreme cases of loss of plasticity (known as negative transfer or primacy bias ), where prior learning can significantly hamper the performance in new tasks (Nikishin et al., 2022; Ahn et al., 2024); see Figure 1 for an example. All these suggest that the problem is more involved than one might think.

From the optimization perspective, the above issues might be attributed to the lack of stability under gradient descent. That is, the weights of the agent's parameterized policy can drift far away from the origin (or a good initialization), leading to a variety of undesirable behaviors. 1 Fitting this narrative, it has been shown that simply adding a L 2 regularizer to the optimization objective (Kumar et al., 2023) or periodically resetting the weights (Dohare et al., 2021; Asadi et al., 2024; Sokar et al., 2023; Ahn et al., 2024) can help mitigate the problem. However, a particularly important limitation is their use of hyperparameters , such as the magnitude of the regularizer and the resetting frequency 2 . Good performance hinges on the suitable environment-dependent hyperparameter, but how can one confidently choose that before interacting with the environment? The classical cross-validation approach would violate the one-shot nature of lifelong RL (and online learning in general; see Chapter 1 of Orabona, 2023), since it is impossible to experience the same environment multiple times. This leads to the contributions of the present work.

Contribution The present work addresses the key challenges in lifelong RL using the principled theory of Online Convex Optimization (OCO). Specifically, our contributions are two fold.

Algorithm: TRAC Building on a series of results in OCO (Cutkosky & Orabona, 2018; Cutkosky, 2019; Cutkosky et al., 2023; Zhang et al., 2024b), we propose a (hyper)parameter-free optimizer for lifelong RL, called TRAC (Adap T ive R egulariz A tion in C ontinual environments). Intuitively, the idea is a refinement of regularization: instead of manually selecting the magnitude of regularization beforehand, TRAC chooses that in an online, data-dependent manner. From the perspective of OCO theory, TRAC is insensitive to its own hyperparameter, which means that no hyperparameter tuning is necessary in practice. Furthermore, as an optimization approach to lifelong RL, TRAC is compatible with any policy parameterization method.

Experiment Using Proximal Policy Optimization (PPO) (Schulman et al., 2017), we conduct comprehensive experiments on the instantiation of TRAC called TRAC PPO. A diverse range of lifelong RL environments are tested (based on Procgen, Atari, and Gym Control), with considerably larger scale than prior works. In settings where existing approaches (Abbas et al., 2023; Kumar et al., 2023; Nath et al., 2023) struggle, we find that TRAC PPO

-mitigates mild and extreme loss of plasticity;

-and rapidly adapts to new tasks when distribution shifts are introduced.

Such findings might be surprising: the theoretical advantage of TRAC is motivated by the convexity in OCO, but lifelong RL is both nonconvex and nonstationary in terms of optimization.

Organization Section 2 surveys the basics of lifelong RL. Section 3 introduces our parameter-free algorithm TRAC, and experiments are presented in Section 4. We defer the discussion of related works and results to Section 5. Finally, Section 6 concludes the paper.

2 Lifelong RL

As a sequential decision making framework, reinforcement learning (RL) is commonly framed as a Markov Decision Process (MDP) defined by the state space S , the action space A , the transition dynamics P ( s t +1 | s t , a t ) , and the reward function R ( s t , a t , s t +1 ) . In the t -th round, starting from a state s t ∈ S , the learning agent needs to choose an action a t ∈ A without knowing P and R . Then, the environment samples a new state s t +1 ∼ P ( ·| s t , a t ) , and the agent receives a reward

r t = R ( s t , a t , s t +1 ) . There are standard MDP objectives driven by theoretical tractability, but from a practical perspective, we measure the agent's performance by its cumulative reward ∑ T t =1 r t .

The standard setting above concerns a stationary MDP. Motivated by the prevalence of distribution shifts in practice, the present work studies a nonstationary variant called lifelong RL, where the transition dynamics P t and the reward function R t can vary over time. Certainly, one should not expect any meaningful 'learning' against arbitrary unstructured nonstationarity. Therefore, we implicitly assume P t and R t to be piecewise constant over time, and each piece is called a task -just like our example of Spot in the introduction. The main challenge here is to transfer previous learning progress to new tasks. This is reasonable when tasks are similar, but we also want to reduce the degradation when tasks turn out to be very different.

Lifelong RL as online optimization Deep RL approaches, including PPO (Schulman et al., 2017) and others, crucially utilize the idea of policy parameterization . Specifically, a policy refers to the distribution of the agent's action a t (conditioned on the historical observations), and we use θ t ∈ R d to denote the parameterizing weight vector . After sampling a t and receiving new observations, the agent could define a loss function J t ( θ ) that characterizes the 'hypothetical performance' of each weight θ ∈ R d . Then, by computing the policy gradient g t = ∇ J t ( θ t ) , one could apply a first order optimization algorithm 3 OPT to obtain the updated weight, θ t +1 = OPT ( θ t , g t ) .

For the rest of this paper, we will work with such an abstraction. The feedback of the environment is treated as a policy gradient oracle G , which maps the time t and the current weight θ t into a policy gradient g t = G ( t, θ t ) . Our goal is to design an optimizer OPT well suited for lifelong RL.

Lifelong vs. Continual In the RL literature, the use of 'lifelong' and 'continual' varies significantly across studies, which may lead to confusion. Abel et al. (2024) characterized continual reinforcement learning (CRL) as a never-ending learning process. However, much of the literature cited under CRL, such as (Abbas et al., 2023; Ahn et al., 2024), primarily focuses on the problem of backward transfer (avoiding catastrophic forgetting). Various policy-based architectures, such as those proposed by Rolnick et al. (2019); Schwarz et al. (2018); Nath et al. (2023), focus on tackling this issue. Conversely, the present work addresses the problem of forward transfer , which refers to the rapid adaptation to new tasks. Because of this we use 'lifelong' rather than 'continual' in our exposition, similar to (Thrun, 1996; Abel et al., 2018b; Julian et al., 2020).

3 Method

Inspired by (Cutkosky et al., 2023), we study lifelong RL by exploiting its connection to Online Convex Optimization (OCO; Zinkevich, 2003). The latter is a classical theoretical problem in online learning, and much effort has been devoted to designing parameter-free algorithms that require minimum tuning or prior knowledge (Streeter & Mcmahan, 2012; McMahan & Orabona, 2014; Orabona & Pál, 2016; Foster et al., 2017; Cutkosky & Orabona, 2018; Mhammedi & Koolen, 2020; Chen et al., 2021; Jacobsen & Cutkosky, 2022). The surprising observation of Cutkosky et al. (2023) is that several algorithmic ideas closely tied to the convexity of OCO can actually improve the nonconvex deep learning training, suggesting certain notions of 'near convexity' on its

New

Task

Previous

Task

Weight Parameterization

Loss

Base

Base

Update

θ

ref

Scaled

Update

TRAC

decides strength

of regularization

loss landscape. We find that lifelong RL (which is both nonconvex and nonstationary in terms of optimization) exhibits a similar behavior, therefore a particularly strong algorithm (named TRAC) can be obtained from principled results in parameter-free OCO. Let us start from the background.

Basics of (parameter-free) OCO As a standalone theoretical topic, OCO concerns a sequential optimization problem where the convex loss function l t can vary arbitrarily over time. In the t -th

iteration, the optimization algorithm picks an iterate x t and then observes a gradient g t = ∇ l t ( x t ) . Motivated by the pursuit of 'convergence' in optimization, the standard objective is to guarantee low (i.e., sublinear in T ) static regret , defined as

where T is the total number of rounds, and u is a comparator that the algorithm does not know beforehand. In other words, the goal is to make Regret T ( l 1: T , u ) small for all possible loss sequence l 1: T and comparator u . Note that for nonstationary OCO problems analogous to lifelong RL, it is better to consider a different objective called the discounted regret . Algorithms there mostly follow the same principle as in the stationary setting, just wrapped by loss rescaling (Zhang et al., 2024a).

For minimizing static regret, classical minimax algorithms like gradient descent (Zinkevich, 2003) would assume a small uncertainty set U at the beginning. Then, by setting the hyperparameter (such as the learning rate) according to U , it is possible to guarantee sublinear worst case regret ,

In contrast, parameter-free algorithms use very different strategies 4 to bound Regret T ( l 1: T , u ) directly (without taking the maximum) by a function of both l 1: T and u . The resulting bound is more refined than Eq.(1) (Orabona, 2023, Chapter 9), and crucially, since there is no need to pick an uncertainty set U , much less hyperparameter tuning is needed. This is where its name comes from.

TRAC for Lifelong RL: In lifelong RL, a key issue is the excessive drifting of weights θ t , which can detrimentally affect adapting to new tasks. To address this, TRAC enforces proximity to a well-chosen reference point θ ref , providing a principled solution derived from a decade of research in parameter-free OCO. Unlike traditional methods such as L 2 regularization or resetting, TRAC avoids hyperparameter tuning, utilizing the properties of OCO to maintain weight stability and manage the drift effectively.

The core of TRAC, similar to other parameter-free optimizers, incorporates three techniques:

Direction-Magnitude Decomposition : Inspired by Cutkosky & Orabona (2018), this technique employs a carefully designed one-dimensional algorithm, the "parameter-free tuner," atop a base optimizer. This setup acts as a data-dependent regularizer, controlling the extent to which the iterates deviate from their initialization, thereby minimizing loss of plasticity, which is crucial given the high plasticity at the initial policy parameterization (Abbas et al., 2023).

Erfi Potential Function : Building on the previous concept, the tuner utilizes the Erfi potential function, as developed by Zhang et al. (2024a). This function is crafted to effectively balance the distance of the iterates from both the origin and the empirical optimum. It manages the update magnitude by focusing on the gradient projection along the direction θ t -θ ref .

Additive Aggregation: The tuner above necessitates discounting. Thus, we employ Additive Aggregation by Cutkosky (2019). This approach enables the combination of multiple parameter-free OCO algorithms, each with different discount factors, to approximate the performance of the best-performing algorithm. Importantly, it facilitates the automatic selection of the optimal discount factor during training.

These three components crucially work together to guarantee good regret bounds in the convex setting and are the minimum requirement for any reasonable parameter-free optimizer.

Without going deep into the theory, here is an overview of the important ideas (also see Figure 2 for a visualization).

First, TRAC is a meta-algorithm that operates on top of a 'default' optimizer BASE. It can simply be gradient descent with a constant learning rate, or ADAM (Kingma & Ba, 2014) as in our experiments. Applying BASE alone would be equivalent to enforcing the scaling parameter S t +1 ≡ 1 in TRAC, but this would suffer from the drifting of θ Base t +1 (and thus, the weight θ t +1 ).

Algorithm 1 TRAC: Parameter-free Adaption for Continual Environments.

1: Input: A policy gradient oracle G ; a first order optimization algorithm BASE; a reference point θ ref ∈ R d ; n discount factors β 1 , . . . , β n ∈ (0 , 1] (default: 0 . 9 , 0 . 99 , . . . , 0 . 999999 ).

2: Initialize: Create n copies of Algorithm 2, denoted as A 1 , . . . , A n . For each j ∈ [1 : n ] , A j uses the discount factor β j . Initialize the algorithm BASE at θ ref . Let θ 1 = θ ref .

3: for t = 1 , 2 , . . . do

4: Obtain the t -th policy gradient g t = G ( t, θ t ) ∈ R d .

5: Send g t to BASE as its t -th input, and get its output θ Base t +1 ∈ R d .

6: For all j ∈ [1 : n ] , send ⟨ g t , θ t -θ ref ⟩ to A j as its t -th input, and get its output s t +1 ,j ∈ R .

7: Define the scaling parameter S t +1 = ∑ n j =1 s t +1 ,j .

8: Update the weight of the policy,

9: end for

Algorithm 2 1D Discounted Tuner of TRAC.

1: Input: Discount factor β ∈ (0 , 1] ; small value ε > 0 (default: 10 -8 ).

2: Initialize: The running variance v 0 = 0 ; the running (negative) sum σ 0 = 0 .

3: for t = 1 , 2 , . . . do

4: Obtain the t -th input h t .

5: Let v t = β 2 v t -1 + h 2 t , and σ t = βσ t -1 -h t .

6: Select the t -th output

where erfi is the imaginary error function queried from standard software packages.

7: end for

To fix this issue, TRAC uses the tuner (Algorithm 2) to select the scaling parameter S t +1 , making it data-dependent . Typically S t +1 is within [0 , 1] (see Figure 17 to 19), therefore essentially, we define the updated weight θ t +1 as a convex combination of the BASE's weight θ Base t and the reference point θ ref ,

This brings the weight closer to θ ref , which is known to be 'safe' (i.e., not overfitting any particular lifelong RL task), although possibly conservative.

To inject the right amount of conservatism without hyperparameter tuning, the tuner (Algorithm 2) applies an unusual decision rule based on the erfi function. Theoretically, this is known to be optimal in an idealized variant of OCO (Zhang et al., 2022, 2024b), but removing the idealized assumptions requires a tiny bit of extra conservatism, which is challenging (and not necessarily practical). Focusing on the lifelong RL problem that considerably deviates from OCO, we simply apply the erfi decision rule as is. This is loosely motivated by deep learning training dynamics, e.g., (Cohen et al., 2020; Ahn et al., 2023; Andriushchenko et al., 2023), where an aggressive optimizer is often observed to be better.

Finally, the tuner requires a discount factor β . This crucially controls the strength of regularization (elaborated next), but also introduces a hyperparameter tuning problem. Following (Cutkosky, 2019), we aggregate tuners with different β (on a log-scaled grid) by simply summing up their outputs. This is justified by the adaptivity of the tuner itself: in OCO, if we add a parameter-free algorithm A 1 to any other algorithm A 2 that already works well, then A 1 can automatically identify this and 'tune down' its aggressiveness, such that A 1 + A 2 still performs as well as A 2 .

Connection to regularization Despite its nested structure, TRAC can actually be seen as a parameter-free refinement of L 2 regularization (Kumar et al., 2023). To concretely explain this intuition, let us consider the following two optimization dynamics.

First, suppose we run gradient descent with learning rate η , on the policy gradient sequence { g t } with the L 2 regularizer λ 2 ∥ θ -θ ref ∥ 2 . Quantitatively, it means that starting from the t -th weight θ t ,

That is, the updated weight θ t +1 is determined by a ( 1 -λη )-discounting with respect to the reference point θ ref , followed by a gradient step -ηg t .

Alternatively, consider applying the following simplification of TRAC on the same policy gradient sequence { g t } : ( i ) BASE is still gradient descent with learning rate η ; ( ii ) there is just one discount factor β ; and ( iii ) the one-dimensional tuner (Algorithm 2) is replaced by the β -discounted gradient descent with learning rate α , i.e., S t +1 = βS t -αh t . In this case, we have

̸

Notice that S t is a β -discounted sum of αh 1 , . . . , αh t -1 , thus in the typical situation of β ≈ 1 one might expect αh t ≪| S t | . Then, the resulting update of θ t +1 is similar to Eq.(2), with quantitative changes on the 'effective discounting' 1 -λη → β , and the 'effective learning rate' η → ηS t +1 .

The main message here is that under a simplified setting, TRAC is almost equivalent to L 2 regularization. The latter requires choosing the hyperparameters λ and η , and similarly, the above simplified TRAC requires choosing β and η . Going beyond this simplification, the actual TRAC removes the tuning of β using aggregation, and the tuning of η using the erfi decision rule.

On the hyperparameters Although TRAC is called 'parameter-free', it still needs the β -grid, the constant ε and the algorithm BASE as inputs. The idea is that TRAC is particularly insensitive to such choices, as supported by the OCO theory. As the result, the generic default values recommended by Cutkosky et al. (2023) are sufficient in practice. We note that those are proposed for training supervised deep learning models, thus should be agnostic to the lifelong RL applications we consider.

4 Experiment

Does TRAC experience the common pitfalls of loss of plasticity? Does it rapidly adapt to distribution shifts? To answer these questions, we test TRAC in empirical RL benchmarks such as vision-based games and physics-based control environments in lifelong settings (Figure 3). Specifically, we instantiate PPO with two different optimizers: ADAM with constant learning rate for baseline comparison, and TRAC for our proposed method (with exactly the same ADAM as the input BASE). We also test ADAM PPO with concatenated ReLU activations (CReLU; Shang et al., 2016), previously shown to mitigate loss of plasticity in certain deep RL settings (Abbas et al., 2023). Our numerical results are summarized in Table 1. Across every lifelong RL setting, we observe substantial improvements in the cumulative episode reward by using TRAC PPO compared to ADAM PPO or CReLU. Below are the details, with more in the Appendix.

Obs +=

[noise]

Obs +=

[noise]

Obs +=

[noise]

TRAC

Base

Atari

S

TRAC

Base

S

TRAC

Base

S

TRAC

Base

S

TRAC

Base

S

TRAC

Base

S

Procgen

Control

Procgen We first evaluate on OpenAI Procgen, a suite of 16 procedurally generated game environments (Cobbe et al., 2020). We introduce distribution shifts by sampling a new procedurally generated level of the current game every 2 million time steps, treating each level as a distinct task.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

0

10

20

30

40

Mean Episode Rewards

Starpilot

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

5

0

5

10

15

20

25

Mean Episode Rewards

Dodgeball

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

5.0

2.5

0.0

2.5

5.0

7.5

Mean Episode Rewards

Fruitbot

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

0

2

4

6

8

10

Mean Episode Rewards

Chaser

TRAC PPO

CReLU

We evaluate game environments including StarPilot, Dodgeball, Fruitbot, and Chaser. In all of these environments, we observe in Figure 4 that both ADAM PPO and CReLU encounter a continually degrading loss of plasticity as these distribution shifts are introduced. In contrast, TRAC PPO avoids this loss of plasticity, which contributes to its rapid reward increase when adapting to new levels. In the cumulative reward across all the Procgen levels, TRAC PPO reveals normalized average improvements of 3,212.42% and 120.88% over ADAM PPO and CReLU respectively (see Table 1). For later levels, in all games, TRAC PPO's reward does not decline as sharply as the baselines, potentially indicating positive transfer of skills from one level to the next.

One key advantage of TRAC is that it functions as an optimizer, making it orthogonal to various policy methods such as PPO, as well as other baselines like Online EWC (Schwarz et al., 2018), IMPALA (Espeholt et al., 2018), Modulating Masks (Nath et al., 2023), and CLEAR (Rolnick et al., 2019). In Appendix C, we evaluate these methods using both TRAC and ADAM on the Procgen setup. We find that in every environment, TRAC improves the performance of these algorithms.

Atari The Arcade Learning Environment (ALE) Atari 2600 benchmark is a collection of classic arcade games designed to assess reinforcement learning agents' performance across a range of diverse gaming scenarios (Bellemare et al., 2013). We introduce distribution shifts by switching to a new Atari game every 4 million timesteps, where each game switch introduces a new task. This benchmark is more challenging compared to OpenAI Procgen: it requires the agent to handle distribution shifts in both the input (state) and the target (reward).

In this experiment, we assessed two online settings distinguished by games with action spaces of 6 and 9. From Figure 5, both ADAM PPO and CReLU sometimes failed to learn in certain games. In contrast, TRAC PPO shows a substantial increase in reward over different games compared to the baselines. For example, during the first 12 million steps (3 games) in Atari 6, TRAC PPO not only achieves a significantly higher mean reward but also demonstrates rapid reward increase. Over both experiment settings, TRAC PPO shows an average normalized improvement of 329.73% over ADAM PPO and 68.71% over CReLU (Table 1). In rare instances, such as the last 2 million steps of Atari 6, CReLU performs comparably to TRAC PPO. This observation aligns with findings from (Abbas et al., 2023), which noted that CReLU tends to avoid plasticity loss in continual Atari setups.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

0

500

1000

1500

2000

2500

Mean Episode Rewards

Atari (6)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

0

5000

10000

Mean Episode Rewards

Atari (9)

TRAC PPO

CReLU

0

500

1000

1500

2000

2500

3000

Timesteps

0

100

200

300

400

500

Mean Episode Rewards

CartPole-v1

0

500

1000

1500

2000

2500

3000

Timesteps

500

400

300

200

100

0

Mean Episode Rewards

Acrobot-v1

0

500

1000

1500

2000

2500

3000

Timesteps

1000

750

500

250

0

250

Mean Episode Rewards

LunarLander-v2

TRAC PPO

CReLU

Gym Control We use the CartPole-v1 and Acrobot-v1 environments from the Gym Classic Control suite, along with LunarLander-v2 from Box2d Control. To introduce distribution shifts, Mendez et al. (2020) periodically alters the environment dynamics. Although such distribution shifts pose only mild challenges for robust methods like PPO with ADAM (Appendix D). We instead implement a more challenging form of distribution shift. Every 200 steps we perturb each observation dimension with random noise within a range of ± 2 , treating each perturbation phase as a distinct task.

Here (Figure 6), we notice a peculiar behavior after introducing the first distribution shift in both ADAM PPO and CReLU: policy collapse. We describe this as an extreme form of loss of plasticity. Surprisingly, TRAC PPO remains resistant to these extreme distribution shifts. As we see in the Acrobot experiment, TRAC PPO shows minimal to no policy damage after the first few distribution shifts, whereas ADAM PPO and CReLU are unable to recover a policy at all. We investigate if TRAC's behavior here indicates positive transfer in Appendix A. Across the three Gym Control environments,

TRAC PPO shows an average normalized improvement of 204.18% over ADAM PPO and 1044.24% over CReLU (Table 1).

5 Discussion

Related work Combating loss of plasticity has been studied extensively in lifelong RL. A typical challenge for existing solutions is the tuning of their hyperparameters, which requires prior knowledge on the nature of the distribution shift, e.g., (Asadi et al., 2024; Nath et al., 2023; Nikishin et al., 2024; Sokar et al., 2023; Mesbahi et al., 2024). An architectural modification called CReLU is studied in (Abbas et al., 2023), but our experiments suggest that its benefit might be specific to the Atari setup. Besides, Abel et al. (2018a,b) presented a theoretical analysis of skill transfer in lifelong RL, based on value iteration. Moreover, related contributions in nonstationary RL, where reward and state transition functions also change unpredictably, are limited to theoretical sequential decision-making settings with a focus on establishing complexity bounds (Roy et al., 2019; Cheung et al., 2020; Wei &Luo, 2021; Mao et al., 2020).

Our algorithm TRAC builds on a long line of works on parameter-free OCO (see Section 3). To our knowledge, the only existing work applying parameter-free OCO to RL is (Jacobsen & Chan, 2021), which focuses on estimating the value function (i.e., policy evaluation). Our scope is different, focusing on empirical RL in lifelong problems by exploring the key connection between parameterfree OCO and regularization.

Particularly, we are inspired by the MECHANIC algorithm from (Cutkosky et al., 2023), which goes beyond the traditional convex setting of parameter-free OCO to handle stationary deep learning optimization tasks. Lifelong reinforcement learning, however, introduces a layer of complexity with its inherent nonstationarity. Furthermore, compared to MECHANIC, TRAC improves the scale tuner there (which is based on the coin-betting framework; Orabona & Pál, 2016) by the erfi algorithm that enjoys a better OCO performance guarantee. As an ablation study, we empirically compare TRAC and MECHANIC in the Appendix G (Table 3). We find that TRAC is slightly better, but both algorithms can mitigate the loss of plasticity, suggesting the effectiveness of the general 'parameter-free' principle in lifelong RL.

TRAC encourages positive transfer In our experiments, we observe that TRAC's reward decline due to distribution shifts is less severe than that of baseline methods. These results may suggest TRAC facilitates positive transfer between related tasks. To investigate this further, we compared TRAC to a privileged weight-resetting approach, where the network's parameters are reset for each new task, in the Gym Control environments (see Appendix A). Our results show that TRAC maintains higher rewards during tasks than privileged weight-resetting and avoids declining to the same low reward levels as privileged weight-resetting at the start of a new task (Figure 8).

On the choice of θ ref In general, the reference point θ ref should be good or 'safe' for TRAC to perform effectively. One might presume that achieving this requires 'warmstarting', or pre-training using the underlying BASE optimizer. While our experiments validate that such warmstarting is indeed beneficial (Appendix B), our main experiments show that even a random initialization of the policy's weight serves as a good enough θ ref, even when tasks are similar (Figure 4).

This observation aligns with discussions by Lyle et al. (2023), Sokar et al. (2023), and Abbas et al. (2023), who suggested that persistent gradient steps away from a random initialization can deactivate ReLU activations, leading to activation collapse and loss of plasticity in neural networks. Our results also support Kumar et al. (2023)'s argument that maintaining some weights close to their initial values not only prevents dead ReLU units but also allows quick adaptation to new distribution shifts.

Tuning L 2 regularization The success of TRAC suggests that an adaptive form of regularizationanchoring to the reference point θ ref -may suffice to counteract both mild and extreme forms of loss of plasticity. From this angle, we further elaborate the limitation of the L 2 regularization approach considered in (Kumar et al., 2023). It requires selecting a regularization strength parameter λ through cross-validation, which is incompatible with the one-shot nature of lifelong learning settings. Furthermore, it is nontrivial to select the search grid: for example, we tried the λ -grid suggested by (Kumar et al., 2023), and there is no effective λ value within the grid for the lifelong RL environments we consider. All the values are too small.

1

2

3

4

5

6

7

8

9

10

Distribution Shift

0

20

40

Best

Sensitivity

0

500

1000

1500

Timesteps

0

200

400

Mean Episode Rewards

CartPole-v1

=25

0

500

1000

1500

Timesteps

400

200

0

Mean Episode Rewards

Acrobot-v1

=15

0

500

1000

1500

Timesteps

500

0

Mean Episode Rewards

LunarLander-v2

=45

Acrobot

CartPole

LunarLander

Well-tuned

TRAC PPO

Continuing this reasoning, we conduct a hyperparameter search for λ , over various larger values [0 . 2 , 0 . 8 , 1 , 5 , 10 , 15 , 20 , 25 , 30 , 35 , 40 , 45 , 50] . Given the expense of such experiments, only the more sample-efficient control environments are considered. We discover that each environment and task responds uniquely to these regularization strengths (see bar plot of λ values in Figure 7). This highlights the challenges of tuning λ in a lifelong learning context, where adjusting for each environment, let alone each distribution shift, would require extensive pre-experimental analysis.

In contrast, TRAC offers a parameter-free solution that adapts dynamically with the data in an online manner. The scaling output of TRAC adjusts autonomously to the ongoing conditions, consistently competing with well-tuned λ values in the various environments, as demonstrated in the reward plots for CartPole, Acrobot, and LunarLander (Figure 7).

TRAC compared to other plasticity methods Both layer normalization and plasticity injection Nikishin et al. (2024); Lyle et al. (2023) have been shown to combat plasticity loss. For instance, Appendix E Figure 15 demonstrates that both layer normalization and plasticity injection are effective at reducing plasticity loss when applied to the CartPole environment using ADAM as a baseline optimizer. We implemented plasticity injection following the methodology laid out by Nikishin et al. (2024), where plasticity is injected at the start of every distribution shift. While this approach does help in reducing the decline in performance due to plasticity loss, our results indicate that it is consistently outperformed by TRAC across all three control environments-CartPole, Acrobot, and LunarLander. Moreover, while layer normalization improves ADAM's performance, it too is outperformed by TRAC across the same control settings (Figure 15). Notably, combining layer normalization with TRAC resulted in the best performance gains.

Near convexity of lifelong RL Our results demonstrate the rapid adaptation of TRAC, in lifelong RL problems with complicated function approximation. From the perspective of optimization, the latter requires tackling both nonconvexity and nonstationarity, which is typically regarded intractable in theory. Perhaps surprisingly, when approaching this complex problem using the theoretical insights from OCO, we observe compelling results. This suggests a certain 'hidden convexity' in this problem, which could be an exciting direction for both theoretical and empirical research (e.g., policy gradient methods provably converge to global optimizers in linear quadratic control (Hu et al., 2023)).

Limitations While TRAC offers robust adaptability in nonstationary environments, it can exhibit suboptimal performance at the outset. In the early stages of deployment, TRAC might underperform compared to the baseline optimizer. We address this by proposing a warmstarting solution detailed in Appendix B, which helps increase the initial performance gap.

6 Conclusion

In this work, we introduced TRAC, a parameter-free optimizer for lifelong RL that leverages the principles of OCO. Our approach dynamically refines regularization in a data-dependent manner, eliminating the need for hyperparameter tuning. Through extensive experimentation in Procgen, Atari, and Gym Control environments, we demonstrated that TRAC effectively mitigates loss of plasticity and rapidly adapts to new distribution shifts, where baseline methods fail. TRAC's success leads to a compelling takeaway: empirical lifelong RL scenarios may exhibit more convex properties than previously appreciated, and might inherently benefit from parameter-free OCO approaches.

7 Acknowledgments

We thank Ashok Cutkosky for insightful discussions on online optimization in nonstationary settings. We are grateful to David Abel for his thoughtful insights on loss of plasticity in relation to lifelong reinforcement learning. We appreciate Kaiqing Zhang and Yang Hu for their comments on theoretical and nonstationary RL. This project is partially funded by Harvard University Dean's Competitive Fund for Promising Scholarship.

References

Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents , pp. 620-636. PMLR, 2023.

David Abel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State abstractions for lifelong reinforcement learning. In International Conference on Machine Learning , pp. 10-19. PMLR, 2018a.

David Abel, Yuu Jinnai, Sophie Yue Guo, George Konidaris, and Michael Littman. Policy and value transfer in lifelong reinforcement learning. In International Conference on Machine Learning , pp. 20-29. PMLR, 2018b.

David Abel, André Barreto, Benjamin Van Roy, Doina Precup, Hado P van Hasselt, and Satinder Singh. A definition of continual reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, and Taesup Moon. Catastrophic negative transfer: An overlooked problem in continual reinforcement learning, 2024. URL https://openreview.net/forum? id=o7BwUyXz1f .

Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via edge of stability. Advances in Neural Information Processing Systems , 36, 2023.

Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning , pp. 903-925. PMLR, 2023.

Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep rl: An empirical study. Advances in Neural Information Processing Systems , 36, 2024.

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.

Liyu Chen, Haipeng Luo, and Chen-Yu Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Conference on Learning Theory , pp. 1216-1259. PMLR, 2021.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In International conference on machine learning , pp. 1843-1854. PMLR, 2020.

Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning , pp. 2048-2056. PMLR, 2020.

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations , 2020.

Ashok Cutkosky. Combining online learning guarantees. In Conference on Learning Theory , pp. 895-913. PMLR, 2019.

Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in banach spaces. In Conference On Learning Theory , pp. 1493-1529. PMLR, 2018.

Ashok Cutkosky, Aaron Defazio, and Harsh Mehta. Mechanic: A learning rate tuner. Advances in Neural Information Processing Systems , 36, 2023.

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7):3366-3385, 2021.

Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325 , 2021.

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning , pp. 1407-1416. PMLR, 2018.

Huang Fang, Nicholas JA Harvey, Victor S Portella, and Michael P Friedlander. Online mirror descent and dual averaging: keeping pace in the dynamic case. Journal of Machine Learning Research , 23(1):5271-5308, 2022.

Dylan J Foster, Satyen Kale, Mehryar Mohri, and Karthik Sridharan. Parameter-free online learning via model selection. Advances in Neural Information Processing Systems , 30, 2017.

Bin Hu, Kaiqing Zhang, Na Li, Mehran Mesbahi, Maryam Fazel, and Tamer Ba¸ sar. Toward a theoretical foundation of policy optimization for learning control policies. Annual Review of Control, Robotics, and Autonomous Systems , 6:123-158, 2023.

Andrew Jacobsen and Alan Chan. Parameter-free gradient temporal difference learning. arXiv preprint arXiv:2105.04129 , 2021.

Andrew Jacobsen and Ashok Cutkosky. Parameter-free mirror descent. In Conference on Learning Theory , pp. 4160-4211. PMLR, 2022.

Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Efficient adaptation for end-to-end vision-based robotic manipulation. In 4th Lifelong Machine Learning Workshop at ICML 2020 , 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308.11958 , 2023.

Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations , 2022.

Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning , pp. 23190-23211. PMLR, 2023.

Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks, 2024. URL https://arxiv.org/ abs/2402.18762 .

Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi-Levi, and Tamer Ba¸ sar. Model-free non-stationary rl: Near-optimal regret and applications in multi-agent rl and inventory control. arXiv preprint arXiv:2010.03161 , 2020.

H Brendan McMahan and Francesco Orabona. Unconstrained online linear learning in hilbert spaces: Minimax algorithms and normal approximations. In Conference on Learning Theory , pp. 1020-1039. PMLR, 2014.

Jorge Mendez, Boyu Wang, and Eric Eaton. Lifelong policy gradient learning of factored policies for faster training without forgetting. Advances in Neural Information Processing Systems , 33:14398-14409, 2020.

Golnaz Mesbahi, Olya Mastikhina, Parham Mohammad Panahi, Martha White, and Adam White. Tuning for the unknown: Revisiting evaluation strategies for lifelong rl. arXiv preprint arXiv:2404.02113 , 2024.

Zakaria Mhammedi and Wouter M Koolen. Lipschitz and comparator-norm adaptivity in online learning. In Conference on Learning Theory , pp. 2858-2887. PMLR, 2020.

Saptarshi Nath, Christos Peridis, Eseoghene Ben-Iwhiwhu, Xinran Liu, Shirin Dora, Cong Liu, Soheil Kolouri, and Andrea Soltoggio. Sharing lifelong reinforcement learning knowledge via modulating masks. In Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup (eds.), Proceedings of The 2nd Conference on Lifelong Learning Agents , Proceedings of Machine Learning Research. PMLR, 2023.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning , pp. 16828-16847. PMLR, 2022.

Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems , 36, 2024.

Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2023.

Francesco Orabona and Dávid Pál. Coin betting and parameter-free online learning. Advances in Neural Information Processing Systems , 29, 2016.

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems , pp. 348-358, 2019. URL https://arxiv.org/abs/1811.11682 .

Abhishek Roy, Krishnakumar Balasubramanian, Saeed Ghadimi, and Prasant Mohapatra. Multi-point bandit algorithms for nonstationary online nonconvex optimization. arXiv preprint arXiv:1907.13616 , 2019.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Jonathan Schwarz, Jelena Luketina, Wojciech M Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In ICML , 2018. URL https://arxiv.org/abs/1805.06370 .

Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning , pp. 2217-2225. PMLR, 2016.

Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning , pp. 32145-32168. PMLR, 2023.

Matthew Streeter and Brendan Mcmahan. No-regret algorithms for unconstrained online convex optimization. Advances in Neural Information Processing Systems , 25, 2012.

Thrun. Explanation-Based Neural Network Learning: A Lifelong Learning Approach . Kluwer Academic Publishers, Boston, MA, 1996.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Conference on learning theory , pp. 4300-4354. PMLR, 2021.

Annie Xie and Chelsea Finn. Lifelong robotic reinforcement learning by retaining experiences, 2022.

Zhiyu Zhang, Ashok Cutkosky, and Ioannis Paschalidis. Pde-based optimal strategy for unconstrained online learning. In International Conference on Machine Learning , pp. 26085-26115. PMLR, 2022.

Zhiyu Zhang, David Bombara, and Heng Yang. Discounted adaptive online learning: Towards better regularization. In International Conference on Machine Learning , 2024a.

Zhiyu Zhang, Heng Yang, Ashok Cutkosky, and Ioannis C Paschalidis. Improving adaptive online learning using refined discretization. In International Conference on Algorithmic Learning Theory , pp. 1208-1233. PMLR, 2024b.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning , pp. 928-936, 2003.

Appendix

A TRAC Encourages Positive Transfer

To explore whether TRAC encourages positive transfer, we introduce a privileged weight-reset baseline. This baseline is "privileged" in the sense that it knows when a distribution shift is introduced and resets the parameters to a random initialization at the start of each new task. We applied this baseline to three Gym control tasks: CartPole-v1, Acrobot-v1, and LunarLander-v2, and compared it to TRAC PPO and ADAM PPO, as shown in Figure 8.

We observe that the privileged weight-reset baseline exhibits spikes in reward at the beginning of each new task. Surprisingly, TRAC maintains even higher rewards than the privileged weight-reset baseline, even at its peak learning phases. Additionally, TRAC's reward does not decline to the reward seen at the start of new tasks with privileged weight-resetting (TRAC does not have to "start over" with each task), suggesting that TRAC successfully transfers skills positively between tasks.

0

500

1000

1500

2000

2500

3000

Timesteps

0

100

200

300

400

500

Mean Episode Rewards

CartPole-v1

0

500

1000

1500

2000

2500

3000

Timesteps

500

400

300

200

100

0

Mean Episode Rewards

Acrobot-v1

0

500

1000

1500

2000

2500

3000

Timesteps

1000

750

500

250

0

250

Mean Episode Rewards

LunarLander-v2

TRAC PPO

Privileged Weight Reset

B Warmstarting

In our theoretical framework, we hypothesize that a robust parameter initialization, denoted as θ ref , could enhance the performance of our models, suggesting that empirical implementations might benefit from initializing parameters using a base optimizer such as ADAM prior to deploying TRAC. Contrary to this assumption, our experimental results detailed in Section 4 reveal that warmstarting is not essential for TRAC's success. Below, we examine the performance of ADAM PPO and TRAC PPO when warmstarted.

Both TRAC PPO and ADAM PPO were warmstarted using ADAM for the initial 150,000 steps in all games for the Atari and Procgen environments, and for the first 30 steps in the Gym Control experiments. As seen in Figure 9, in games like Starpilot, Fruitbot, and Dodgeball, TRAC PPO surpasses ADAM PPO in the first level/task of the online setup, with its performance closely matching that of ADAM PPO in Chaser. Importantly, TRAC PPO continues to avoid the loss of plasticity encountered by ADAM PPO, even when both are warmstarted. This makes sense since all of the distributions share some foundational game dynamics; the initial learning phases likely explore these dynamics, so leveraging a good parameter initialization to regularize in this early region can be beneficial for TRAC-we observe that forward transfer occurs somewhat in later level distribution shifts as the reward does not drop back to zero where it initially started from.

Our findings indicate that warmstarting does not confer a significant advantage in the Atari games. This makes sense because a parameter initialization that is good in one game setting is likely a random parameterization for another setting, which is equivalent to the setup without warmstarting where TRAC regularizes towards a random parameter initialization. In the Gym Control experiments although warmstarted TRAC PPO manages to avoid the extreme plasticity loss and policy collapse seen in warmstarted ADAM PPO, it does not perform as well as non-warmstarted TRAC PPO. This result underscores that the efficacy of warmstarting is environment-specific and highlights the

challenge in predicting when ADAM PPO may achieve a parameter initialization that is advantageous for TRAC PPO to regularize towards.

From an overall perspective, warmstarting TRAC PPO in every setting still shows substantial improvement over ADAM PPO (Table 2).

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

0

10

20

30

40

50

Mean Episode Rewards

Starpilot

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

0

5

10

15

20

Mean Episode Rewards

Dodgeball

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

4

2

0

2

4

6

Mean Episode Rewards

Fruitbot

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

2

4

6

Mean Episode Rewards

Chaser

TRAC PPO Warmstarted

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

0

500

1000

1500

2000

Mean Episode Rewards

Atari (6)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Timesteps

1e7

4000

2000

0

2000

4000

6000

Mean Episode Rewards

Atari (9)

TRAC PPO Warmstarted

C Other RL Baselines

While PPO is a widely used policy gradient method in reinforcement learning, it is not the only approach applicable to lifelong RL. Other continual RL methods, such as IMPALA (Espeholt et al., 2018), Online EWC (Schwarz et al., 2018), CLEAR (Rolnick et al., 2019), and Modulating Masks (Nath et al., 2023), are designed to address challenges like catastrophic forgetting in dynamic, nonstationary environments. We incorporated these algorithm implementations adapted from the code from Nath et al. (2023) into our experiments to offer a more comprehensive evaluation. These methods vary in their mechanisms for maintaining task performance over time but may still suffer from plasticity loss in later stages of training.

0

500

1000

1500

2000

2500

3000

Timesteps

0

100

200

300

400

Mean Episode Rewards

CartPole-v1

0

500

1000

1500

2000

2500

3000

Timesteps

400

300

200

100

Mean Episode Rewards

Acrobot-v1

0

500

1000

1500

2000

2500

3000

Timesteps

800

600

400

200

0

Mean Episode Rewards

LunarLander-v2

TRAC PPO Warmstarted

Mitigating plasticity loss across policy methods: Figure 12 demonstrates the performance of various continual RL methods when paired with ADAM and TRAC optimizers. The results indicate that when using ADAM, methods like IMPALA, Online EWC, CLEAR, and Modulating Masks exhibit a noticeable decline in performance over time due to plasticity loss, particularly in later levels of the Procgen environments. In contrast, pairing these methods with TRAC instead of ADAM leads to significant improvements, mitigating plasticity loss and enhancing reward performance across subsequent distribution shifts.

To quantify these improvements, Figures 13, 12 present the average normalized rewards over five seeds and 120M timesteps for each method across four different Procgen environments: Starpilot, Dodgeball, Chaser, and Fruitbot. Across all environments, methods that use TRAC outperform their Adam-based counterparts, consistently maintaining higher rewards over time.

On average, across the Procgen environments, TRAC led to performance improvements over ADAM by the following margins: 21.83% for IMPALA, 15.86% for Online EWC, 14.41% for CLEAR, and 10.14% for Modulating Masks.

General Applicability of TRAC: It is important to highlight that TRAC is orthogonal to the learning or policy algorithms themselves. It can be seamlessly integrated into various reinforcement learning architectures by simply replacing their optimizer (e.g., ADAM or RMSPROP). Our results demonstrate that TRAC enhances performance across different algorithms and environments, consistently outperforming ADAM in mitigating plasticity loss.

D Gravity Based Distribution Shifts

One method to introduce distribution changes in reinforcement learning environments is by altering the dynamics Mendez et al. (2020), such as adjusting the gravity in the CartPole environment. In this set of experiments, we manipulate the gravity by a magnitude of ten, randomly adding noise for one distribution shift, and then inversely, dividing by ten and adding random noise for the next shift. This process continues throughout the experiment.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Timesteps

1e8

0

10

20

30

40

50

Average Reward

IMPALA Adam

IMPALA TRAC

CLEAR Adam

CLEAR TRAC

Online EWC Adam

Online EWC TRAC

Mask Adam

Mask TRAC

IMPALA Adam

IMPALA TRAC

CLEAR Adam

CLEAR TRAC

Online EWC Adam

Online EWC TRAC

Mask Adam

Mask TRAC

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Average Normalized Reward

IMPALA Adam

IMPALA TRAC

CLEAR Adam

CLEAR TRAC

Online EWC Adam

Online EWC TRAC

Mask Adam

Mask TRAC

Starpilot with LRL Baselines

CLEAR

IMPALA

Online EWC

Mask

0.0

0.2

0.4

0.6

0.8

1.0

Average Normalized Reward

Dodgeball

CLEAR

IMPALA

Online EWC

Mask

0.0

0.2

0.4

0.6

0.8

1.0

Chaser

CLEAR

IMPALA

Online EWC

Mask

0.0

0.2

0.4

0.6

0.8

1.0

Fruitbot

Clear Adam

Clear TRAC

Impala TRAC

Online ewc Adam

Online ewc TRAC

Mask Adam

Mask TRAC

Other Procgen Games with LRL Baselines

Our observations suggest that ADAM PPO is robust to such dynamics-based distribution shifts, as shown in Figure 14. This indicates that while ADAM PPO implicitly models the dynamics of the environment well-where changes in dynamics minimally impact performance-it struggles more with adapting to out-of-distribution observations such as seen in the main experiments (Figure 6) and in the warmstarting experiments (Figure 11).

E LayerNorm, Plasticity Injection, and Weight Decay

To evaluate TRAC alongside other methods that aim to mitigate plasticity loss, we compare it against LayerNorm (Lyle et al., 2023), Plasticity Injection (Nikishin et al., 2024), and tuning weight decay (Lyle et al., 2024).

As discussed in Section 5, we confirm that both layer normalization and plasticity injection (applied at the start of every distribution shift) (Nikishin et al., 2024; Lyle et al., 2023) are effective in reducing plasticity loss (Figure 15). While these methods help slow the decline in performance due to plasticity loss, TRAC consistently outperforms them across the three Gym Control environments. Importantly, because TRAC is an optimizer, it can be combined with layer normalization, and doing so resulted in the best performance gains in our Control setups.

0

500

1000

1500

2000

2500

3000

Timesteps

0

100

200

300

400

Mean Episode Rewards

CartPole-v1-Gravity

Tuning weight decay: In addition to LayerNorm and Plasticity Injection, we also evaluated the effects of tuning weight decay using PyTorch's AdamW optimizer. We conducted a hyperparameter sweep across three control environments with 15 seeds for each of the following weight decay values: 0.0001, 0.001, 0.01, 0.1, 1.0, 5.0, 10.0, 15.0, and 50.0. Figure 16 presents the average normalized reward for each weight decay value over 15 seeds and 3000 timesteps, compared to TRAC.

The results indicate that while tuning weight decay with Adam does provide some benefit, these values consistently underperform in comparison to TRAC across all three control environments. Figure 16 plots the performance of the best-performing weight decay value with Adam over 10 distribution shifts in the control environments. We observe that weight decay values are highly sensitive to the specific environment and the nature of the distribution shift.

Interestingly, in our initial experiments, we set the weight decay to zero, yet TRAC still outperformed Adam with various weight decay values. This suggests that while weight decay can mitigate plasticity loss to some extent, it does not match the overall effectiveness of TRAC.

0

500

1000

1500

2000

2500

3000

Timestep

0

200

400

Average Reward

CartPole-v1

CartPole-v1

Acrobot-v1

LunarLander-v2

0.00

0.25

0.50

0.75

Average Normalized Reward

All Control Environments

Adam

TRAC

Layer norm Adam

Layer norm TRAC

Plasticity Injection Adam

F Scaling-Value Convergence

As discussed in the algorithm section (see Section 3), TRAC operates as a meta-algorithm on top of a standard optimizer, denoted as BASE. The crucial component of TRAC involves the dynamic adjustment of the scaling parameter S t +1 , managed by the tuner algorithm (Algorithm 2). This parameter is data-dependent and typically ranges between [0 , 1] . The weight update θ t +1 is consequently defined as a convex combination of the current optimizer's weight θ BASE t and a predetermined reference point θ ref .

This section presents the convergence behavior of the scaling parameter S t +1 across different environments, analyzed through the mean values over multiple seeds.

wd_0.0001

wd_0.001

wd_0.01

wd_0.02

wd_0.1

wd_1.0

wd_10.0

wd_14.0

wd_15.25

wd_5.0

wd_50.0

TRAC

0.2

0.4

0.6

0.8

Average Normalized Reward

CartPole-v1

wd_0.0001

wd_0.001

wd_0.01

wd_0.1

wd_1.0

wd_10.0

wd_15.25

wd_5.0

wd_50.0

TRAC

Average Normalized Reward

Acrobot-v1

wd_0.0001

wd_0.001

wd_0.01

wd_0.1

wd_1.0

wd_10.0

wd_15.25

wd_5.0

wd_50.0

TRAC

Average Normalized Reward

LunarLander-v2

Average Normalized Reward Across Control Environments (over 25 seeds)

TRAC

0.0

0.2

0.4

0.6

0.8

1.0

Update Step

1e6

0.00

0.02

0.04

0.06

0.08

0.10

s

Starpilot

0.0

0.2

0.4

0.6

0.8

1.0

Update Step

1e6

0.00

0.02

0.04

0.06

0.08

0.10

0.12

s

Dodgeball

0.0

0.2

0.4

0.6

0.8

1.0

Update Step

1e6

0.00

0.02

0.04

0.06

0.08

0.10

0.12

s

Chaser

0.0

0.2

0.4

0.6

0.8

1.0

Update Step

1e6

0.00

0.02

0.04

0.06

0.08

s

Fruitbot

The convergence of the scaling parameter S t +1 observed across the Procgen and Gym Control environments, as depicted in Figures 17 and 19, reflects a good scaling value that effectively determines the strength of regularization towards the initialization points, yielding robust empirical outcomes in lifelong RL settings. Interestingly, in Procgen environments, this converged scaling value exhibits consistency across various games, typically hovering between 0.02 and 0.03, as shown in Figure 17. In contrast, in the Gym Control environments, the scaling values are lower, ranging between 0.005 and 0.01, as illustrated in Figure 19.

G Comparison to MECHANIC

In our analysis, we extend the examination to other OCO-based optimizers within the lifelong RL setup. Table 3 presents a comparative assessment of TRAC PPO and MECHANIC PPO (Cutkosky et al., 2023) for the lifelong Gym Control tasks (with 300 seed runs). The p-values were calculated using two-sample t-tests to test the hypothesis that the means between TRAC and MECHANIC are the same (Null Hypothesis, H 0 ) against the alternative hypothesis that they are different (Alternative Hypothesis, H 1 ). The results indicate that while MECHANIC effectively mitigates plasticity loss and adapts quickly to new distribution shifts, it slightly underperforms in comparison to TRAC.

0

50000

100000

150000

200000

250000

300000

350000

400000

Update Step

0.00

0.01

0.02

0.03

0.04

0.05

0.06

s

Atari (6)

0

50000

100000

150000

200000

250000

300000

350000

400000

Update Step

0.00

0.02

0.04

0.06

0.08

0.10

0.12

s

Atari (9)

s

0

25000

50000

75000

100000 125000

Update Step

0.00

0.01

0.02

0.03

0.04

CartPole-v1

0

25000

50000

75000

100000 125000

Update Step

0.00

0.02

0.04

s

Acrobot-v1

0

25000

50000

75000

100000 125000

Update Step

0.00

0.02

0.04

0.06

0.08

s

LunarLander-v2

H Experimental Setup

Procgen and Atari Vision backbone For both the Atari and Procgen experiments, the Impala architecture was used as the vision backbone. The Impala model had 3 Impala blocks, each containing a convolutional layer followed by 2 residual blocks. The output of this is flattened and connected to a fully connected layer. The impala model parameters are initialized using Xavier uniform initialization.

Policy and Value Networks Across all experiments-including Gym Control, Atari, and Procgen-the policy and value functions are implemented using a multi-layer perceptron (MLP) architecture. This architecture processes the input features into action probabilities and state value estimates. The MLP comprises several fully connected layers activated by ReLU. The output from the final layer uses a softmax activation.

TRAC TRAC, for all experiments, was implemented using the same experiment-specific baseline architectures and baseline optimizer. For the Procgen and Atari experiments, the base ADAM optimizer was configured as the same as baseline, with a learning rate of 0.001, and for the Gym Control experiments, a learning rate of 0.01 was used. Both learning rates were tested for all experiments and found to have negligible differences in performance outcomes. Other than the learning rate, we use the default ADAM parameters, including weight decay and betas, followed by the specifications outlined in the PyTorch Documentation. 5

The setup for TRAC included β values for adaptive gradient adjustments: 0.9, 0.99, 0.999, 0.9999, 0.99999, and 0.999999. Both S t and ε were initially set to ( 1 × 10 -8 ). Modifications were made to a PyTorch error function library, which accepts complex inputs to accommodate the necessary computations for the imaginary error function. This library can be found at Torch Erf GitHub. 6

Distribution Shifts In the Atari experiments, game environments were switched every 4 million steps. The sequence for games with an action space of 6 included 'BasicMath', 'Qbert', 'SpaceInvaders', 'UpNDown', 'Galaxian', 'Bowling', 'Demonattack', 'NameThisGame', while games with an action space of 9 included 'LostLuggage', 'VideoPinball', 'BeamRider', 'Asterix', 'Enduro', 'CrazyClimber', 'MsPacman', 'Koolaid'.

For Procgen experiments, individual game levels were sampled using a seed value as the start_level parameter, which was incremented sequentially to generate new levels. Each new environment was introduced every 2 million steps.

In the Gym Control experiments, each observation dimension was randomly perturbed by a value ranging from 0 to 2. This perturbation was constant for 200 timesteps, after which a new perturbation was applied, effectively switching the environmental conditions every 200 steps.

Statistical Significance The Procgen and Atari experiments were conducted with 8 seeds/runs, while the Gym Control experiments utilized 25 seeds/runs (with the exception of the Mechanic experiments in Table 3 which utilized 300 seeds). The exception was in the L 2 initialization experiments, which used 15 seeds/runs per regularization strength. In Figures 4, 5, 6, 7, 9, 10, 11, 15, 12, 14, the plotted lines represent the mean of all of the mean episode rewards from the different seeds/runs, and the shaded error bands indicate the standard deviation of all of the mean episode rewards from the different seeds/runs.

Compute Resources For the Procgen and Atari experiments, each was allocated a single A100 GPU, typically running for 3-4 days to complete. The Gym Control experiments were conducted using dual-core CPUs, generally concluding within a few hours. In both scenarios, an allocation of 8GB of RAM was sufficient to meet the computational demands.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As said in our abstract, we offer a parameter-free optimizer that performs well in empirical lifelong RL environments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a limitations paragraph in our discussion section (5).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not provide a novel theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include a comprehensive experimental setup section in our Appendix (H). We also provide a code submission.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a python jupyter notebook in our supplementary material to walk through how to setup and reproduce the results for the computationally less expensive experiments (which have a similar setup to the computationally expensive experiments).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The full experimental details can be found in H and in the code submission in our supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our reward plots include standard deviation error bars around the curves. We detail this in the experimental setup section of our Appendix (H).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer:[Yes]

Justification: Full details of our copmute for each experiment can be found in our H.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the NeurIPS Code of Ethics in every manner.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: Not applicable.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: Not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: We dot use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for avoiding plasticity loss in 'lifelong RL', a setting in which distribution shift occurs during RL training. Here, the setting is split into a number of discrete tasks which change at a fixed interval. The method involves including a penalty on updates, rooted in convex optimization, which encourages movement back to a reference parameterisation. Upon testing, they find that performance sees drastic improvement over two baselines, and does not suffer from the effects of plasticity loss in the same way.\n\nSTRENGTHS:\n- Results are clearly drastically improved with the introduction of PACE, compared to the algorithms which it is benchmarked against *in the main paper*.\n- The work seems well motivated, with the section on Lifelong RL outlining the justification for why this is a useful direction to study.\n- The work was grounded in theory. The justification for design decisions in the context of OCO literature clearly enhanced the method.\n- The paper was clearly written in many places, with clarity lost in only a small number of places (see weaknesses).\n- I have personally not seen previous work which pursues this direction of hyperparameter-free plasticity loss mitigation, particularly in this way. I think this is project has good novelty.\n- The results are demonstrated on a wide array of environments, which I believe to be a sensible distribution for evaluation in this setting.\n- The figures were great for offering intuition about what was happening in the method, and the problem setting.\n- The extra analysis in the appendix was quite enlightening and was appreciated.\n\nWEAKNESSES:\nThere are a few key weaknesses that I think really hold this paper back, which I will detail below. However, I am willing to increase my score based on rebuttals to the following weaknesses and questions.\n- The key thing for me is that it seems Mechanic, in the appendix, performs *very* similarly to PACE (and well within the standard error, making it hard to make any claims about statistical significance - undermining the claim that Mechanic 'slightly underperforms in comparison to PACE'). While I am unfamiliar with this preexisting method, I think it would be good to have included this result into the main paper, with tests across more environments. I appreciate that the discussion mentions that Mechanic has worse guarantees, but empirical demonstration for why PACE is better is pretty necessary given that Mechanic has been included.\n- I also think there are missing comparisons to other methods referenced in the paper, which are designed to mitigate plasticity loss. While I appreciate these may not come under the 'hyperparameter-free' setting, comparison to other methods like plasticity injection would be beneficial for showing why PACE is worth using instead.\n- In places, the method section was quite confusing. Making the algorithms and subsequent text more concise would be useful as it took me quite a few reads to properly parse and understand.\n- The way the results are compared in the table is quite disingenuous, since I believe (particularly in Atari) the results are warped by performance in a very small number of the selected games which had a higher reward scale. This is made clear by the return curves, which show the same story as the table but in an arguably more demonstrative way. It could be worth either normalising scores between each environment, or doing away with the tables entirely.\n- As a small nitpick, it felt like lifelong RL is introduced far before its accompanying reference, making it sound like a new setting created for the paper rather than a term used in this context commonly.\n- The opening paragraph read in quite a different tone than the rest of the paper. It comes across in a slightly unserious manner that I think detracts from the rest of the paper. Having real-world motivation of the problem setting is very good, but this could have been written better (I appreciate this could be quite a personal opinion, and so did not incorporate it into my score).\n\nQUESTIONS:\n- As mentioned earlier, could you please offer a more rigorous comparison with Mechanic (or explain in further detail why PACE is much better, since the difference seems so far very marginal)?\n- How much weight decay was used for Adam? Since this method is very closely aligned to weight decay, and the S value was usually *roughly* a constant scale, could tuning the extent of weight decay provide the same benefit as PACE? As pointed out by [1], weight decay and L2 regularisation are marginally different when using Adam. I appreciate this inroduces an additional tunable hyperparemeter, which is against the spirit of the paper.\n\n\n\n[1] Kumar, Saurabh et al. “Maintaining Plasticity in Continual Learning via Regenerative Regularization.” (2023).\n\nLIMITATIONS:\nThe authors discuss some limitations of their work, such as limited analysis with respect to different algorithms and a brief discussion of Mechanic, another method which performs very similarly to PACE. While I have briefly discussed additional results in regards to limitations, I feel the authors are upfront about a lot of the limitations of their work and that these are not overly limiting to the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMany methods have been proposed recently for continual learning that can maintain their plasticity (e.g., through resetting or regularization towards initialization). There is, however, one critical limitation of those methods, which is their hyper-parameter tuning. Although those methods are designed for lifelong learning that can potentially last forever, their performance is highly dependent on the hyperparameter choices. This paper attempts to address this gap in research by introducing a tuning-free approach inspired by online convex optimization for continual learning by incorporating multiple ideas from online convex optimization research and showing that their method PACE is insensitive to hyperparameter choices and can work in a wide range of problems.\n\nSTRENGTHS:\nThis paper addresses an important limitation of existing continual learning methods, which is their dependence on hyperparameter tuning. The paper presents a novel approach that requires no tuning for hyperparameters and demonstrates the effectiveness of their method on a range of problems. The implications of tuning-free continual learning are immense and can advance future research on continual learning.\n\nWEAKNESSES:\n- Writing needs to be improved. Several parts of the paper seem incomplete, which may require significant restructuring of the paper.\n  - The information on online convex optimization doesn’t fully introduce relevant concepts used in the algorithm. Thus, they either need to be removed, and the paper would rely on general intuition to motivate their method and explain how it’s generally related to OCO or to introduce the method in terms of OCO rigorously. The paper, as it stands, does not do either option.\n  - The algorithm is not fully explained. For example, how the three components work together is not explained, and the paper relies entirely on citing the papers in which they incorporate their ideas in the algorithm. This prevents the paper from being self-contained. Additionally, what does the dot product between the instantaneous gradient $g_t$ and $\\theta_t- \\theta_{\\text{ref}}$ represent? This was never explained, although it’s a crucial part of the algorithm. The following questions should be answered by reading the algorithm and its explanation: 1) the role of the erfi function and 2) why are we keeping track of the first and second moments of the dot product between the current gradient and $\\theta-\\theta_{\\text{ref}}$? 3) how does the additive aggregation work? \n  - Writing should be more focused on what is crucial for following ideas explained in the paper. For example, the paper explained static regret and then mentioned that discounted regret is more suitable only in one sentence without a proper introduction. Similarly, the paper explains the minimax algorithms for static regret but never explains how parameter-free methods are different. I don’t thinking static regret or minmax formulation is helping in the flow of the paper, instead it may hurt it since it distracts the reader from the main message without mentioning the importance of parts for following along.\n- The effectiveness of the three components of the proposed method has not been investigated. Specifically, an ablation study is needed to understand the role of each component and how they interact with each other. For example, the authors might add an ablation study where one component is removed at a time or one component is added at a time.\n\n**Minor issues and suggestions:**\n- The following claim in the conclusion section is unsubstantiated: “PACE’s success leads to a compelling takeaway: empirical lifelong RL scenarios may exhibit more convex properties than previously appreciated and might inherently benefit from parameter-free OCO approaches”. The effectiveness of your method doesn’t mean it works because the problem has convex properties. More work is needed to support such a claim and find direct evidence.\n- I suggest the method may be called tuning-free, not parameter-free, to enhance readability for a wide range of audiences.\n- “From the optimization perspective, the above issues might be attributed to the lack of stability under gradient descent.” -> This requires a citation. Also, what does stability mean in this context?\n- line 78: $T$ is not defined. Also, it's doubly used to mean a different thing in line 120.\n- Notation in equation 1 is not fully defined (e.g., what is $o(T)$?)\n- The word \"conservatism” is used loosely without properly defining what it means.\n- I believe the “Lifelong vs. Continual” paragraph needs to be dropped. The two words mean the same thing. There is no need to use them in different contexts to mean different things. This would lead to unnecessary division in using the word. Further, I don't think Abbas et al. (2023) focused on backward transfer; instead, they focused on the loss of plasticity. \n- the axis in Figure 4 should be the “average episodic return” or \"average cumulative rewards\" not “average episode rewards”.\n\nQUESTIONS:\n- What is a round? (line 121)\n- No definition of discounted regret has been given. How is it defined?\n- “Algorithms there mostly follow  the same principle as in the stationary setting, just wrapped by loss rescaling (Zhang et al., 2024a).\" What does this mean?\n- What does the distribution shift (from 1 to 10) mean in Figure 7 on the x-axis of the first plot? \n- Why is the cumulative sum of rewards used while the authors mentioned that discounted regret is suitable for lifelong RL? \n- The paragraph starting at line 126 is unclear. What does uncertainty set $\\mathcal{U}$ mean or represent?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work. However, I think since PACE exhibits suboptimal performance in the early stages of deployment, it might suggest that it doesn’t also adapt to a fast-changing sequence of tasks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new method designed to mitigate the loss of plasticity when a model is continuously retrained on a set of tasks. The approach uses online convex optimization (OCO) via the minimisation of a regret function. The approach is compared against two baselines: PPO with ADAM (fixed learning rate) and CReLU. The benchmarks used are ProcGen, the Arcade Learning Environment (ALE) and Gym Control. The empirical tests show promising results with respect to standard reward metrics. Overall, the paper engages with a new and emerging field in machine learning that assume changing input distributions, multiple tasks in various contexts, e.g. supervised and reinforcement learning. Moreover, the paper addresses a recently discovered property of optimizers that reduce the ability of the network to continuously adapt to new tasks if continuously retrained (plasticity loss).\n\nSTRENGTHS:\n- The development of a parameter-free optimiser to enhanced performance under non-stationary condition is a interesting research venue in the scope of lifelong reinforcement learning.\n- Addressing the issue of loss of plasticity in dynamic scenarios with shifting distribution is an emerging area in lifelong learning. \n- The paper has a good set of benchmarks that are specific to lifelong reinforcement learning challenges.\n\nWEAKNESSES:\n1. The core of the novel algorithm is explained with pseudo code in Algorithm 1 and 2. I understand that the authors are very familiar with all the details of several papers by Cutkosky (at least 4 main sources), on which this work builds on\n- direction-magnitude decomposition from (Cutkosky & Orabona, 2018), \n- the additive aggregation from(Cutkosky, 2019), \n- and the erfi potential function from (Zhang et al., 2024b). \n-  (Cutkosky et al., 2023) for the recommended default parameter setup.\n\nHowever, it's not obvious to me how clear the methodology is given the explanation in Section 3 to machine learning experts that are not very familiar with Cutkosky's work. \n\n2. A clear highlight of the core technical novelty is missing in relation to previous work. I understand that this work \"builds on\" the cited work above, but it is generally not a sufficient statement of novelty. More effort is required to clearly spell the novel aspects with respect to previous work. \n\n3. The most notable limitation that mostly affects my final evaluation is that this method for lifelong reinforcement learning (LRL) **is compared to only two baselines that are not lifelong reinforcement learning algorithms**. I understand that such choice might derive from the narrow focus of preventing plasticity loss. However, a strong paper in LRL needs to engage with the field more thoroughly, describe notable existing LRL, and how the proposed methods contributes to advancing the field. Candidate baselines could include RL implementations of EWC, Experience Relay based RL methods (e.g. Rolnick's CLEAR), PackNet by Mallya and Lazebink or Modulating Masks for LRL by Ben-Iwhiwhu, just to mentioned a few. I find it odd that in the limitation section, the suggestion is the future application of the algorithm to SAC and DQN, which again are not LRL algorithms.\n\n4. As a follow-up to the previous limitation, the paper would benefit from engaging with the LRL field from the start, not only by describing the Spot example (which sounds like the authors are the first to highlight this to the community), but also describing methods that have been developed to address that problem over the last decades, their achievements, their limitations, and why this particular new approach is needed. \n\n5. Limited analysis is provided on the inner-working of the algorithm (e.g. no ablation studies). \n\n6. Improving the clarity of the hypotheses would benefit the paper. Is the hypothesis the fact that OCO can be used to reduce plasticity loss? Or that it can be used to reduce plasticity loss better than existing algorithms that do that? Or that it can be used to implement LRL? These questions are only partially answered or addressed in the paper.\n\nQUESTIONS:\nMy questions are a direct follow-up on the limitations\n1. Would you consider comparing your approach to lifelong reinforcement learning algorithms?\n2. Why are all the three techniques from the parameter-free OCO required? What do they accomplish individually? Could you run ablation studies? \n3. How does the ADAM default (adaptive learning rate) perform? I understand the need to freeze the learning rate to allow the proposed algorithm perform the adaptation, but it would be nevertheless valuable to assess the performance of the standard version of ADAM.\n\nLIMITATIONS:\nThe limitation section is mainly focused on a narrow aspects, e.g. the need for a warm-up. However, this is also in contradiction with the Appendix in which it is suggested that such a warm-up is not required. \n\nMy conclusion is that the narrow scope of this limitations section stems from the lack of integration of this work with the broader field of lifelong reinforcement learning. Addressing the weaknesses that I pointed out, and engaging with a more thorough comparison of this approach with existing LRL approaches, a more insightful view of the pro and cons should emerge.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of plasticity loss in RL, where a neural network may have reduced performance on learning new tasks after a task switch. The authors develop a hyperparameter-free algorithm, PACE, using ideas from online convex optimization. It can be interpreted as an automatically-tuned weight decay adapted during training. Experiments on large-scale benchmark tasks demonstrate the effectiveness of the approach compared to different baselines.\n\nSTRENGTHS:\nThis paper builds a bridge between the fields of lifelong RL and online convex optimization (OCO), successfully using ideas from OCO to develop an algorithm addressing plasticity loss. It can be difficult to use fruitfully utilize theoretical ideas in a practical context so I commend the authors for that.\n\nThe paper is written clearly and generally easy to follow. Intuitions for the theory are provided which are helpful to understand the algorithm.\nThe evaluation is extensive and covers a wide variety of RL tasks, with larger-scale benchmarks such as ProcGen.\nI also appreciate some of the more in-depth analyses such as inspecting the scaling $S_t$ over time in appendix C.\n\nWEAKNESSES:\nGenerally, I have not identified any major weaknesses for this paper.\nSome areas that could be improved (but are still acceptable):\n- Only a couple of baselines are run, including weight decay and using CReLU activation. \n\tA suggestion would be to try layer norm since it is also parameter-free and has shown some benefits to address plasticity loss [1].\n- While the algorithm is theoretically-motivated, I did not see any theoretical results included in the paper. If possible, it could be nice to have a theoretical guarantee for PACE even if it is in a different setting such as OCO. Such a result could give additional confidence that the algorithm is sound.\n\nQUESTIONS:\n_Clarification questions_\n- Fig. 2: I do not fully understand what this figure is trying to convey. In particular, how is the figure demonstrating that PACE may be beneficial in this scenario? Maybe an explanation in the caption could help.\n\n- For the section on tuning $\\ell_2$ regularization (line 280), could you include a hyperparameter sensitvity plot i.e. performance vs. hyperparameter value? I would be curious to see how the performance varies around the best values.\n\n- Fig. 6 In the experiments, it seems like even PACE suffers from some amount of plasticity loss since the learning curve does not reach the same maximum as in the first task (more evident in CartPole or LunarLander). Do you have any hypotheses to explain this?\n\n_Comments and suggestions_ \n\n\n- Looking at the values of $S_t$ over time (in appendix C) they are usually around 0.01-0.1, which seems fairly small and have a distinct trend of decreasing over time.\nThis looks inversely correlated to the parameter norm, a quantity that is often measured in the context of plasticity loss. Have you looked into this relationship? I wonder if plotting $S_t \\times ||\\theta^{Base}||$ shows a roughly constant effective parameter size. \n\n- If I understand PACE correctly, it looks like overall, it produces the next parameter $\\theta_t$ through a weight $S_t$ which is used to obtain convex combination between the initial parameters and the base-optimized parameters $\\theta^{Base}_t $. \nWhat I find counterintuitive is that the gradients for the next step are then evaluated at $\\theta_t$ but used to update $\\theta^{Base}_t$, which could be quite different in terms of the policy implied. \nDo you have any thoughts about this?\n\n- A potential explanation, as discussed in [3], is that rescaling may also preserve the argmax for classification tasks (and the policy in this case) although it does affect the size of the logits. This would not explain effects on the value network though. \n\n- On a related note, do the neural networks feature normalization layers e.g. layer norm? \nWhen normalization layers are used, there is a scale-invariance introduced in the weights (well, if there are no bias units, see page 1. of [2]). I could see PACE being more effective in conjunction with normalization layers due to this rescaling operation not impacting (or having a small impact on) the function learned.  \n\n- About \"near convexity\", line 298. While the algorithm components are developed for the OCO setting, do you think it is possible that, instead of the problem being \"near convex\", it is the algorithms that have wider applicability than previous appreciated? That is, they may work well in a more general class of problems?\n\n[1] \"Understanding plasticity in neural networks\" Lyle et al. \n\n[2] \"Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction\" Lyu et al. \n\n[3] \"On Warm-Starting Neural Network Training\" Ash and Adams\n\nLIMITATIONS:\nThese are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}