Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning
Authors: Aneesh Muppidi
Zhiyu Zhang
Heng Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: lifelong reinforcement learning
parameter-free optimization
continual reinforcement learning
loss of plasticity
Primary area: reinforcement_learning
TL;DR: We propose a parameter-free optimizer for lifelong reinforcement learning that mitigates loss of plasticity and rapidly adapts to new distribution shifts.

Abstract: A key challenge in lifelong reinforcement learning (RL) is the loss of plasticity, where previous learning progress hinders an agent's adaptation to new tasks. While regularization and resetting can help, they require precise hyperparameter selection at the outset and environment-dependent adjustments. Building on the principled theory of online convex optimization, we present a parameter-free optimizer for lifelong RL, called TRAC, which requires no tuning or prior knowledge about the distribution shifts. Extensive experiments on Procgen, Atari, and Gym Control environments show that TRAC works surprisingly well—mitigating loss of plasticity and rapidly adapting to challenging distribution shifts—despite the underlying optimization problem being nonconvex and nonstationary.

Fast TRAC:
A Parameter-Free Optimizer for Lifelong Reinforcement Learning
Aneesh Muppidi
Zhiyu Zhang Harvard University
Harvard College
anoeshmupi.ci@collegs harvard.edi
zhiyuzOseas harvard. edu
Heng Yang
Harvard University
hankyang@seas harvard.edi
Abstract
A key challenge in lifelong reinforcement learning (RL) is the loss of plasticity, where previous learning progress hinders an agent's adaptation to new tasks. While regularization and resetting can help, they require precise hyperparameter selection at the outset and anvironment-deeendeee theory of online convex optimization, we present a parameter-fie frre optimizer for lifelong RL, called TRAC, which requires no tuning or prior knowledge about the distribution shifts. Extensive experiments Procgen, Atari, and Gym Control environments show that TRAC works surprisingly well-mitigating loss of plastic- ity and rapidly adapting to challenging distribution desttte the underlying optimization problem being nonconvex and nonstationary. Project website and code is available here.
adjustments. Building on the principled
Introduction
Spot, the agile robot dog, has been learning to walk confidently across soft, lush grass. But when Spot moves from the grassy field
Starpilot
gravel surface, the small stones shift beneath her feet, causing her to stumble. When Spot tries walk across a sandy beach or on ice, the challenges multiply, and her once-steady walk becomes erratic. Spot wants to
Meen Eptoode
adjust quickly to these new terrains, but the patterns she learned on grass are not suited to gravel, sand, or ice. Furthermore, she never knows when the
terrain will change again and how different will
be, therefore must continually plan for the unknown while avoiding reliance on outdated experiences.
Figure 1: Severe loss of plasticity in Procgen (Starpilot). There is steady decline in reward with each distribution shift.
Spot's 's struggle exemplifies well-known and ex- tensively studied challenge in real-world decision making: lifelong reinforcement learning (lifelong RL) Abel al. (2024): Nath et al. (2023); Mendez et al. (2020); Xie & Finn (2022). In lifelong RL, the learning agent must continually acquire new knowledge to adapt to the nonstationarity the environment. At first glance, there appears to be
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

an obvious solution: given policy gradient oracle, the agent could just keep running gradient descent nonstop. However, recent experiments have demonstrated an intriguing behavior called loss of plasticity (Dohare et al., 2021; Lyle et al., 2022; Abbas et al., 2023; Sokar et al., 2023): despite persistent gradient steps, such an agent can gradually lose its responsiveness to incoming observations. There are even extreme cases of loss of plasticity (known as negative transfer or primacy bias), where prior learning can significantly hamper the performance in new tasks (Nikishin et al., 2022: Ahn et al., 2024); see Figure 1 for example. All these suggest that the problem is more involved than one might think.
From the optimization perspective, the above issues might be attributed to the lack of stability under gradient descent. That is, the weights of the agent's parameterized policy can drift far away from the origin (or a good initialization). leading to a variety of undesirable behaviors.' Fitting this narrative, it has been shown that simply adding a T22 regularizer to the optimization objective (Kumar et al., 2023) or periodically resetting the weights (Dohare et al., 2021; Asadi et al., 2024; Sokar et al., 2023; Ahn et al., 2024) can help mitigate the problem. However, a particularly important limitation is their use of hyperparameters, such the magnitude of the regularizer and the resetting frequency². Good performance hinges on the suitable euviromment-dependemn hyperparameter, but how can one confidently choose that before interacting with the environment? The classical cross-validation approach would violate the one-shot nature of lifelong RL (and online learning in general: see Chapter of Orabona, 2023), since it is impossible to experience the same environment multiple times. This leads to the contributions of the present work.
Contribution
The present work addresses the key challenges in lifelong RL using the principled theory of Online Convex Optimization (OCO). Specifically, our contributions are two fold.
Algorithm: TRAC Building on series of results in OCO (Cutkosky & Orabona, 2018; Cutkosky.
2019; Cutkosky et al., 2023; Zhang et al., 2024b), we propose a (lypeer)purrumeler-fre optimizer for lifelong RI., called TRAC (AdapTTive RegularizAtion in Continual environments). Intuitively, the idea is a refinement of regularization: instead of manually selecting the magnitude of regularization beforehand, TR chooses that in an online, data-dependent manner. From the perspective of OCO theory. TRAC is insensitive to its own hyperparameter, which means that no hyperparameter tuning is necessary in practice. Furthermore, an optimization approach to lifelong RL, TRAC is compatible with any policy parameterization method.
Experiment Using Proximal Policy Optimization (PPO) (Schulman et al., 2017), we conduct comprehensive experiments on the instantiation of TRAC called TRAC PPO. A diverse range of lifelong RL environments are tested (based on Procgen, Atari, and Gym Control), with considerably larger scale than prior works. In settings where existing approaches (Abbas et al., 2023; Kumar et al., 2023; Nath et al., 2023) struggle, we find that TRAC PPO
mitigates mild and extreme loss of plasticity;
and rapidly adapts to new tasks when distribution shifts are introduced.
Such findings might be surprising: the theoretical advantage of TRAC is motivated by the convexity in OCO, but lifelong RL is both nonconvex and nonstationary in terms of optimization.
Organization Section 2 surveys the basics of lifelong RL. Section 3 introduces our parameter-fiee freee algorithm TRAC, and experiments are presented in Section 4. We defer the discussion of related works and results to Section 5. Finally, Section 6 concludes the paper.
2 Lifelong RL
As a sequential decision making framework, reinforcement learning (RL) is commonly framed as Markov Decision Process (MDP) defined by the state space S, the action space A, the transition dynamics P($t41 (8t, a,), and the reward function H(st, Gr 9t+1). In the t-th round, starting from a state s, E S, the learning agent needs to choose an action ag E A without knowing P and R. Then, the environment samples a new state St+1 P(-|st, az), and the agent receives reward
SSuch the inactivation of many neurons, due the ReLU activation function (Abbas et al., 2023; Sokar al., 2023).
'Indeed, hyperparameter selection, in general, is a well-known problem in lifelong as well as continual learning settings (De Lange et al., 2021).
2

---

= R(st, de, S141). There are standard MDP objectives driven by theoretical tractability, but from a practical perspective, measure the agent's performance by its cumulative reward Σ Pt-
The standard setting above concerns a stationary MDP. Motivated by the prevalence of distribution shifts in practice, the present work studies nonstationary variant called lifelong RL, where the transition dynamics P: and the reward function R2 can vary over time. Certainly, one should not expect any meaningful "learning" against arbitrary unstructured nonstationarity. Therefore, we implicitly assume Pt and R3 to be piecewise constant over time, and each piece is called a task just like our example of Spot in the introduction. The main challenge here is to transfer previous learning progress to new tasks. This is reasonable when tasks are similar, but we also want to reduce the degradation when tasks turn out to be very different.
Lifelong RL as online optimization Deep RL approaches, including PPO (Schulman et al., 2017) and others, crucially utilize the idea policy parameterization. Specifically, a policy refers to the distribution of the agent's 's action a, (conditioned on the historical observations), and we use 0, E R to denote the parameterizing weight vector. After sampling a, and receiving new observations, the agent could define a loss function Jz (f) that characterizes the "hypothetical performance" of each weight 0 E Rd. Then, by computing the policy gradient 9: (.,)),), one could apply a afirst order optimization algorithm" OPT to obtain the updated weight, 0 (O,,(0
For the rest of this paper. we will work with such an abstraction. The feedback of the environment is treated policy gradient oracle G, which maps the time and the current weight 0g into policy gradient 9: = [it, (t). Our goal is to design an optimizer OPT well suited for lifelong RL.
Lifelong vs. Continual In the RL literature, the use of "lifelong" and "continual" varies significantly across studies, which may lead to confusion. Abel et al. (2024) characterized continual reinforcement learning (CRL) as a never-ending learning process. However, much of the literature cited under CRL, such as (Abbas et al., 2023; Ahn et al., 2024), primarily focuses on the problem of backward transfer (avoiding catastrophic forgetting). Various policy-based architectures, such as those proposed by Rolnick et al. (2019); Schwarz et al. (201 8); Nath et al. (2023), focus on tackling this issue. Conversely. the present work addresses the problem of forward transfer, which refers to the rapid adaptation to new tasks. Because of this we use "lifelong" rather than "continual" in our exposition, similar to (Thrun, 1996; Abel et al., 2018;; Julian et al., 2020).
3 Method
Inspired by (Cutkosky al., 2023), we study lifelong RL by exploiting its connection to Online Conver Optimization (OCO; Zinkevich, 2003). The latter is a classical theoretical problem in online learn- ing, and much effort has been devoted to design- ing parameter-free algorithms that require minimum tuning or prior knowledge (Streeter & Mcmahan, 2012; McMahan & Orabona, 2014; Orabona & Pál, 2016; Foster et al., 2017: Cutkosky & Orabona, 2018; Mhammedi & Koolen, 2020; Chen et al., 2021; Jacob- sen& & Cutkosky, 2022). The surprising observation of Cutkosky al. (2023) is that several algorithmic ideas closely tied to the convexity of OCO can actu- ally improve the nonconvex deep learning training, suggesting certain notions of "near convexity" on its loss landscape. We find that lifelong RL (which is both nonconvex and nonstationary in terms of optimization) exhibits a similar behavior, therefore a particularly strong algorithm (named TRAC) can be obtained from principled results in parameter-free OCO. Let us start from the background.
Previous Task
New Task
Base
strength reg regutartzation
Upddate
Updatte
Weight Porameterizaatio
Figure 2: Visualization of TRAC's key idea.
Basics of (parameter-free) OCO
As standalone theoretical topic, OCO concerns sequential optimization problem where the convex loss function 4, can vary arbitrarily over time. In the t-th
Formally, dynamical system that given its state 0: and input ge outputs the new state (P,(O., gt).
3

---

iteration, the optimization algorithm picks an iterate and then observes a gradient Ir - (z)). Motivated by the pursuit of "convergence" in optimization, the standard objective to guarantee low (i.e., sublinear in T) static regret, defined as
Regret u) Σ (m)) Σ C(w),,
where T' is the total number of rounds, and w is a comparator that the algorithm does not know beforehand. In other words, the goal is to make (egret-p((1:r u) small for all possible loss sequence hir and comparator u. Note that for nonstationary OCO problems analogous lifelong RI., it is better to consider different objective called the discounted regret. Algorithms there mostly follow the same principle as the stationary setting, just wrapped by loss rescaling (Zhang et al., 2024a).
For minimizing static regret, classical minimax algorithms like gradient descent (Zinkevich, 2003) would assume a small uncertainty set U the beginning. Then, by setting the hyperparameter (such as the learning rate) according to U, it is possible to guarantee sublinear worst case regret,
max (1.2,,))
Regrety u) = o(T).
In contrast, parameter-free algorithms very different strategies* to bound Regrety (41.7, u) directly (without taking the maximum) by a function of both 44:7 and u. The resulting bound is more refined than (1) (Orabona, 2023, Chapter 9), and crucially, since there is no need to pick uncertainty set U, much less hyperparameter tuning is needed. This is where its name comes from.
TRAC for Lifelong RL: In lifelong RL, key issue is the excessive drifting of weights 6,, which can detrimentally affect adapting to new tasks. To address this, TRAC enforces proximity to a well-chosen reference point Oref. providing principled solution derived from a decade of research in parameter-free OCO. Unlike traditional methods such as L2 regularization or resetting, TRAC avoids hyperparameter tuning, utilizing the properties of OCO to maintain weight stability and manage the drift effectively.
The core of TRAC, similar to other parameter-free optimizers, incorporates three techniques:
Anrection-Magnitudee Decomposition: Inspired by Cutkosky & Orabona (2018), this technique employs a carefully designed onc-dimensional algorithm, the "parameter- free tuner," atop base optimizer. This setup acts as data-dependent regularizer, controlling the extent to which the iterates deviate from their initialization, thereby minimizing loss of plasticity, which is crucial given the high plasticity at the initial policy parameterization (Abbas et al., 2023).
Erfi Potential Function: Building on the previous concept, the tuner utilizes the Erfi poten- tial function, as developed by Zhang et al. (2024a). This function is crafted to effectively balance the distance of the iterates from both the origin and the empirical optimum. manages the update magnitude by focusing on the gradient projection along the direction 0, Oref-
Additive Aggregation: The tuner above necessitates discounting. Thus, we employ Additive Aggregation by Cutkosky (2019). This approach enables the combination of multiple parameter-free free OCO algorithms, each with different discount factors, to approximate the performance of the pest-performing algorithm. Importantly, facilitates the automatic selection of the optimal discount factor during training.
These three components crucially work together to guarantee good regret bounds in the convex setting and are the minimum requirement for any reasonable parameter-free optimizer.
Without going deep into the theory, here is overview of the important ideas (also Figure 2 for visualization).
First, TRAC is mela-algorithm that operates on top of a "default" optimizer BASE. It can simply be gradient descent with a constant learning rate, or ADAM (Kingma & Ba, 2014) as in our experiments. Applying BASE alone would be equivalent to enforcing the scaling parameter S 1 in TRAC, but this would suffer from the drifting of BBae (and thus, the weight 6.41).
'The key difference with gradient descent is the use of intricate L2) regularizers. See (Fang et al., 2022; Jacobsen & Cutkosky, 2022) for theoretical justification of their importance.

---

Algorithm TRAC: Parameter-free Adaption for Continual Environments.
1: Input: A policy gradient oracle 9; first order optimization algorithm BASE; reference point
Ore E R n discount factors $1...., Sn E (0. 1] (default: 0.9, 0.99, 0.999999).
2: Initialize: Create n copies of Algorithm 2, denoted A1,... ........ An. For each [1 n), Aj
uses the discount factor 3,. Initialize the algorithm BASE at fer. Let 8 = 8
3: for [. 1, 2, ... do
4: Obtain the /-th policy gradient - (l, 0z) E Rd
5: Send gr to BASE as its t-th input, and get its output #+1 E R
6: For all j E [1 n]] send (gt, 0. Oret) to Aj its t-th input, and get its output 81+1,j E R.
7: Define the scaling parameter S = 2== 8141..
Σ}=
8:
Update the weight of the policy,
B2+1 = Pre + (0 Ozet) S1+1-
9: end for
Algorithm 2 ID Discounted Tuner of TRAC.
1: Input: Discount factor / E (0, 1]; small value E > 0 (default: 10-5).
2: Initialize: The running variance to O; the running (negative) sum do 0.
3: for = 1, 2, do
4: Obtain the t-th input hy-
5: Let = -2 hig and = 30r-1 le-
6: Select the 1-th output
St+1
erfi 22 + +
erfi(1/
where erfi is the imaginary ervor function queried from standard software packages. 7: end for
To fix this issue, TRAC uses the tuner (Algorithm 2) to select the scaling parameter S making it data-dependent. Typically S is within [0, 1] (see Figure 17 to 19), therefore essentially, we define the updated weight 0.41 convex combination of the BASE'S weight 0 and the reference point Ozefs
0.+1 = S BBae + (1 A11-Sppp (11-S41100ef-
This brings the weight closer Oref, which is known to be "safe" (i.e., overfitting any particular lifelong RI. task), although possibly conservative.
To inject the right amount of conservatism without hyperparameter tuning, the tuner (Algorithm 2) applies an unusual decision rule based on the erfi function. Theoretically, this is known to be optimal in an idealized variant of OCO (Zhang et al., 2022, 2024b), but removing the idcalized assumptions requires tiny bit of extra conservatism. which is challenging (and not necessarily practical). Focusing on the lifelong RL problem that considerably deviates from OCO, we simply apply the erfi decision rule as This loosely motivated by deep learning training dynamies, e.g.. (Cohen et al., 2020; Ahn et al., 2023; Andriushchenko et al., 2023), where an aggressive optimizer is often observed to be better.
Finally, the tuner requires a discount factor 3. This erucially controls the strength of regularization (claborated next), but also introduces hyperparameter tuning problem. Following (Cutkosky, 2019), we aggregate tuners with different 3 (on a log-scaled grid) by simply summing up their outputs. This is justified by the adaptivity of the tuner itself:: in OCO. if we add parameter-fiee algorithm A₁ to any other algorithm Ag that already works well, then A₁ can automatically identify this and "tune down" its aggressiveness, such that At + Ag still performs as well as Ag.
Connection to regularization Despite its nested structure, TRAC can actually be seen as parameter-free refinement of La regularization (Kumar et al., 2023). concretely explain this intuition, let consider the following two optimization dynamics.
5

---

First, suppose we run gradient descent with learning rate 7,, on the policy gradient sequence {9r with the L2 regularizer 2/2 ||0 relll Quantitatively, it means that starting from the t-th weight Oz.
O2+1 - 0; - 19: A (0: 0ror)] = O2+1 - O = (1-An) (0, - Oref) - n9tt- (2)
That is, the updated weight 0z+1 is determined by (1 - An)-discounting with respect to the reference point Oret, followed by gradient -rjtt
Alternatively, consider applying the following simplification of TRAC the same policy gradient sequence {gr (i) BASE still gradient descent with learning rate vi (ii) there is just one discount factor 3: and (tii) the one-dimensional tuner (Algorithm 2) is replaced by the 3-discounted gradient descent with learning rate i.c., St+1 = 3S, alt. In this case, we have
Btt1 - Ozef = S1+1 (0 Bret)
(3S, ahh)) (0 Oref 7199))
- a35 h4) (0r Orot) 11St-19t-
(mildly assuming St # 0)
Notice that S₁ is 3-discounted sum of odt, a thus in the typical situation of 3 x =11 one might expect ah, < I.S.. Then, the resulting update of 0.+1 is similar to Eq.(2), with quantitative changes on the "effective discounting"
xny - B, and the "effective learning rate" 7) - 1/5++1-
The main message here is that under simplified setting, TRAC is almost equivalent to L2 regulariza- tion. The latter requires choosing the hyperparameters À and 1,, and similarly, the above simplified TRAC requires choosing 3 and 7). Going beyond this simplification, the actual TRAC removes the tuning of 3 using aggregation, and the tuning of r using the erfi decision rule.
On the hyperparameters Although TRAC is called "parameter-free", it still needs the B-grid, the constant and the algorithm BASE as inputs. The idea is that TRAC is particularly insensitive such choices, supported by the OCO theory. As the result, the generic default values recommended by Cutkosky al. (2023) are sufficient in practice. We note that those are proposed for training supervised deep learning models, thus should be agnostic to the lifelong RL applications consider.
4 Experiment
Does TRAC experience the common pitfalls of loss of plasticity? Does rapidly adapt to distribution shifts? To answer these questions, test TRAC in empirical RL benchmarks such vision- based games and physics-based control environments in lifelong settings (Figure 3). Specifically, we instan- tiate PPO with two different optimizers: ADAM with constant learning rate for baseline comparison, and TR AC for our proposed method (with exactly the same ADAM as the input BASE). We also test ADAM PPO with concatenated ReLU activations (CReLU; Shang et al., 2016), previously shown to mitigate loss of plasticity in certain deep RL settings (Abbas et al., 2023). Our numerical results are summarized in Table 1, Across every lifelong RL setting, we observe substantial improvements in the cumulative episode reward by using TRAC PPO compared to ADAM PPO or CReLU, Below are the details, with more in the Appendix.
TRAC
TRAC
TRAC
TRAC
TRAC
TRAC
Bane
Base
Daso
Dasa
Dase
Base
ASA
Obe [noise]
Obe += [noise]
Obs [nois]]
Procgen
Atari
Control
Figure 3: Experimental setup for lifelong RL.
Procgen We first evaluate OpenAl Procgen, suite of 16 procedurally generated game envi- ronments (Cobbe al., 2020). We introduce distribution shifts by sampling a new procedurally generated level of the current game every 2 million time steps, treating each level as a distinct task.

---

Adam PPO
TRAC PPO
CReLU
Starpilot
Dodgeball
Mean Episode Rewards
MeanEpleode Rewards
26
20
20
20
10
Timesteps Fruitbot
167
Timesteps
Chaser
Mean Episode Rewards
75
Mcan Epleade Rewards
5.0
6
26
-5.0
Timesteps
167
Timesteps
Figure 4: Reward the lifelong Procgen environments for StarPilot, Dodgeball, Fruitbot, and Chaser. There is steady loss of plasticity in agents using ADAM PPO and CReLU, characterized by their inability maintain performance through succesive Procgen levels. contrast, TRAC avoids this loss of plasticity, quickly achieving high performance with each new task.
We evaluate game environments including StarPilot, Dodgeball, Fruitbot, and Chaser. In all of these environments, observe in Figure that both ADAM PPO and CReLU encounter continually degrading loss of plasticity as these distribution shifts are introduced. In contrast, TRAC PPO avoids this loss of plasticity, which contributes to its rapid reward increase when adapting to new levels. In the cumulative reward across all the Procgen levels, TRAC PPO reveals normalized average improvements of 3,212.42% and 120.88% over ADAM PPO and CReLU respectively (see Table 1). For later levels, in all games, TRAC PPO's reward does not decline as sharply as the baselines, potentially indicating positive transfer of skills from one level to the next.
One key advantage of TRAC that functions as an optimizer, making it orthogonal various policy methods such as PPO, well other basclines like Online EWC (Schwarz et al., 2018), IMPALA (Espeholt et al., 2018), Modulating Masks (Nath et al., 2023), and CLEAR (Rolnick et al., 2019). In Appendix C, evaluate these methods using both TRAC and ADAM the Procgen setup. We find that in every environment, TRAC improves the performance of these algorithms.
Atari The Arcade Learning Environment (ALE) Atari 2600 benchmark is a collection of classic arcade games designed assess reinforcement learning agents' performance across a range diverse gaming scenarios (Bellemare et al., 2013). We introduce distribution shifts by switching to new Atari game every million timesteps, where each game switch introduces a new task. This benchmark is more challenging compared to OpenAl Procgen: it requires the agent to handle distribution shifts in both the input (state) and the target (reward).
In this experiment, we assessed two online settings distinguished by games with action spaces of and 9. From Figure 5, both ADAM PPO and CReLU sometimes failed learn in certain games. In contrast, TRAC PPO shows a substantial increase in reward over different games compared to the baselines. For example, during the first 12 million steps games) in Atari 6. TRAC PPO not only achieves a significantly higher mean reward but also demonstrates rapid reward increase. Over both experiment settings, TRAC PPO shows an average normalized improvement of 329.73% over ADAM PPO and 68. 71% over CReLU (Table 1). In rare instances, such as the last 2 million steps of Atari 6, CReL.U performs comparably TRAC PPO. This observation aligns with findings from (Abbas et al., 2023), which noted that CReLU tends avoid plasticity loss in continual Atari setups.

---

Adam PPO
TRAC PPO
CReLU
Atari (6)
Atari (9)
Moan Episode Rowards
Episode Rewards
2000
10000
1800
8000
1.5 Timesteps
Timesteps
Figure 5: Reward in the lifelong Atari environments, across games with action spaces of 6 and 9. TRAC PPO rapidly adapts to new tasks, in contrast to the ADAM PPO and CReI.U which struggle to achieve high reward, indicating mild loss of plasticity.
Adam
TRAC PPO
CReLU
Mean Episode Rewards
500
CartPole-v1
Mean Eplsode Rewards
Acrobot-v1
Episode Rewards
250
LunarLander-v2
300
200
-250
200-
-300
-500
100
0.
400
2000
Timesteps
Timesteps
Timesteps
Figure 6: Reward performance across CartPole. Acrobot, and LunarLander Gym Control tasks. Both ADAM PPO and CReLU experience extreme plasticity loss, failing to recover after the initial distribution shift. Conversely, TRAC PPO successfully avoids such plasticity loss, rapidly adapting when facing extreme distribution shifts.
Gym Control We use the CartPole-vl and Acrobot-v environments from the Gym Classic Control suite, along with LunarLander-v2 from Box2d Control. To introduce distribution shifts, Mendez et al (2020) periodically alters the environment dynamics. Although such distribution shifts pose only mild challenges for robust methods like PPO with ADAM (Appendix D). We instead implement more challenging form of distribution shift. Every 200 steps we perturb each observation dimension with random noise within range of 22 treating each perturbation phase as distinct task.
Table I: Cumulative sum of mean episode reward for TRAC PPO, ADAM PPO, and CReLU on Procgen, Atari, and Gym Control environments. Rewards are scaled by 10a; higher is better.
Environment
ADAM PPO
CReLU
TRAC PPO (Ours)
Starpilot
3.4
3.6
12.5
Dodgeball
1.9
2.3
5.2
Chaser
1.4
1.7
2.2
Fruitbot
0.1
1.0
1.8
CartPole
5.1
1.2
39.6
Acrobot
14.3
-13.9
-12.9
LunarL.ander
221.7
-19.4
-8.6
Atari 6
3.1
4.8
10.5
Atari
3.9
17.0
20.2
Here (Figure 6), we notice a peculiar behavior after introducing the first distribution shift in both ADAM PPO and CReLU: policy collapse. We describe this as an extreme form of loss of plasticity. Surprisingly, TRAC PPO remains resistant to these extreme distribution shifts. As we see in Acrobot experiment, TRAC PPO shows minimal to policy damage after the first few distribution shifts, whereas ADAM PPO and CReLU are unable recover a policy at all. We investigate if TRAC's behavior here indicates positive transfer in Appendix A. Across the three Gym Control environments,

---

TRAC PPO shows an average normalized improvement of 10%. over ADAM PPO and 1044.24%
over CReLU (Table 1).
5
Discussion
Related work Combating loss of plasticity has been studied extensively in lifelong RL. A typical challenge for existing solutions is the tuning of their hyperparameters, which requires prior snowledge on the nature of the distribution shift, e.g., (Asadi et al., 2024; Nath al., 2023; Nikishin et al., 2024; Sokar et al., 2023; Mesbahi et al., 2024). An architectural modification called CReLU is studied in (Abbas al., 2023), but our experiments suggest that its benefit might be specific the Atari setup. Besides, Abel et al. (2018a,b) presented a theoretical analysis of skill transfer in lifelong RL, based on value iteration. Moreover, related contributions in nonstationary RL, where reward and state transition functions also change unpredictably. are limited to theoretical sequential decision-mate making settings with focus on establishing complexity bounds (Roy et al., 2019; Cheung et al., 2020; Wei & Luo, 2021; Mao et al., 2020).
Our algorithm TRAC builds on a long line of works on parameter-free OCO (see Section 3). To our knowledge, the only existing work applying parameter-free OCO to RL is (Jacobsen & Chan, 2021), which focuses on estimating the value function (i.e., policy evaluation). Our scope is different, focusing on empirical RL in lifelong problems by exploring the key connection between parameter- free OCO and regularization.
Particularly, we are inspired by the MECHANIC algorithm from (Cutkosky et al., 2023), which goes beyond the traditional convex setting of parameter-free OCO to handle stationary deep learning optimization tasks. Lifelong reinforcement learning, however, introduces a layer of complexity with its inherent nonstationarity. Furthermore, compared to MECHANIC, TRAC improves the scale tuner there (which is based on the coin-belling framework; Orabona & Pál, 2016) by the erfi algorithm that enjoys a better OCO performance guarantee. As an ablation study, we empirically compare TRAC and MECHANIC in the Appendix G (Table 3). We find that TRAC is slightly better, but both algorithms can mitigate the loss of plasticity, suggesting the effectiveness of the general "parameter-fice principle in lifelong RL.
TRAC encourages positive transfer In our experiments, observe that TRAC'S reward decline due to distribution shifts is less severe than that of baseline methods. These results may suggest TRAC facilitates positive transfer between related tasks. To investigate this further, compared TRAC to a privileged weight-resetting approach, where the network's parameters are reset for each new task, in the Gym Control environments (see Appendix A). Our results show that TRAC maintains higher rewards during tasks than privileged weight-resetting and avoids declining to the same low reward levels privileged weight-resetting the start of new task (Figure 8).
On the choice of Oref In general, the reference point jet should be good or "safc" for TRAC to perform effectively. One might presume that achieving this requires "warmstarting". or pre-training using the underlying BASE optimizer. While our experiments validate that such warmstarting is indeed beneficial (Appendix B), main experiments show that even random initialization of the policy's 's weight serves as a good enough fget. even when tasks are similar (Figure 4).
This observation aligns with discussions by Lyle et al. (2023), Sokar et al. (2023), and Abbas et al. (2023), who suggested that persistent gradient steps away from a random initialization can deactivate ReI.U activations, leading to activation collapse and loss of plasticity in neural networks. Our results also support Kumar et al. (2023)'s 's argument that maintaining some weights close to their initial values not only prevents dead ReLU units but also allows quick adaptation to new distribution shifts.
Tuning La regularization The success of TRAC suggests that an adaptive form of regularization- anchoring to the reference point Oref Oref-may suffice to counteract both mild and extreme forms of loss of plasticity. From this angle, we further elaborate the limitation of the L2 regularization approach considered in (Kumar et al., 2023). It requires selecting a regularization strength parameter À through cross-validation, which incompatible with the one-shot nature of lifelong learning settings. Furthermore, it is nontrivial to select the search grid: for example, we tried the X-grid suggested by (Kumar et al., 2023), and there no effective À value within the grid for the lifelong RI. environments we consider. All the values are too small.

---

Acrobot
CariPole
Lunart.andrr
Wall
TRAC
Mean Episode Rewards
Mean Epleoda Rewards
Miaa Epieods Rewards
X Sensitivity
CartPole-v1
Acrobot-v1
LunarLander-122
Best 3
sce.
300
5
10
500
1000
500
1000
500
1000
1200
Distribution Shilt
Timesteps
Timesteps
Timesleps
Figure 7: For each Gym Control environment and the initial ten tasks, we identified the best X. which the regularization strength that maximizes reward for each task's specific distribution shift. We also determined the best overall weell-tuned) X for each environment. The results demonstrate that each environment and each task's distribution shift is sensitive to different À and that TRAC PPO performs competitively with each environment's ' well-tuned X.
Continuing this reasoning, we conduct a hyperparameter search for A, over various larger values [0.2, 1, 10, 15, 20, 25, 30, 35, 40, 45, 50). Given the expense of such experiments, only the more sample-efficient control environments are considered. We discover that each environment and task responds uniquely to these regularization strengths (see bar plot of values in Figure 7). This highlights the challenges of tuning À in lifelong learning context, where adjusting for each environment, let alone each distribution shift, would require extensive pre-experimental analysis.
In contrast, TRAC offers a parameter-free solution that adapts dynamically with the data in an online manner. The scaling output of TRAC adjusts autonomously the ongoing conditions, consistently competing with well-tuned À values in the various environments, as demonstrated in the reward plots for CartPole, Acrobot, and LunarLander (Figure 7).
TRAC compared to other plasticity methods Both layer normalization and plasticity injection Nikishin et al. (2024); Lyle al. (2023) have been shown to combat plasticity loss. For instance, Appendix Figure 15 demonstrates that both layer normalization and plasticity injection are effective at reducing plasticity loss when applied to the CartPole environment using ADAM optimizer. We implemented plasticity injection following the methodology laid out by Nikishin al. (2024), where plasticity is injected the start of every distribution shift. While this approach does help in reducing the decline in performance due to plasticity loss, our results indicate that consistently outperformed by TRAC across all three control environments CartPole, Acrobot, and LunarLander. Moreover, while layer normalization improves ADAM's performance, it too is outperformed by TRAC across the same control settings (Figure 15). Notably, combining layer normalization with TRAC resulted in the best performance gains.
baseline
Near convexity of lifelong RL Our results demonstrate the rapid adaptation of TRAC, in lifelong RL problems with complicated function approximation. From the perspective of optimization, the latter requires tackling both nonconvexity and nonstationarity, which is typically regarded intractable in theory. Perhaps surprisingly. when approaching this complex problem using the theoretical insights from OCO, we observe compelling results. This suggests certain "hidden convexity" in this problem, which could be an exciting direction for both theoretical and empirical research (e.g., policy gradient methods provably converge to global optimizers in linear quadratic control (Hu al., 2023)).
Limitations While TRAC offers robust adaptability in nonstationary environments, can exhibit suboptimal performance at the outset. In the early stages of deployment, TRAC might underperform compared to the baseline optimizer. We address this by proposing a warmstarting solution detailed in Appendix B, which helps increase the initial performance gap.
6 Conclusion
In this work, we introduced TRAC, parameter-free optimizer for lifelong RL that leverages the principles of OCO. Our approach dynamically refines regularization in a dependent manner, eliminating the need for hyperparameter tuning. Through extensive experimentation in Procgen, Atari, and Gym Control environments, we demonstrated that TRAC effectively mitigates loss of plasticity and rapidly adapts to new distribution shifts, where baseline methods fail. TRAC's success leads to a compelling takeaway: empirical lifelong RI scenarios may exhibit more convex properties than previously appreciated, and might inherently benefit from parameter-fiee free OCO approaches.
10

---

7 Acknowledgments
We thank Ashok Cutkosky for insightful discussions on online optimization in nonstationary settings. We are grateful to David Abel for his thoughtful insights on loss of plasticity in relation to lifelong reinforcement learning. We appreciate Kaiqing Zhang and Yang Hu for their comments on theoretical and nonstationary RL. This project is partially funded by Harvard University Dean's Competitive Fund for Promising Scholarship.
References
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp. 620-636. PMLR, 2023.
David Ahel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State ahstractions for lifelong reinforcement learning. In International Conference on Machine Learning, pp. 10-19. PMLR, 2018a.
David Abel, Yuu Jinnai, Sophie Yue Guo, George Konidaris, and Michael Littman. Policy and value transfer in lifelong reinforcement learning. In International Conference on Machine Learning, pp. 20-29. PMLR, 2018b.
David Abel, André Barreto, Benjamin Van Roy. Doina Precup, Hado Hasselt, and Satinder Singh. definition continual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, and Taesup Moon. Catastroophic negative transfer: An overlooked problem in continual reinforcement learning, 2024. URL https: / www.:ew..eentooorrrreeo id=o7BaUyX21f.
Kwangjun Ahn, Séhastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via edge of stability. Advunces in Neural Information Processing Systems, 36, 2023.
Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pp. 903-925. PMLR, 2023.
Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting optimizer deep rl: An empirical study. Advances Neural Information Processing Systems, 36, 2024.
Marc Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents, Journal of Artificial Intelligence Research, 47:253 279, 2013.
Liyu Chen, Haipeng Luo, and Chen-Ye Yu Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Conference on Learning Theory, pp. 1216-1259, PMLR, 2021.
Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In International conference on machine learning, pp.
1843-1854. PMLR, 2020.
Karl Cobbe. Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation benchmark reinforcement learning. In International conference on machine learning, pp. 2048-2056. PMLR, 2020.
Jeremy Cohen, Simran Kaur, Yuanzhi Li, Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2020.
Ashok Cutkosky. Combining online learning guarantecs. In Conference Learning Theory, pp. 895-913. PMLR, 2019.
Ashok Cutkosky and Francesco Orabona. Black-hox reductions for parameter-free online learning in banach spaces. In Conference On Learning Theory, pp. 1493-1529. PMLR, 2018.
Ashok Cutkosky, Aaron Defazio, and Harsh Mehta. Mechanie: learning rate tuner. Advances in Neural Information Processing Systems, 36, 2023.
Matthias De Lange Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3382 2021.

---

Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint urXiv:2108.06325, 2021.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning. pp. 1407-1416. PMLR, 2018.
Huang Fang. Nicholas JA Harvey, Victor S Portella, and Michael P Friedlander. Online mirror descent and dual averaging: keeping pace the dynamic case. Journal of Machine Learning Research, 23(1):5271 1-5088 2022.
Dylan J Foster, Satyen Kale, Mchryar Mohri, and Karthik Sridharan. Parameter-free online learning via model selection. Advances in Neural Information Processing Systems, 30, 2017.
Bin Hu, Kaiqing Zhang, Li, Mehran Mesbahi, Maryam Fazel, and Tamer Başar. Toward theoretical foundation policy optimization for learning control policies. Annual Review of Control, Robotics, and Autonomous Systems, 6:1 23-1888, 2023.
Andrew Jacobsen and Alan Chan, Parameter-free gradient temporal difference learning. arXiv preprint arXiv:2105.04129, 2021.
Andrew Jacobsen and Ashok Cutkosky. Parameter-free mirror descent. In Conference on Learning Theory, pp- 4160-4211. PMLR, 2022.
Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Efficient adaptation for end-ta-end vision-based robotic manipulation. In 4th Lifelong Machine Learning Workshop at ICML 2020. 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6688) 2014.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308. 1958, 2023.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Under- standing plasticity in neural networks. In International Conference on Machine Learning, pp. PMLR, 2023.
Clare Lyle. Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt. Razvan Pascanu. James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks, 2024. URL /ttpp:://axii....... abs/2402. 18762.
Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi- Levi, and Tamer Başar. Model-free non-stationary rl: Near-optimal regret and applications in multi-agent rl and inventory control. arXiv preprint 2rX0v:2010.03.66, 2020.
Brendan McMahan and Francesco Orabona. Unconstrained online linear learning hilbert spaces: Minimax algorithms and normal approximations. Conference on Learning Theory, pp. 1020-1039. PMLR, 2014.
Jorge Mendez, Boyu Wang, and Eric Eaton. Lifelong policy gradient learning of factored policies for faster training without forgetting. Advances in Neural Information Processing Systems, 10:11439--4440 2020.
Golnaz Meshahi, Olya Mastikhina, Parham Mohammad Panahi, Martha White, and Adam White. Tuning for the unknown: Revisiting evaluation strategies for lifelong rl. arXiv preprint arXiv:2404.02 2024.
Zakaria Mhammedi and Wouter M Koolen. Lipschitz and comparator-norm adaptivity in online learning. In Conference on Learning Theory, pp. 2858-2887. PMLR, 2020.
Saptarshi Nath, Christos Peridis, Escoghene Ben-twhiwhu, Xinran Liu, Shirin Dora, Cong Liu, Soheil Kolouri, and Andrea Soltoggio. Sharing lifelong reinforcement learning knowledge via modulating masks. In Sarath Chandar, Razvan Pascanu. Hanie Sedghi, and Doina Precup (eds.). Proceedings of The 2nd Conference on Lifelong Learning Agents, Proceedings of Machine Learning Research. PMLR, 2023.
Evgenii Nikishin, Max Schwarzer, Pierluca D' Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pp. 16828 16847. PMLR, 2022.
12

---

Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney. and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36, 2024.
Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213. 2023.
Francesco Orabona and Dávid Pál. Coin betting and parameter-free online learning. Advances in Neural Information Processing Systems, 29, 2016.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pp. 348-358, 2019. URL https:/ http///axvvv/v.////v//ee///aaa.. wrwg abs/ 1811. 11682.
Abhishek Roy, Krishnakumar Balasubramanian, Saeed Ghadimi, and Prasant Mohapatra. Multi-point bandit algorithms nonstationary ouline nonconvex optimization. arXiv preprint arXiv:1907.13616, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177.v:170..0634,, 2017.
Jonathan Schwarz, Jelena I Luketina, Wojeiech M Czamecki,, Agnieszka Grahska-Barwinse Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: scalable framework continual learning. ICML, 2018. URL https: /tttpp:://rriiii aarii.iiivvv....//aaaa/aaaaav.aa org/ /aob///0..... 1130000660000.0
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning, pp- 2217-2222 PMLR, 2016.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. reinforcement learning. In International Conference Machine Learning, pp. 32145-32168 PMLR, 2023.
The dormant neuron phenomenon in deep
Matthew Streeter and Brendan Mcmahan. No-regret algorithms for unconstrained online convex optimization. Advances in Neurul Information Processing Systems, 25, 2012.
S. Thrun. Explanation- Based Neural Network Learning: A Lifelong Learning Approach. Kluwer Academic Publishers, Boston, MA, 1996.
Chen-Yo Yu Wei and Haipeng Luo. Non-stationary reinforcement lcarning without prior knowledge: An optimal black-box approach. In Conference on learning theory. pp. 4350-4334. PMLR, 2021.
Annie Xie and Chelsea Finn. Lifelong robotic reinforcement learning by retaining experiences, 2022.
Zhiyu Zhang, Ashok Cutkosky, and loannis Paschalidis. Pde-based optimal strategy for unconstrained online learning. In International Conference on Machine Learning, pp. 26085-26111 15. PMLR, 2022.
Zhiyu Zhang, David Bomhara, and Heng Yang. Discounted adaptive online learning: Towards hetter regulariza- tion. In International Conference on Machine Learning, 2024a.
Zhiyu Zhang, Heng Yang, Ashok Cutkosky, and loannis Paschalidis. Improving adaptive online learning using refined discretization. In International Conference on Algorithunic Learning Theory, pp. 1208 1233, PMLR, 2024b.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradicnt ascent. In International Conference on Machine Learning, pp. 928-936, 2003.
13

---

Appendix
A TRAC Encourages Positive Transfer
To explore whether TRAC encourages positive transfer, we introduce a privileged weight-reset baseline. This baseline is "privileged" in the sense that knows when a distribution shift is introduced and resets the parameters to a random initialization at the start of each new task. We applied this baseline to three Gym control tasks: CartPole-vl, Acrobot-vl, and unnrr_ander-v2, and compared to TRAC PPO and ADAM PPO, as shown in Figure 8.
We observe that the privileged weight-reset baseline exhibits spikes in reward at the beginning of each new task. Surprisingly, TRAC maintains even higher rewards than the privileged weight-reset baseline, even at its peak learning phases. Additionally, TRAC'S reward does decline to the reward seen at the start of new tasks with privileged weight-resetting (TRAC does not have to "start over" with each task), suggesting that TRAC successfully transfers skills positively between tasks.
Adam PPO
PPO
TRAC PPO
Privllged Weight Resat
Mean Episode Rewards
8n0 400. 200-
CartPole-v1
Mean Eplsode Rewards
Acrobot-v1
Mean Episode Rewards
250
LunarLander-v2
-100
-200
2n0-
300
-500
-400
-750
1000
1900
2000
2500
3000
-500
1030
2000
5000
1090
2000
3000
Timesteps
Timesteps
Timesteps
Figure 8: Reward comparison of TRAC PPO, ADAM PPO, and privileged weight-resetting on Cartpole-vl, Acrobot-vl, and Luner-anner--22 TRAC PPO encourages positive transfer between tasks.
B Warmstarting
In our theoretical framework, we hypothesize that robust parameter initialization, denoted as ffef could enhance the performance of our models, suggesting that empirical implementations might benefit from initializing parameters using base optimizer such ADAM prior to deploying TRAC. Contrary to this assumption, our experimental results detailed in Section reveal that warmstarting is not essential for TRAC's success. Below, examine the performance of ADAM PPO and TRAC PPO when warmstarted.
Both TRAC PPO and ADAM PPO were warmstarted using ADAM for the initial 150,000 steps in all games for the Atari and Proegen environments, and for the first 30 steps in the Gym Control experiments. As seen in Figure 9, in games like Starpilot, Fruitbot, and Dodgeball, TRAC PPO surpasses ADAM PPO in the first level/task of the online setup, with its performance closely matching that of ADAM PPO in Chaser. Importantly, TRAC PPO continues to avoid the loss of plasticity encountered by ADAM PPO, even when both are warmstarted. This makes sense since all of the distributions share some foundational game dynamics; the initial learning phases likely explore these dynamics, so leveraging good parameter initialization to regularize in this carly region can be beneficial for TRAC-wc observe that forward transfer occurs somewhat in later level distribution shifts as the reward does not drop back to zero where initially started from.
Our findings indicate that warmstarting does not confer significant advantage in the Atari games. This makes sense because a parameter initialization that is good in one game setting is likely random parameterization for another setting, which is equivalent to the setup without warmstarting where TRAC regularizes towards a random parameter initializaaton. the Gym Control experiments although warmstarted TRAC PPO manages to avoid the extreme plasticity loss and policy collapse seen in warmstarted ADAM PPO, it does not perform as well as non-warmstarted TRAC PPO. This result underscores that the efficacy of warmstarting is environment-specife and highlights the
14

---

challenge in predicting when ADAM PPO may achieve a parameter initialization that is advantageous for TRAC PPO regularize towards.
From an overall perspective, warmstarting TRAC PPO in every setting still shows substantial im- provement ADAM PPO (Table 2).
Adam PPO Warmstarted
TRAC PPO Warmstarted
Starpilot
Dodgeball
Mean Episode Rewards
bo
Mean Episode Rewards
20 15 20
20-
10-
Timesteps Fruitbot
Timesteps
Chaser
Mcan Episode Rewards
Mean Episode Rewards
2-
0
-2-
00
0.6
1.0
1.5
2.0
25
3.0
0.5
1.0
1.
20
25
3.0
Timesteps
Timesteps
Figure 9: Reward in the lifelong Proegen environments for StarPilot, Dodgeball, Fruitbot, and Chaser with warmstarted TRAC PPO and warmstarted ADAM PPO. Inital performance of TRAC PPO is improved with warmstarting and continues to avoid loss of plasticity.
Adam PPO Warmstarted
TRAC PPO Warmstarted
Atari (6)
Atari (9)
Mean Epleode Rewards
Mcan Episode Rewards
1000
800
-2000
44000
Timesteps
167
Timesteps
Figure 10: Reward in the lifelong Atari environments with warmstarted TRAC PPO and warmstarted
ADAM PPO. No significant benefit found by warmstarting TRAC PPO compared to not warmstarting it.
C Other RL Baselines
While PPO widely used policy gradient method in reinforcement learning, is the only approach applicable to lifelong RL. Other continual RL methods, such IMPALA (Espeholt et al., 2018), Online EWC (Schwarz al., 2018), CLEAR (Rolnick al., 2019), and Modulating Masks (Nath et al., 2023), are designed to address challenges like catastrophic forgetting in dynamic, nonstationary environments. We incorporated these algorithm implementations adapted from code from Nath et al. (2023) into our experiments to offer more comprehensive evaluation. These methods vary in their mechanisms for maintaining task performance over time but may still suffer from plasticity loss in later stages of training.
15

---

Adam PPO Warmstarted
TRAC PPO Warmstarted
Mean Episode Rewards
CartPole-v1
Mean Episode SpisodeReward Rewards
Acrobot-v1
Mean Episode Rewards
LunarLander-v2
-100
200-
-200
200
100
-300
-500
VV
-400
-800
1000
1800
2000
2600
000
1000
2000
3000
600
1000
2000
2000
3000
Timesteps
Timesteps
Timesteps
Figure 11: Reward in the lifelong Gym Control environments for CartPole-vl, Acrobot-vl, and LunarL-ander-ve with warmstarted TRAC PPO and warmstarted ADAM PPO. TRAC PPO still avoids loss of plasticity and policy collapse.
Table 2:
Cumulative sum of mean episode reward over all distributions for ADAM PPO warmstarted and TRAC PPO warmstarted on Procgen, Atari, and Gym Control environments. Rewards are scaled by 10"; higher is better.
Environment
ADAM PPO
TRAC PPO (Ours)
Starpilot
3.0
10.2
Dodgeball
1.2
2.5
Chaser
1.3
1.6
Fruitbot
-0.4
0.6
CartPole
4.6
22.8
Acrobot
1142.9
-114.5
LunarLander
1990.7
-97.3
Atari6
16.7
72.2
Atari9
34.6
80.6
Mitigating plasticity loss across policy methods: Figure 12 demonstrates the performance of various continual RL methods when paired with ADAM and TRAC optimizers. The results indicate that when using ADAM, methods like IMPALA, Online EWC, CLEAR, and Modulating Masks exhibit noticeable decline in performance over time due to plasticity loss, particularly in later levels of the Procgen environments. contrast, pairing these methods with TRAC instead of ADAM leads to significant improvements, mitigating plasticity loss and enhancing reward performance across subsequent distribution shifts.
To quantif'y these improvements, Figures 13, 12 present the average normalized rewards over five seeds and 120M timesteps for each method across four different Procgen environments: Starpilot, Dodgeball, Chaser, and Fruitbot. Across all environments, methods that use TRAC outperform their Adam-based counterparts, consistently maintaining higher rewards over time.
On average, across the Procgen environments, TRAC AC led to performance improvements over ADAM by the following margins: 21.83% for IMPALA, 15.86% for Online EWC, 14.41 % for CLEAR, and 10.14% for Modulating Masks.
General Applicability of TRAC: It is important to highlight that TRAC orthogonal to the learning or policy algorithms themselves. It can be seamlessly integrated into various reinforcement learning architectures by simply replacing their optimizer (e.g., ADAM RMSPROP). Our results demon- strate that TRAC enhances performance across different algorithms and environments, consistently outperforming ADAM in mitigating plasticity loss.
D Gravity Based Distribution Shifts
One method to introduce distribution changes in reinforcement learning environments is by altering dynamics Mendez et (2020), such as adjusting gravity in the CartPole environment. this set of experiments, we manipulate the gravity by magnitude of ten, randomly adding noise for one distribution shift, and then inversely, dividing by ten and adding random noise for the next shift. This process continues throughout the experiment.
16

---

Starpilot with LRL Baselines
APALAASAm MPALATRAC
IMPALADA
0.8
INPAL
CLEARASAm
.... CLEAR TRAC
Average Nomralzzd Rewardd
0.5
Online EMC Adim
Average Roward
Oedine EMC TRAC
ESCC TMMC
Mesi Adamm
Mesk TRAC
04
Adam
TRAC.
0.2
0.
Timesteps
SPALLAd
APOLA TRCC
CLEAR entter
CLESR rRsc
Chima BBC AdMo
Onlne BiC TRC
Mask TRCC
Figure 12: Performance comparison between Adam- based and TRAC-based continual RL methods (IMPALA, Online EWC, CLEAR, Modulating Masks) in Starpilot. While ADAM suffers from plasticity loss in later levels, TRAC effectively mitigates this and maintains better performance distribution shifts. For clarity, standard deviation fills are omitted here but included the bar plot.
Other Procgen Games with LRL Baselines Chaser
Dodgeball
Fruitbot
1.0
1.0
1.0
Average Norallzzedewe Reward
Ooa Adan
Cer TRAC
Impe Adamn
Impa a TRAC
0.6
0.6
Online one Adan
Online one TRAC
Maed Adben
0.4
0.4
0.4
Mask TRAC
0.0
CLEAR
IMPALA
Onlna EWC
Mask
0.0
CLEAR
IMPALA
Online EWC
Mask
0.0
CLEAR
IPALL
Online EWC
Mook
Figure 13: Average normalized rewards over five seeds and 120M timesteps for Dodeball, Chaser, and Fruitbot. Each method (IMPALA, Online EWC, CLEAR, and Modulating Masks) is evaluated using both Adam and TRAC. TRAC consistently outperforms ADAM across all methods and environments, with improvements ranging from 10% 21% %.
Our observations suggest that ADAM PPO is robust to such dynamics-based distribution shifts, as shown in Figure 14. This indicates that while ADAM PPO implicitly models the dynamics of the environment well-where changes in dynamics minimally impact performance- struggles more with adapting to out-of-distributin observations such seen in the main experiments (Figure 6) and the warmstarting experiments (Figure I1).
E LayerNorm, Plasticity Injection, and Weight Decay
To evaluate TRAC alongside other methods that aim to mitigate plasticity loss, compare it against LayerNorm (Lyle et al., 2023), Plasticity Injection (Nikishin et al., 2024), and tuning weight decay (Lyle et al., 2024).
As discussed in Section we confirm that both layer normalization and plasticity injection (applied at the start of every distribution shift) (Nikishin et al., 2024; Lyle et al., 2023) are effective in reducing plasticity loss (Figure 15). While these methods help slow the decline in performance due to plasticity loss, TRAC consistently outperforms them across the three Gym Control environments. Importantly, because TRAC is an optimizer, it can combined with layer normalization, and doing so resulted in the best performance gains in our Control setups.
17

---

CartPole-v1-Gravity
Mean Episode Rewards
800
200
100
503
1000
2000
2530
3000
Timesteps
Figure 14: Mean Episode Reward for ADAM PPO on CartPole-vl with varying gravity. ADAM PPO demonstrates robust policy recovery across most gravity- -assdd distribution shifts.
Tuning weight decay: In addition to LayerNorm and Plasticity Injection, we also evaluated the effects of tuning weight decay using PyTorch's 's AdamW optimizer. We conducted a hyperparameter sweep across three control environments with 15 seeds for each of the following weight decay values: 0.0001, 0.001, 0.01, 0.1, , 1.0, 5.0, 10.0, 15.0, and 50.0. Figure 16 presents the average normalized reward for each weight decay value over 15 seeds and 3000 timesteps, compared to TRAC.
The results indicate that while tuning weight decay with Adam does provide some benefit, these values consistently underperform in comparison to TRAC across all three control environments. Figure 16 plots the performance of the best-performing weight decay value with Adam over 10 distribution shifts in the control environments. We observe that weight decay values are highly sensitive to the specific environment and the nature the distribution shift.
Interestingly, in our initial experiments, we set the weight decay to zero, yet TRAC still outperformed Adam with various weight decay values. This suggests that while weight decay can mitigate plasticity loss to some extent, does not match the overall effectiveness of TRAC.
CartPole-v1
All Control Environments
Averag Nornalzed led Rewardd
Average Reward
400
0.75
0.50
200
0.25
0.00
CartPole-vl
Acrobot-vi
LunarLander-ve
Timestep
Adam
TRAC
Layer norm Adam
norm TRAC
Plasticity Injection Adam
Figure 15: Performance comparison of plasticity loss mitigation techniques across Gym Control environments. Both layer normalization and plasticity injection reduce plasticity loss when applied with ADAM. TRAC outperforms both layer norm ADAM and plasticity injection ADAM, with the combination of layer norm and TRAC achieving the highest performance.
F Scaling-Vall Value Convergence
As discussed in the algorithm section (see Section 3), TRAC operates as a meta-algorithm on top of standard optimizer, denoted as BASE. The crucial component of TRAC involves the dynamic adjustment of the scaling parameter S1+1, managed by the tuner algorithm (Algorithm 2). This parameter is data-dependent and typically ranges between [0, 1]. The weight update 8,+1 consequently defined as a convex combination of the current optimizer's 's weight gRaSE and predetermined reference point O
This section presents the convergence behavior of the scaling parameter S across different environments, analyzed through the mean values over multiple seeds.
18

---

Average Normalized Reward Across Control Environments (over 25 seeds)
Car1Pole-v1
Acrobot-v1
LunarLander-vv
Average Nermalized Reward
Adam Weight Decay
Average Normalized Reward
Average Nermalized Reward
TRAC
0.4
0.2
we 0.0001
0,001
dor
oz
01
1.0
10,0
18.0
feat
wid 5.0
60.0
TRAC
wel 000000
ne 0.001
we 0.01
we 0.1
nel 1.0
wed 10.0
med (622
5.0
we 50.0
TRAC
wel 0.0007 we 0.e01
0000
wel wel 1.0
10.0
wil 16,26
60.0
TRAC
wed
and
Figure 16: Effect weight decay on performance in the three Gym Control environments. Bar plots show the average normalized rewards over 25 seeds for different weight decay values using ADAM across 3000 timesteps, compared to TRAC with no weight decay.
Starpilot
Dodgeball
0.13
0.09
0.94
0.02
0.02
0.0
32
E4
6.6
0.6
1.0
3.0
03
0.6
68
19
Update Step Chaser
Update Step Fruitbot
8.10
0.02
0.22
Update Step
Update Step
Figure 17: Convergence the scaling parameter S.41 in the Procgen environments.
The convergence of the scaling parameter S141 observed across the Proegen and Gym Control envi- ronments, depicted in Figures 17 and 19, reflects good scaling value that effectively determines the strength regularization towards the initialization points, yielding robust empirical outcomes in lifelong RL settings. Interestingly, in Procgen environments, this converged scaling value exhibits consistency across various games, typically hovering between 0.02 and 0.03, as shown in Figure 17. In contrast, in the Gym Control environments, the scaling values are lower, ranging between 0.005 and 0.01, as illustrated in Figure 19.
G Comparison to MECHANIC
In our analysis, we extend the examination to other OCO-based optimizers within the lifelong RL setup. Table presents comparative assessment of TRAC PPO and MECHANIC PPO (Cutkosky et al., 2023) for the lifelong Gym Control tasks (with 300 seed runs). The p-values were calculated using two-sample t-tests test the hypothesis that means between TRAC and MECHANIC are the same (Null Hypothesis, Hn) against the alternative hypothesis that they are different (Alternative Hypothesis, H1). The results indicate that while MECHANIC effectively mitigates plasticity loss and adapts quickly to new distribution shifts, it slightly underperforms in comparison to TRAC.
19

---

Alari (6)
Atari (9)
0.96
0.10
0.04
3.56
0.55
wun
My M
0/91
0.00
20000
190060
108000
20006
200030
393600
350033
433000
50000
100300
120003
200330
330000
500100
330300
100000
Update Step
Update Slep
Figure 18: Evolution of the scaling parameter St-1 in the Atari environments. Here don't 't see meaningful convergence of S1+1-
CartPole-vl
Acrobot-v1
LunarLander-v2
0.08
0.04
0.06
9
0.02
0.04
0.01
0.02
0.00
0.00
26000
GO000 76000 100000
129000
20000
60000
76000
100000
125000
26000
GO000
70000
100000
120000
Update Step
Update Step
Update Step
Figure 19: Convergence of the scaling parameter S2+1 in the Gym Control environments.
H Experimental Setup
Procgen and Atari Vision backbone For both the Atari and Procgen experiments. the Impala architecture was used as the vision backbone. The Impala model had 3 Impala blocks, each containing convolutional layer followed by residual blocks. The output of this is flattened and connected to fully connected layer. The impala model parameters are initialized using Xavier uniform initialization.
Policy and Value Networks Across all experiments sxperiment--includnng Gym Control, Atari, and Proc- gen-thhe policy and value functions are implemented using multi-layer perceptron (MLP) architec- ture. This architecture processes the input features into action probabilities and state value estimates. The MIP comprises several fully connected layers activated by ReI.U. The output from the final layer uses a softmax activation.
TRAC TRAC, for all experiments, was implemented using the same experiment-specific baseline architectures and baseline optimizer. For the Procgen and Atari experiments, the base ADAM optimizer configured the same baseline, with learning rate of 0.001, and for the Gym Control experiments, a learning rate of 0.01 was used. Both learning rates were tested for all experiments and found have negligible differences in performance outcomes. Other than the learning rate, we use the default ADAM parameters, including weight decay and betas, followed by the specifications outlined in the PyTorch Documentation.
The setup for TRAC included values for adaptive gradient adjustments: 0.9, 0.99, 0.999, 0.9999, 0.99999, and 0.999999. Both St and were initially set to (1 x 10 5). Modifications were made to Py Torch error function library, which accepts complex inputs to accommodate the necessary computations for the imaginary error function. This library be found at Torch Erf GitHub.
Distribution Shifts In the Atari experiments, game environments were switched every 4 million steps. The sequence for games with an action space of 6 included "BasicMath", "Qbert", "SpaceIn- vaders", "UpNDown", "Galaxian", "Bowling". "Demonattack" "NameThisGame" while games with an action space of included "Lostl "LostI uggage"" "VideoPinball", "BeamRider", "Asterix", "Enduro", "CrazyClimber", "MsPacman". "Koolaid".
http:///ttootob.orr///sssssstttt wtwreatttddtt......eee http:/ttcc.c................ Adam Admm.html
20

---

Table 3: Performance comparison between TRAC and MECHANIC across three Gym Control envi-
ronments. The mean, standard error, and p-values reflect the performance over multiple runs, with bolded values highlighting TRAC's superior results.
Task
Method
Mean
Std Error
p-value
Lunarl ander-22
TRAC
0.6018
0.0036
0.0000
Mechanic
0.5755
0.0027
CartPole-vl
TRAC
0.3518
0.0244
0.0021
Mechanic
0.3008
0.0230
Acrobot-vl
TRAC
0.7044
0.0221
0.0000
Mechanic
0.6396
0.0239
Table 4: PPO Parameters for Atari, Procgen, and Gym Control Experiments
Parameter
Atari
Procgen
Control
Steps per update
2,000
1,000
800 (2 episodes with 400 steps)
Batch size
250
125
32
Epochs per update
Epsilon clip for PPO
0.2
0.2
0.2
Value coefficient
0.5
0.5
0.5
Entropy coefficient
0.01
0.01
0.01
Base Optimizer
ADAM (LR: 0.001)
ADAM (LR: 0.001)
ADAM (LR: 0.01)
Architecture
Impala + MLP
Impala MLP
MLP
For Procgen experiments, individual game levels were sampled using a seed value as the levellevel parameter, which was incremented sequentially to generate new levels. Each new environment was introduced every million steps.
In the Gym Control experiments, each observation dimension was randomly perturbed by value ranging from 0 This perturbation was constant for 200 timesteps, after which new perturbation was applied, effectively switching the environmental conditions every 200 steps.
Statistical Significance The Proegen and Atari experiments were conducted with seeds/runs,
while the Gym Control experiments utilized 25 seeds/runs (with the exception of the Mechanic experiments in Table 3 which utilized 300 seeds). The exception was in the L2 initialization experiments, which used 15 seeds/runs 15, 12, 14, the plotted lines represent the mean of all of the mean episode rewards from the different seeds/runs, and the shaded error bands indicate the standard deviation of all of the mean episode rewards from the different seeds/runs.
regularization strength. In Figures 4, 5, 6, 7, 9, 10, 11,
Compute Resources
For the Procgen and Atari experiments, each was allocated single A100 GPU, typically running for 3-4 days to complete. The Gym Control experiments were conducted using dual-core CPUs, generally concluding within few hours. In both scenarios, an allocation of 8GB of RAM was sufficient meet the computational demands.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As said in our abstract, we offer parameter- free optimizer that performs well in empirical lifelong RL environments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have a limitations paragraph in our discussion section (5).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We do not provide novel theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include a comprehensive experimental setup section in our Appendix (H). We also provide code submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: We provide python jupyter notebook in our supplementary material to walk through how setup and reproduce the results for the computationally less expensive experiments (which have similar setup to the computationally expensive experiments).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/ Sewiiddes/CoeeSuiisssiiii.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The full experimental details can found and in the code submission in our supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our reward plots include standard deviation error bars around the curves. We detail this in the experimental setup section of our Appendix (H).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer:[Yen Yes]
Justification: Full details of our copmute for each experiment can be found in our H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: http///eeerppp.eeeeeeeeee ce wc/ppbblc//teiiieeeeeeeeee httppuuicc/ttiiccciiiiiiiieeesss
didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics in every manner.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: Not applicable.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA].
Justification:
We dot existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "af9pWcp7AC", "reviewer": "Reviewer_eCfR", "summary": "The paper tackles the problem of plasticity loss in RL, where a neural network may have reduced performance on learning new tasks after a task switch. The authors develop a hyperparameter-free algorithm, PACE, using ideas from online convex optimization. It can be interpreted as an automatically-tuned weight decay adapted during training. Experiments on large-scale benchmark tasks demonstrate the effectiveness of the approach compared to different baselines.", "questions": "_Clarification questions_\n- Fig. 2: I do not fully understand what this figure is trying to convey. In particular, how is the figure demonstrating that PACE may be beneficial in this scenario? Maybe an explanation in the caption could help.\n\n- For the section on tuning $\\ell_2$ regularization (line 280), could you include a hyperparameter sensitvity plot i.e. performance vs. hyperparameter value? I would be curious to see how the performance varies around the best values.\n\n- Fig. 6 In the experiments, it seems like even PACE suffers from some amount of plasticity loss since the learning curve does not reach the same maximum as in the first task (more evident in CartPole or LunarLander). Do you have any hypotheses to explain this?\n\n_Comments and suggestions_ \n\n\n- Looking at the values of $S_t$ over time (in appendix C) they are usually around 0.01-0.1, which seems fairly small and have a distinct trend of decreasing over time.\nThis looks inversely correlated to the parameter norm, a quantity that is often measured in the context of plasticity loss. Have you looked into this relationship? I wonder if plotting $S_t \\times ||\\theta^{Base}||$ shows a roughly constant effective parameter size. \n\n- If I understand PACE correctly, it looks like overall, it produces the next parameter $\\theta_t$ through a weight $S_t$ which is used to obtain convex combination between the initial parameters and the base-optimized parameters $\\theta^{Base}_t $. \nWhat I find counterintuitive is that the gradients for the next step are then evaluated at $\\theta_t$ but used to update $\\theta^{Base}_t$, which could be quite different in terms of the policy implied. \nDo you have any thoughts about this?\n\n- A potential explanation, as discussed in [3], is that rescaling may also preserve the argmax for classification tasks (and the policy in this case) although it does affect the size of the logits. This would not explain effects on the value network though. \n\n- On a related note, do the neural networks feature normalization layers e.g. layer norm? \nWhen normalization layers are used, there is a scale-invariance introduced in the weights (well, if there are no bias units, see page 1. of [2]). I could see PACE being more effective in conjunction with normalization layers due to this rescaling operation not impacting (or having a small impact on) the function learned.  \n\n- About \"near convexity\", line 298. While the algorithm components are developed for the OCO setting, do you think it is possible that, instead of the problem being \"near convex\", it is the algorithms that have wider applicability than previous appreciated? That is, they may work well in a more general class of problems?\n\n[1] \"Understanding plasticity in neural networks\" Lyle et al. \n\n[2] \"Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction\" Lyu et al. \n\n[3] \"On Warm-Starting Neural Network Training\" Ash and Adams", "limitations": "These are discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper builds a bridge between the fields of lifelong RL and online convex optimization (OCO), successfully using ideas from OCO to develop an algorithm addressing plasticity loss. It can be difficult to use fruitfully utilize theoretical ideas in a practical context so I commend the authors for that.\n\nThe paper is written clearly and generally easy to follow. Intuitions for the theory are provided which are helpful to understand the algorithm.\nThe evaluation is extensive and covers a wide variety of RL tasks, with larger-scale benchmarks such as ProcGen.\nI also appreciate some of the more in-depth analyses such as inspecting the scaling $S_t$ over time in appendix C.", "weaknesses": "Generally, I have not identified any major weaknesses for this paper.\nSome areas that could be improved (but are still acceptable):\n- Only a couple of baselines are run, including weight decay and using CReLU activation. \n\tA suggestion would be to try layer norm since it is also parameter-free and has shown some benefits to address plasticity loss [1].\n- While the algorithm is theoretically-motivated, I did not see any theoretical results included in the paper. If possible, it could be nice to have a theoretical guarantee for PACE even if it is in a different setting such as OCO. Such a result could give additional confidence that the algorithm is sound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for highlighting the strengths of our work!\n\n## Layer normalization helps:\nThank you for suggesting the use of layer normalization, as highlighted by Lyle et al. for addressing plasticity loss, in our experiments. We implement layer normalization in our control experiments for both PPO with Adam and PPO with PACE (Figure 3 in Supplementary PDF). Our results show that while layer normalization improves the performance of baseline Adam, it does not outperform PACE. Moreover, we find your hypothesis to be true: PACE with layer normalization performs better than PACE without layer normalization  (see Figure 3b in Supplementary PDF).\n\n\n## Comments\n\n* Thank you for your close reading of our method and the appendix. We unfortunately did not have the opportunity to examine the relationship between the parameter norm and our scaling approach, but we are excited to explore this analysis in future work!\n\n\n* Regarding your question about updating $\\theta_{t}^{Base}$ by evaluating the gradients at $\\theta_{t}$, intuitively, there is more stability around the reference point. Therefore, updating the gradients from a base update near this safe region helps prevent issues with gradients that might otherwise explode or vanish if updated at $\\theta_t$\\.\n\n* We agree with your observation that the success of parameter-free optimization may not indicate a “hidden” convexity, but rather that these methods may have broader applicability. This is an important point, and future research should indeed investigate why these methods are effective in such settings.\n\n* Regarding Figure 2, we appreciate your feedback and will update the caption to better explain our visual abstraction of the setting.\n\n* You correctly pointed out that in Figure 6, while PACE combats plasticity loss, it may still be affected by it, as the reward does not reach the same level as during the first distribution shift. Although good, we believe PACE's scaling isn’t always optimal, especially in a non-convex setting. There may be a more \"optimal\" set of scaling values such that the performance is as good as the first task, and this is something that may evolve as parameter-free methods continue to improve.\n\n\nPlease let us know if you have any other concerns or comments, and we'll be glad to address them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PzUZbvELhR", "reviewer": "Reviewer_346D", "summary": "This work proposes a new method designed to mitigate the loss of plasticity when a model is continuously retrained on a set of tasks. The approach uses online convex optimization (OCO) via the minimisation of a regret function. The approach is compared against two baselines: PPO with ADAM (fixed learning rate) and CReLU. The benchmarks used are ProcGen, the Arcade Learning Environment (ALE) and Gym Control. The empirical tests show promising results with respect to standard reward metrics. Overall, the paper engages with a new and emerging field in machine learning that assume changing input distributions, multiple tasks in various contexts, e.g. supervised and reinforcement learning. Moreover, the paper addresses a recently discovered property of optimizers that reduce the ability of the network to continuously adapt to new tasks if continuously retrained (plasticity loss).", "questions": "My questions are a direct follow-up on the limitations\n1. Would you consider comparing your approach to lifelong reinforcement learning algorithms?\n2. Why are all the three techniques from the parameter-free OCO required? What do they accomplish individually? Could you run ablation studies? \n3. How does the ADAM default (adaptive learning rate) perform? I understand the need to freeze the learning rate to allow the proposed algorithm perform the adaptation, but it would be nevertheless valuable to assess the performance of the standard version of ADAM.", "limitations": "The limitation section is mainly focused on a narrow aspects, e.g. the need for a warm-up. However, this is also in contradiction with the Appendix in which it is suggested that such a warm-up is not required. \n\nMy conclusion is that the narrow scope of this limitations section stems from the lack of integration of this work with the broader field of lifelong reinforcement learning. Addressing the weaknesses that I pointed out, and engaging with a more thorough comparison of this approach with existing LRL approaches, a more insightful view of the pro and cons should emerge.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The development of a parameter-free optimiser to enhanced performance under non-stationary condition is a interesting research venue in the scope of lifelong reinforcement learning.\n- Addressing the issue of loss of plasticity in dynamic scenarios with shifting distribution is an emerging area in lifelong learning. \n- The paper has a good set of benchmarks that are specific to lifelong reinforcement learning challenges.", "weaknesses": "1. The core of the novel algorithm is explained with pseudo code in Algorithm 1 and 2. I understand that the authors are very familiar with all the details of several papers by Cutkosky (at least 4 main sources), on which this work builds on\n- direction-magnitude decomposition from (Cutkosky & Orabona, 2018), \n- the additive aggregation from(Cutkosky, 2019), \n- and the erfi potential function from (Zhang et al., 2024b). \n-  (Cutkosky et al., 2023) for the recommended default parameter setup.\n\nHowever, it's not obvious to me how clear the methodology is given the explanation in Section 3 to machine learning experts that are not very familiar with Cutkosky's work. \n\n2. A clear highlight of the core technical novelty is missing in relation to previous work. I understand that this work \"builds on\" the cited work above, but it is generally not a sufficient statement of novelty. More effort is required to clearly spell the novel aspects with respect to previous work. \n\n3. The most notable limitation that mostly affects my final evaluation is that this method for lifelong reinforcement learning (LRL) **is compared to only two baselines that are not lifelong reinforcement learning algorithms**. I understand that such choice might derive from the narrow focus of preventing plasticity loss. However, a strong paper in LRL needs to engage with the field more thoroughly, describe notable existing LRL, and how the proposed methods contributes to advancing the field. Candidate baselines could include RL implementations of EWC, Experience Relay based RL methods (e.g. Rolnick's CLEAR), PackNet by Mallya and Lazebink or Modulating Masks for LRL by Ben-Iwhiwhu, just to mentioned a few. I find it odd that in the limitation section, the suggestion is the future application of the algorithm to SAC and DQN, which again are not LRL algorithms.\n\n4. As a follow-up to the previous limitation, the paper would benefit from engaging with the LRL field from the start, not only by describing the Spot example (which sounds like the authors are the first to highlight this to the community), but also describing methods that have been developed to address that problem over the last decades, their achievements, their limitations, and why this particular new approach is needed. \n\n5. Limited analysis is provided on the inner-working of the algorithm (e.g. no ablation studies). \n\n6. Improving the clarity of the hypotheses would benefit the paper. Is the hypothesis the fact that OCO can be used to reduce plasticity loss? Or that it can be used to reduce plasticity loss better than existing algorithms that do that? Or that it can be used to implement LRL? These questions are only partially answered or addressed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and constructive feedback!\n\n## Lifelong RL baselines\n\nFirstly, we acknowledge your comment on the lack of baselines. We recognize the important contributions of Ben-Iwhiwhu et al., 2023, Rolnick et al., 2019, Schwarz et al., 2018, and others in the field of lifelong RL. However, we didn't include these baselines since we consider our setting and scope to be a bit different (i.e., no task recycling, shared curricula, or privileged task boundaries; more on this soon). However, following your suggestion, we incorporated these baselines and found that the results (detailed below) support our hypothesis.\n\n### Results\n\nUsing open-source code from Ben-Iwhiwhu et al., 2023, we evaluate Adam and PACE across Impala, Online EWC, CLEAR, and Modulating Masks in our Procgen games. We modify the continual RL setup from Ben-Iwhiwhu et al. (2023) and Powers et al. (2022) to our lifelong RL setting (modifications are detailed in the Author Rebuttal).\n\nWe present a comparison of average normalized rewards over 5 seeds for each method and game in Supplementary PDF Figures 1b-e and plot the average reward curves, excluding STD fill for clarity, in Supplementary PDF Figure 1a for Starpilot.\n\nIn the Procgen environments, **PACE consistently outperforms Adam across various learning algorithms**: \n \n* IMPALA **by an average of 21.83%**\n* Online EWC **by an average of 15.86%**\n* CLEAR **by an average of 14.41%**\n* Modulating Masks **by an average of 10.14%**.\n\nIn Supplementary PDF Figure 1a, we also observe when using Adam, *these methods are still vulnerable to loss of plasticity* in later levels (especially Online EWC). \n\nWe are happy to include these results in our work and revise the introduction to acknowledge the history of these methods in the context of lifelong RL.\n## Clarifying our Lifelong Setting \n\n* Our setting does not revisit old tasks nor assure any task similarity. \n* We operate in a strictly online setup, prioritizing the evaluation of rapid adaptation to new tasks without task boundaries (which mask-based and EWC-based continual RL approaches require). \n* In our setting, we demonstrate that plasticity loss occurs when adapting to new tasks never seen before. \n\n\nIn contrast,\n* Ben-Iwhiwhu et al. (2023) focus on task recycling to balance learning new tasks with performance on previously seen ones. We consider this as a continual RL setting; indeed, Ben-Iwhiwhu et al., 2023 use \"CORA: A Continual Reinforcement Learning Benchmark\" (Powers et al., 2022) for Procgen experiments, where previously seen tasks are evaluated (backward transfer) during training. \n* In this same setting, Abbas et al. (2023) observe loss of plasticity *when tasks are revisited*.\n\n\n## Clarifying our scope and hypothesis\nWe would like to clarify our work specifically addresses the loss of plasticity problem in lifelong RL through the *lens of optimizers*, such as Adam.\n\n**Hypothesis**: Parameter-free optimization mitigates loss of plasticity more effectively than Adam, regardless of the learning algorithm or architecture employed (e.g., PPO, PPO with CReLU). \n\n**Conclusion:** We introduce a new parameter-free optimizer, PACE, and show it better preserves plasticity compared to Adam across 4 Procgen, 2 Atari, and 3 Gym Control lifelong settings. \n\n**We would like to note that PACE is orthogonal to learning or policy algorithms and architectures  (eg. PPO, CLEAR, Modulating Masks).** As seen in the results above, PACE can be integrated with other RL algorithms simply by replacing the optimizer (like Adam or RMSPROP) that they use to train with. \n\n**Evaluation Choice:** We opted to assess many environments with different input dimensions and distribution shifts, rather than many algorithms, to evaluate different modes of plasticity loss and isolate the effects of Adam and PACE. However, we have addressed your feedback on incorporating more baselines in Supplementary PDF Figure 1 and above.\n\n## Comparison to parameter-free OCO\n\nWe agree that our explanation of parameter-free OCO is somewhat abstract, given the space constraints. We will enhance more clarity in the camera-ready version!\n\n**On the (un-)necessity of an ablation study:** Parameter-free OCO has evolved over the last decade into a standard framework, primarily based on extending 1D algorithms to higher dimensions (Cutkosky & Orabona (2018)) and a data-dependent method for selecting the discount factor (Cutkosky (2019)). These contributions are critical for ensuring optimal performance in convex settings—considered the benchmark for optimizers—and are equally vital in nonconvex applications. Omitting these would undermine the algorithm’s effectiveness and theoretical motivations.\n\n\n**Our novelty.** When moving beyond convexity, existing works on parameter-free OCO consider training DL models. Intuitively, this doesn't fully utilize the capability of such algorithms: by design, OCO algorithms are able to handle even adversarially varying nature, whereas DL training is \"stationary.\" Our work goes beyond this limitation by considering lifelong RL as an intrinsically \"nonstationary\" problem. This is a nontrivial \"out-of-the-box\" move for OCO, as the latter cannot capture the loss of plasticity we aim to address. In our opinion, the insight is pleasingly neat: \n- parameter-free OCO is all about better regularization and\n- better regularization mitigates loss of plasticity. \n\nFurthermore, PACE employs the 1D algorithm from Zhang et al. (2024), which is theoretically optimal yet untested in real-world deep learning training.\n\n\n## Minor: Adam\n\nWe use default PyTorch Adam, which we note is adaptive but *still* relies on an initial LR as a hyperparameter (in contrast, PACE is insensitive to LR).\n\nPlease let us know if you have any other concerns/comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oot4q3xdzJ", "reviewer": "Reviewer_Jqmp", "summary": "Many methods have been proposed recently for continual learning that can maintain their plasticity (e.g., through resetting or regularization towards initialization). There is, however, one critical limitation of those methods, which is their hyper-parameter tuning. Although those methods are designed for lifelong learning that can potentially last forever, their performance is highly dependent on the hyperparameter choices. This paper attempts to address this gap in research by introducing a tuning-free approach inspired by online convex optimization for continual learning by incorporating multiple ideas from online convex optimization research and showing that their method PACE is insensitive to hyperparameter choices and can work in a wide range of problems.", "questions": "- What is a round? (line 121)\n- No definition of discounted regret has been given. How is it defined?\n- “Algorithms there mostly follow  the same principle as in the stationary setting, just wrapped by loss rescaling (Zhang et al., 2024a).\" What does this mean?\n- What does the distribution shift (from 1 to 10) mean in Figure 7 on the x-axis of the first plot? \n- Why is the cumulative sum of rewards used while the authors mentioned that discounted regret is suitable for lifelong RL? \n- The paragraph starting at line 126 is unclear. What does uncertainty set $\\mathcal{U}$ mean or represent?", "limitations": "The authors adequately addressed the limitations of their work. However, I think since PACE exhibits suboptimal performance in the early stages of deployment, it might suggest that it doesn’t also adapt to a fast-changing sequence of tasks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "This paper addresses an important limitation of existing continual learning methods, which is their dependence on hyperparameter tuning. The paper presents a novel approach that requires no tuning for hyperparameters and demonstrates the effectiveness of their method on a range of problems. The implications of tuning-free continual learning are immense and can advance future research on continual learning.", "weaknesses": "- Writing needs to be improved. Several parts of the paper seem incomplete, which may require significant restructuring of the paper.\n  - The information on online convex optimization doesn’t fully introduce relevant concepts used in the algorithm. Thus, they either need to be removed, and the paper would rely on general intuition to motivate their method and explain how it’s generally related to OCO or to introduce the method in terms of OCO rigorously. The paper, as it stands, does not do either option.\n  - The algorithm is not fully explained. For example, how the three components work together is not explained, and the paper relies entirely on citing the papers in which they incorporate their ideas in the algorithm. This prevents the paper from being self-contained. Additionally, what does the dot product between the instantaneous gradient $g_t$ and $\\theta_t- \\theta_{\\text{ref}}$ represent? This was never explained, although it’s a crucial part of the algorithm. The following questions should be answered by reading the algorithm and its explanation: 1) the role of the erfi function and 2) why are we keeping track of the first and second moments of the dot product between the current gradient and $\\theta-\\theta_{\\text{ref}}$? 3) how does the additive aggregation work? \n  - Writing should be more focused on what is crucial for following ideas explained in the paper. For example, the paper explained static regret and then mentioned that discounted regret is more suitable only in one sentence without a proper introduction. Similarly, the paper explains the minimax algorithms for static regret but never explains how parameter-free methods are different. I don’t thinking static regret or minmax formulation is helping in the flow of the paper, instead it may hurt it since it distracts the reader from the main message without mentioning the importance of parts for following along.\n- The effectiveness of the three components of the proposed method has not been investigated. Specifically, an ablation study is needed to understand the role of each component and how they interact with each other. For example, the authors might add an ablation study where one component is removed at a time or one component is added at a time.\n\n**Minor issues and suggestions:**\n- The following claim in the conclusion section is unsubstantiated: “PACE’s success leads to a compelling takeaway: empirical lifelong RL scenarios may exhibit more convex properties than previously appreciated and might inherently benefit from parameter-free OCO approaches”. The effectiveness of your method doesn’t mean it works because the problem has convex properties. More work is needed to support such a claim and find direct evidence.\n- I suggest the method may be called tuning-free, not parameter-free, to enhance readability for a wide range of audiences.\n- “From the optimization perspective, the above issues might be attributed to the lack of stability under gradient descent.” -> This requires a citation. Also, what does stability mean in this context?\n- line 78: $T$ is not defined. Also, it's doubly used to mean a different thing in line 120.\n- Notation in equation 1 is not fully defined (e.g., what is $o(T)$?)\n- The word \"conservatism” is used loosely without properly defining what it means.\n- I believe the “Lifelong vs. Continual” paragraph needs to be dropped. The two words mean the same thing. There is no need to use them in different contexts to mean different things. This would lead to unnecessary division in using the word. Further, I don't think Abbas et al. (2023) focused on backward transfer; instead, they focused on the loss of plasticity. \n- the axis in Figure 4 should be the “average episodic return” or \"average cumulative rewards\" not “average episode rewards”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very thorough review! Especially, we appreciate your valuable comments on the writing and organization of this paper. We'll incorporate those in the camera-ready revision. \n\nRegarding your questions: we realized that we went a bit too fast when introducing the algorithm. So first of all we would like to clarify its main idea, as well as the role of each component.\n\n## Idea of PACE\n\nThe overall structure of PACE results from a decade of research on parameter-free OCO. The latter as a branch of optimization theory concerns proving regret bounds under convex losses, which has a similar spirit as our problem but very much idealized (e.g., there isn't any notion of plasticity etc). \n- The central component of PACE is the direction-magnitude decomposition from (Cutkosky & Orabona, 2018), whose main idea is to apply a carefully designed 1D algorithm (parameter-free tuner) on top of a base optimizer to adjust how far the iterates drift away from the initialization. With the right tuner (Alg 2), this essentially acts as a data-dependent regularizer. It helps mitigating plasticity loss due to the high plasticity associated with the initialization (around the origin; see the insights of Abbas et al, 2023).\n- The tuner we use is based on the erfi function, due to (Zhang et al, 2024b). The design of this tuner is a long story (solving a PDE called the Backward Heat Equation), but intuitively, the idea is that the shape of the erfi function trades off the distance of the iterates from the origin and from the empirical optimum. Such a tradeoff is mathematically optimal in a quite strong sense. \nThe tuner takes the inner product $\\langle g_t,\\theta_t-\\theta_{\\mathrm{ref}} \\rangle$ as input, which is essentially the projection of the gradient along the update direction (since tuner controls the magnitude rather than direction, it only cares about the gradient component on the direction of $\\theta_t-\\theta_{\\mathrm{ref}}$). \n- As in much of practical model training, the tuner requires discounting. To automatically select the discount factor, we use the additive aggregation technique from (Cutkosky, 2019). The argument is very simple: if we *sum up* two parameter-free OCO algorithms, then the resulting algorithm performs almost as well as the best one among the two. That is, (at least in the convex setting) we can select the optimal discount factor for free. \n\nWe note that these three components crucially work together to guarantee good regret bounds in the convex setting. Since this is often the minimum requirement for any reasonable optimizer, we kept all three components in the nonconvex setting as well (otherwise the algorithm wouldn't make sense). This (ruling out design altenatives) is in our opinion a big benefit of theory-driven research.\n\n## Minor issues\n\n- **On the convexity.** We essentially mean that although parameter-free OCO builds on the structure of convexity, it's intriguing to see that it works well on lifelong RL, a nonconvex and nonstationary task. We'll make this clearer. \n- **Stability.** Here we mean the iterates shouldn't drift too far. This is the same reason for the importance of weight decay in DL, which is somewhat widely accepted by the community. \n \n\n## Questions\n- A round refers to an iteration. \n- We agree with you that the concept of discounted regret isn't so relevant to our exposition, so we'll remove it. \n- Loss rescaling means on Line 5 of Alg 2 (the tuner), we multiply $\\beta$ during the update of $v_t$ and $\\sigma_t$. The theory behind this is justified in (Zhang et al, 2024a).\n- **Distribution shift in Figure 7.** In the control environments, a distribution shift refers to altering the observation's dimensions by adding or subtracting random noise for a duration of 200 steps. This noise is changed every 200 steps. On the x-axis of Figure 7, each distribution shift number corresponds to a different 200-step interval. For example, Distribution Shift 1 covers steps 0-200, Shift 2 covers 200-400, and so forth.\n- **Cumulative reward sum.** The discounted regret is a suitable metric for the theoretical analysis in OCO, but in reality (experiments and real world), we only care about cumulative reward sum (as in almost all existing works). Somewhat inevitably, this is one of the common aspects where theory and practice differ. \n- **Uncertainty set $\\mathcal{U}$.** Intuitively (in the context of lifelong RL) this could be thought as the loss landscape: since we only have gradient feedback, the loss landscape remains uncertain for our algorithm throughout its operation. \n\nSince parameter-free OCO is itself a quite sophisticated field, our exposition focused mostly on the high level main idea. Please let us know if you have any other concerns / comments, and we'll be glad to address those!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l02g09CyEn", "reviewer": "Reviewer_Jcag", "summary": "This paper proposes a new method for avoiding plasticity loss in 'lifelong RL', a setting in which distribution shift occurs during RL training. Here, the setting is split into a number of discrete tasks which change at a fixed interval. The method involves including a penalty on updates, rooted in convex optimization, which encourages movement back to a reference parameterisation. Upon testing, they find that performance sees drastic improvement over two baselines, and does not suffer from the effects of plasticity loss in the same way.", "questions": "- As mentioned earlier, could you please offer a more rigorous comparison with Mechanic (or explain in further detail why PACE is much better, since the difference seems so far very marginal)?\n- How much weight decay was used for Adam? Since this method is very closely aligned to weight decay, and the S value was usually *roughly* a constant scale, could tuning the extent of weight decay provide the same benefit as PACE? As pointed out by [1], weight decay and L2 regularisation are marginally different when using Adam. I appreciate this inroduces an additional tunable hyperparemeter, which is against the spirit of the paper.\n\n\n\n[1] Kumar, Saurabh et al. “Maintaining Plasticity in Continual Learning via Regenerative Regularization.” (2023).", "limitations": "The authors discuss some limitations of their work, such as limited analysis with respect to different algorithms and a brief discussion of Mechanic, another method which performs very similarly to PACE. While I have briefly discussed additional results in regards to limitations, I feel the authors are upfront about a lot of the limitations of their work and that these are not overly limiting to the work.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Results are clearly drastically improved with the introduction of PACE, compared to the algorithms which it is benchmarked against *in the main paper*.\n- The work seems well motivated, with the section on Lifelong RL outlining the justification for why this is a useful direction to study.\n- The work was grounded in theory. The justification for design decisions in the context of OCO literature clearly enhanced the method.\n- The paper was clearly written in many places, with clarity lost in only a small number of places (see weaknesses).\n- I have personally not seen previous work which pursues this direction of hyperparameter-free plasticity loss mitigation, particularly in this way. I think this is project has good novelty.\n- The results are demonstrated on a wide array of environments, which I believe to be a sensible distribution for evaluation in this setting.\n- The figures were great for offering intuition about what was happening in the method, and the problem setting.\n- The extra analysis in the appendix was quite enlightening and was appreciated.", "weaknesses": "There are a few key weaknesses that I think really hold this paper back, which I will detail below. However, I am willing to increase my score based on rebuttals to the following weaknesses and questions.\n- The key thing for me is that it seems Mechanic, in the appendix, performs *very* similarly to PACE (and well within the standard error, making it hard to make any claims about statistical significance - undermining the claim that Mechanic 'slightly underperforms in comparison to PACE'). While I am unfamiliar with this preexisting method, I think it would be good to have included this result into the main paper, with tests across more environments. I appreciate that the discussion mentions that Mechanic has worse guarantees, but empirical demonstration for why PACE is better is pretty necessary given that Mechanic has been included.\n- I also think there are missing comparisons to other methods referenced in the paper, which are designed to mitigate plasticity loss. While I appreciate these may not come under the 'hyperparameter-free' setting, comparison to other methods like plasticity injection would be beneficial for showing why PACE is worth using instead.\n- In places, the method section was quite confusing. Making the algorithms and subsequent text more concise would be useful as it took me quite a few reads to properly parse and understand.\n- The way the results are compared in the table is quite disingenuous, since I believe (particularly in Atari) the results are warped by performance in a very small number of the selected games which had a higher reward scale. This is made clear by the return curves, which show the same story as the table but in an arguably more demonstrative way. It could be worth either normalising scores between each environment, or doing away with the tables entirely.\n- As a small nitpick, it felt like lifelong RL is introduced far before its accompanying reference, making it sound like a new setting created for the paper rather than a term used in this context commonly.\n- The opening paragraph read in quite a different tone than the rest of the paper. It comes across in a slightly unserious manner that I think detracts from the rest of the paper. Having real-world motivation of the problem setting is very good, but this could have been written better (I appreciate this could be quite a personal opinion, and so did not incorporate it into my score).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you so much for your insightful review, feedback, and opportunity to improve our work!\n\n## PACE vs. Mechanic\n\nFirstly, we address your feedback by conducting a more rigorous evaluation of Mechanic across our benchmarks. Implementing your suggestion has substantiated our claim that PACE performs slightly better empirically. \n\n### Results\nWe conducted additional experiments, running 10 seeds for Mechanic on Procgen and Atari environments and 25 seeds on Control environments. We show the average normalized reward over the entire lifelong run for each environment for PPO when using both PACE and Mechanic in Supplementary PDF Figure 2.\n\nFollowing your suggestion, all results below are average normalized rewards (normalized scores between each environment, and for atari, normalized for each game; see more details on this in the Author Rebuttal). \n\n\n**Procgen Results for Mechanic vs PACE over 10 seeds**\n| Mechanic Starpilot | PACE Starpilot | Mechanic Dodgeball | PACE Dodgeball | Mechanic Chaser | PACE Chaser | Mechanic Fruitbot | PACE Fruibot  \n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| 0.49±0.0494     | **0.53±0.0468**     | 0.50±0.1267     |**0.51±0.1258**     |0.41±0.1232     |**0.52±0.0759**     |0.57±0.1356     |**0.65±0.1258**     |\n\n**Control Results for Mechanic vs PACE over 25 seeds**\n\n| Mechanic CartPole | PACE CartPole | Mechanic Acrobot | PACE Acrobot | Mechanic LunarLander | PACE LunarLander \n| -------- | -------- | -------- | -------- | -------- | -------- |\n| 0.31±0.1169     | **0.32±0.1092**     | 0.69±0.1150     |**0.72±0.1153**     |0.52±0.0292     |**0.54±0.0165** |\n\n**Atari Results for Mechanic vs PACE over 10 seeds**\n\n| Mechanic Atari (6) | PACE Atari (6) | Mechanic Atari (9) | PACE Atari (9) \n| -------- | -------- | -------- | -------- |\n| 0.45±0.04451     | **0.48±0.0389**     | 0.55±0.05630     |**0.58±0.0407**|\n\n\n\n### Conclusion\n**While Mechanic effectively combats plasticity loss and performs similarly to PACE, it consistently underperforms PACE by a slight margin across all settings.** *This result may align with the theoretical expectation that our scaling tuner provides better OCO performance guarantees.*\n\n### Insight\n\nWhile Mechanic is also a parameter-free optimizer, it has only been evaluated in the context of stationary supervised DL. By demonstrating that Mechanic performs well in a variety of non-stationary RL settings, we strengthen our claim that parameter-free optimization deserves more attention in the context of lifelong RL. We believe these results further validate the insights derived from PACE:\n- parameter-free OCO is all about better regularization and\n- better regularization mitigates loss of plasticity. \n\nWe are happy to include these additional Mechanic results in our main experiments.\n\n\n## Weight Decay Tuning\n\nWe appreciate your question regarding the tuning of weight decay with Adam, as explored by Lyle et al. 2023 for mitigating plasticity loss.\n\n* In our experiments, weight decay was set to 0.\n*  Following your suggestion, we conducted a HP sweep using PyTorch’s AdamW optimizer across three control environments with 15 seeds for each of the following weight decay values: `0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 15.0, 5.0, and 50.0`. \n*  Our findings show varying weight decay values do offer benefits (Supplementary PDF Figures 4a-c), but do not outperform PACE with no weight decay.\n*  Like L2 regularization strengths, the best weight decay values are environment-specific and sensitive to distribution shifts (see Supplementary PDF Figure 4d), complicating the presetting of these values without a prior hyperparameter sweep or knowledge of expected distribution shifts.\n\n\n## Plasticity Loss Mitigation Baselines\nIn appreciation of your suggestion, we have included plasticity injection (following the simple methodology laid out in Nikishin et al.) as a baseline in our control environments.\n\n* We inject plasticity at the start of every distribution shift.\n* While plasticity injection does help mitigate plasticity loss, it is outperformed by PACE across the 3 control settings (Supplementary PDF Figure 3b).\n\nAdditionally, we included another plasticity loss mitigation technique, layer normalization, as suggested by Reviewer eCfr.\n* While layer normalization improves the performance of baseline Adam, it is outperformed by PACE across the 3 control settings. \n* However, we also find that layer normalization with PACE improves over PACE without layer normalization (Supplementary PDF Figure 3b).\n\n\n## Minor: Reward Normalizizaton\nThank you for pointing out that our results can be made more clear by normalizing the scores in Table 1 (we will follow the normalization procedure in the Author Rebuttal).\n\nPlease let us know if you have any other concerns/comments, and we'll be glad to address those!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their detailed feedback. We appreciate their recognition of our parameter-free approach to mitigating plasticity loss in lifelong RL as novel (Reviewers Jqmp, Jcag), its evaluation across a broad set of lifelong RL benchmarks (Jcag, 346D, eCfR), the extra analysis in the appendix (Jcag, eCfR), and the theoretical motivations behind our empirical idea (Jcag, eCfR). We have addressed specific questions and concerns in our individual responses and have included a **Supplementary PDF** with experiments aimed at addressing feedback and questions from reviewers 346D and Jcag. We include an overview of such experiments and their setups below.

## Overview of Figures and Experiments
### PACE mitigates plasticity loss on LRL baselines in Procgen (Figure 1 in Supplementary PDF) (for Reviewer 346D):

* **Figure 1a** demonstrates that while IMPALA Adam, Online-EWC Adam, CLEAR Adam, and Modulating Masks Adam exhibit loss of plasticity in later levels, IMPALA PACE, Online-EWC PACE, CLEAR PACE, and Modulating Masks PACE effectively mitigate this plasticity loss and enhance reward performance over subsequent distribution shifts. STD fill is omitted here for clarity but is included in the bar plots in **Figures 1b-e**.
* **Figures 1b-e** present average normalized rewards over 5 seeds across 120M timesteps for each method using PACE and Adam: **1b** for Starpilot, **1c** for Dodgeball, **1d** for Chaser, and **1e** for Fruitbot.

### PACE performs slightly better than Mechanic on all settings (Figure 2 in Supplementary PDF) (for Reviewer Jcag)
* We compare the average normalized reward for Mechanic and PACE for Procgen and Atari environments (over 10 seeds) and on Gym Control environments (over 25 seeds). We show here that Mechanic consistently underperforms PACE by a slight margin across all settings.


### Layer norm and Plasticity Injection can also help (Figure 3 in the Supplementary PDF) (for Reviewers Jcag, eCfR)

* **Figure 3a** demonstrates that both layer norm and plasticity injection effectively mitigate plasticity loss in CartPole with baseline Adam PPO.
* **Figure 3b** shows that in all three control environments, PACE outperforms both layer norm Adam and plasticity injection Adam. Additionally, combining layer norm with PACE further boosts the agent’s performance compared to using PACE without layer norm.


### Tuning weight decay can help, but not as much as PACE (Figure 4 in Supplementary PDF) (for Reviewer Jcag)
* **Figures 4a-c** plot the average normalized reward of different weight decays with Adam (over 15 seeds over 3000 timesteps) compared to PACE. The bar plots indicate that while different weight decay values with Adam can help, they consistently underperform in comparison to PACE for all 3 control environments.
* **Figures 4d** plots the best performing weight decay with Adam over 10 distribution changes for all 3 of the control environments. We observe that weight decay values are highly sensitive to the specific environment and distribution shift.

## Average Normalized Reward

We calculate the average normalized reward as depicted in the bar plots (**Figures 1b-e; Figure 2; Figure 3b; Figures 4a-c**) by following these steps:
* Normalize rewards based on the maximum and minimum values for each environment, irrespective of the method used.
* Compute the average reward for each timestep across all seeds.
* Average these values across all timesteps.
* *For Atari environments, normalization is also performed within each subtask due to varying reward scales across games.*



## Procgen Experimental Setup (for Reviewer 346D)

To align with our online lifelong RL setting described in the main paper’s Procgen experiments, we have made the following modifications from the continual RL setup used by Ben-Iwhiwhu et al., 2023, and Powers et al., 2022:

* We focus on individual games (e.g., Starpilot, Dodgeball, Chaser, Fruitbot) rather than cycling through multiple games for evaluation.
* Tasks are limited to a single level, preventing training and generalization across 200 levels.
* We have adjusted the Procgen distribution shift mode in the code from easy to hard.
* We switch the level every 8M timesteps.

Meta-review: This paper introduces a parameter-free optimizer called PACE, using ideas from online convex optimization to target plasticity loss in lifelong RL. The novel approach is supported by strong theoretical motivation (although no theoretical guarantees) and extensive empirical evaluations across various environments. There were some initial concerns regarding clarity/presentation, the absence of certain baselines, and similar performance to some other methods (e.g., Mechanic), but these were mostly addressed in the rebuttal through the authors providing additional experiments, including an ablation study and more detailed statistical comparisons. There still seems to be only a slight improvement over Mechanic, but the reviewers in general agree that the paper’s contributions are valuable. The authors are advised to incorporate the reviewers’ suggestions to improve clarity/presentation, the new results, and issues raised in the discussion into the final version of their paper.
Decision: Accept (poster)