Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning

Authors: aneesh  muppidi
zhiyu  zhang
heng  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 00:23:14

Abstract: A key challenge in lifelong reinforcement learning (RL) is the loss of plasticity, where previous learning progress hinders an agent's adaptation to new tasks. While regularization and resetting can help, they require precise hyperparameter selection at the outset and environment-dependent adjustments. Building on the principled theory of online convex optimization, we present a parameter-free optimizer for lifelong RL, called TRAC, which requires no tuning or prior knowledge about the distribution shifts. Extensive experiments on Procgen, Atari, and Gym Control environments show that TRAC works surprisingly well—mitigating loss of plasticity and rapidly adapting to challenging distribution shifts—despite the underlying optimization problem being nonconvex and nonstationary.

Main paper: 
 lifelong rl: 
 As a sequential decision making framework, reinforcement learning (RL) is commonly framed as a Markov Decision Process (MDP) defined by the state space S, the action space A, the transition dynamics P (s t+1 |s t , a t ), and the reward function R(s t , a t , s t+1 ). In the t-th round, starting from a state s t ∈ S, the learning agent needs to choose an action a t ∈ A without knowing P and R.Then, the environment samples a new state s t+1 ∼ P (•|s t , a t ), and the agent receives a reward r t = R(s t , a t , s t+1 ). There are standard MDP objectives driven by theoretical tractability, but from a practical perspective, we measure the agent's performance by its cumulative reward T t=1 r t . The standard setting above concerns a stationary MDP. Motivated by the prevalence of distribution shifts in practice, the present work studies a nonstationary variant called lifelong RL, where the transition dynamics P t and the reward function R t can vary over time. Certainly, one should not expect any meaningful "learning" against arbitrary unstructured nonstationarity. Therefore, we implicitly assume P t and R t to be piecewise constant over time, and each piece is called a taskjust like our example of Spot in the introduction. The main challenge here is to transfer previous learning progress to new tasks. This is reasonable when tasks are similar, but we also want to reduce the degradation when tasks turn out to be very different.Lifelong RL as online optimization Deep RL approaches, including PPO (Schulman et al., 2017) and others, crucially utilize the idea of policy parameterization. Specifically, a policy refers to the distribution of the agent's action a t (conditioned on the historical observations), and we use θ t ∈ R d to denote the parameterizing weight vector. After sampling a t and receiving new observations, the agent could define a loss function J t (θ) that characterizes the "hypothetical performance" of each weight θ ∈ R d . Then, by computing the policy gradient g t = ∇J t (θ t ), one could apply a first order optimization algorithm3 OPT to obtain the updated weight, θ t+1 = OPT(θ t , g t ).For the rest of this paper, we will work with such an abstraction. The feedback of the environment is treated as a policy gradient oracle G, which maps the time t and the current weight θ t into a policy gradient g t = G(t, θ t ). Our goal is to design an optimizer OPT well suited for lifelong RL.Lifelong vs. Continual In the RL literature, the use of "lifelong" and "continual" varies significantly across studies, which may lead to confusion. Abel et al. (2024) characterized continual reinforcement learning (CRL) as a never-ending learning process. However, much of the literature cited under CRL, such as (Abbas et al., 2023; Ahn et al., 2024) , primarily focuses on the problem of backward transfer (avoiding catastrophic forgetting). Various policy-based architectures, such as those proposed by Rolnick et al. (2019) ; Schwarz et al. (2018) ; Nath et al. (2023) , focus on tackling this issue. Conversely, the present work addresses the problem of forward transfer, which refers to the rapid adaptation to new tasks. Because of this we use "lifelong" rather than "continual" in our exposition, similar to (Thrun, 1996; Abel et al., 2018b; Julian et al., 2020) .Weight ParameterizationBase Updateθ refTRAC decides strength of regularization Inspired by (Cutkosky et al., 2023) , we study lifelong RL by exploiting its connection to Online Convex Optimization (OCO; Zinkevich, 2003) . The latter is a classical theoretical problem in online learning, and much effort has been devoted to designing parameter-free algorithms that require minimum tuning or prior knowledge (Streeter & Mcmahan, 2012; McMahan & Orabona, 2014; Orabona & Pál, 2016; Foster et al., 2017; Cutkosky & Orabona, 2018; Mhammedi & Koolen, 2020; Chen et al., 2021; Jacobsen & Cutkosky, 2022) . The surprising observation of Cutkosky et al. (2023) is that several algorithmic ideas closely tied to the convexity of OCO can actually improve the nonconvex deep learning training, suggesting certain notions of "near convexity" on its loss landscape. We find that lifelong RL (which is both nonconvex and nonstationary in terms of optimization) exhibits a similar behavior, therefore a particularly strong algorithm (named TRAC) can be obtained from principled results in parameter-free OCO. Let us start from the background.Basics of (parameter-free) OCO As a standalone theoretical topic, OCO concerns a sequential optimization problem where the convex loss function l t can vary arbitrarily over time. In the t-th iteration, the optimization algorithm picks an iterate x t and then observes a gradient g t = ∇l t (x t ). Motivated by the pursuit of "convergence" in optimization, the standard objective is to guarantee low (i.e., sublinear in T ) static regret, defined asRegret T (l 1:T , u) := T t=1 l t (x t ) - T t=1 l t (u),where T is the total number of rounds, and u is a comparator that the algorithm does not know beforehand. In other words, the goal is to make Regret T (l 1:T , u) small for all possible loss sequence l 1:T and comparator u. Note that for nonstationary OCO problems analogous to lifelong RL, it is better to consider a different objective called the discounted regret. Algorithms there mostly follow the same principle as in the stationary setting, just wrapped by loss rescaling (Zhang et al., 2024a) .For minimizing static regret, classical minimax algorithms like gradient descent (Zinkevich, 2003) would assume a small uncertainty set U at the beginning. Then, by setting the hyperparameter (such as the learning rate) according to U, it is possible to guarantee sublinear worst case regret, maxEQUATIONIn contrast, parameter-free algorithms use very different strategies4 to bound Regret T (l 1:T , u) directly (without taking the maximum) by a function of both l 1:T and u. The resulting bound is more refined than Eq.( 1) (Orabona, 2023, Chapter 9) , and crucially, since there is no need to pick an uncertainty set U, much less hyperparameter tuning is needed. This is where its name comes from.TRAC for Lifelong RL: In lifelong RL, a key issue is the excessive drifting of weights θ t , which can detrimentally affect adapting to new tasks. To address this, TRAC enforces proximity to a well-chosen reference point θ ref , providing a principled solution derived from a decade of research in parameter-free OCO. Unlike traditional methods such as L 2 regularization or resetting, TRAC avoids hyperparameter tuning, utilizing the properties of OCO to maintain weight stability and manage the drift effectively.The core of TRAC, similar to other parameter-free optimizers, incorporates three techniques:• Direction-Magnitude Decomposition: Inspired by Cutkosky & Orabona (2018) , this technique employs a carefully designed one-dimensional algorithm, the "parameter-free tuner," atop a base optimizer. This setup acts as a data-dependent regularizer, controlling the extent to which the iterates deviate from their initialization, thereby minimizing loss of plasticity, which is crucial given the high plasticity at the initial policy parameterization (Abbas et al., 2023) . • Erfi Potential Function: Building on the previous concept, the tuner utilizes the Erfi potential function, as developed by Zhang et al. (2024a) . This function is crafted to effectively balance the distance of the iterates from both the origin and the empirical optimum. It manages the update magnitude by focusing on the gradient projection along the direction θ t -θ ref .• Additive Aggregation: The tuner above necessitates discounting. Thus, we employ Additive Aggregation by Cutkosky (2019) . This approach enables the combination of multiple parameter-free OCO algorithms, each with different discount factors, to approximate the performance of the best-performing algorithm. Importantly, it facilitates the automatic selection of the optimal discount factor during training.These three components crucially work together to guarantee good regret bounds in the convex setting and are the minimum requirement for any reasonable parameter-free optimizer.Without going deep into the theory, here is an overview of the important ideas (also see Figure 2 for a visualization).• First, TRAC is a meta-algorithm that operates on top of a "default" optimizer BASE. It can simply be gradient descent with a constant learning rate, or ADAM (Kingma & Ba, 2014) as in our experiments.Applying BASE alone would be equivalent to enforcing the scaling parameter S t+1 ≡ 1 in TRAC, but this would suffer from the drifting of θ Base t+1 (and thus, the weight θ t+1 ). Obtain the t-th policy gradient g t = G(t, θ t ) ∈ R d .Send g t to BASE as its t-th input, and get its output θ Base t+1 ∈ R d .For all j ∈ [1 : n], send ⟨g t , θ t -θ ref ⟩ to A j as its t-th input, and get its output s t+1,j ∈ R.Define the scaling parameter S t+1 = n j=1 s t+1,j .8:Update the weight of the policy,θ t+1 = θ ref + θ Base t+1 -θ ref S t+1. 9: end for Algorithm 2 1D Discounted Tuner of TRAC.1: Input: Discount factor β ∈ (0, 1]; small value ε > 0 (default: 10 -8 ). 2: Initialize: The running variance v 0 = 0; the running (negative) sum σ 0 = 0. 3: for t = 1, 2, . . . doObtain the t-th input h t .Let v t = β 2 v t-1 + h 2 t , and σ t = βσ t-1 -h t .6:Select the t-th outputs t+1 = ε erfi(1/ √ 2) erfi σ t √ 2v t + ε ,where erfi is the imaginary error function queried from standard software packages. 7: end for • To fix this issue, TRAC uses the tuner (Algorithm 2) to select the scaling parameter S t+1 , making it data-dependent. Typically S t+1 is within [0, 1] (see Figure 17 to 19), therefore essentially, we define the updated weight θ t+1 as a convex combination of the BASE's weight θ Base t and the reference point θ ref ,θ t+1 = S t+1 • θ Base t+1 + (1 -S t+1 )θ ref .This brings the weight closer to θ ref , which is known to be "safe" (i.e., not overfitting any particular lifelong RL task), although possibly conservative.• To inject the right amount of conservatism without hyperparameter tuning, the tuner (Algorithm 2) applies an unusual decision rule based on the erfi function. Theoretically, this is known to be optimal in an idealized variant of OCO (Zhang et al., 2022 (Zhang et al., , 2024b)) , but removing the idealized assumptions requires a tiny bit of extra conservatism, which is challenging (and not necessarily practical). Focusing on the lifelong RL problem that considerably deviates from OCO, we simply apply the erfi decision rule as is. This is loosely motivated by deep learning training dynamics, e.g., (Cohen et al., 2020; Ahn et al., 2023; Andriushchenko et al., 2023) , where an aggressive optimizer is often observed to be better.• Finally, the tuner requires a discount factor β. This crucially controls the strength of regularization (elaborated next), but also introduces a hyperparameter tuning problem. Following (Cutkosky, 2019) , we aggregate tuners with different β (on a log-scaled grid) by simply summing up their outputs. This is justified by the adaptivity of the tuner itself: in OCO, if we add a parameter-free algorithm A 1 to any other algorithm A 2 that already works well, then A 1 can automatically identify this and "tune down" its aggressiveness, such that A 1 + A 2 still performs as well as A 2 .Connection to regularization Despite its nested structure, TRAC can actually be seen as a parameter-free refinement of L 2 regularization (Kumar et al., 2023) . To concretely explain this intuition, let us consider the following two optimization dynamics.• First, suppose we run gradient descent with learning rate η, on the policy gradient sequence {g t } with the L 2 regularizer λ 2 ∥θ -θ ref ∥2 . Quantitatively, it means that starting from the t-th weight θ t ,θ t+1 = θ t -η [g t + λ (θ t -θ ref )] , =⇒ θ t+1 -θ ref = (1 -λη) (θ t -θ ref ) -ηg t . (2)That is, the updated weight θ t+1 is determined by a (1 -λη)-discounting with respect to the reference point θ ref , followed by a gradient step -ηg t . • Alternatively, consider applying the following simplification of TRAC on the same policy gradient sequence {g t }: (i) BASE is still gradient descent with learning rate η; (ii) there is just one discount factor β; and (iii) the one-dimensional tuner (Algorithm 2) is replaced by the β-discounted gradient descent with learning rate α, i.e., S t+1 = βS t -αh t . In this case, we haveθ t+1 -θ ref = S t+1 θ Base t+1 -θ ref = (βS t -αh t ) θ Base t -θ ref -ηg t = β -αS -1 t h t (θ t -θ ref ) -ηS t+1 g t . (mildly assuming S t ̸ = 0) Notice that S t is a β-discounted sum of αh 1 , . . . , αh t-1 , thus in the typical situation of β ≈ 1 one might expect αh t ≪ |S t |.Then, the resulting update of θ t+1 is similar to Eq.( 2), with quantitative changes on the "effective discounting" 1 -λη → β, and the "effective learning rate" η → ηS t+1 .The main message here is that under a simplified setting, TRAC is almost equivalent to L 2 regularization. The latter requires choosing the hyperparameters λ and η, and similarly, the above simplified TRAC requires choosing β and η. Going beyond this simplification, the actual TRAC removes the tuning of β using aggregation, and the tuning of η using the erfi decision rule.On the hyperparameters Although TRAC is called "parameter-free", it still needs the β-grid, the constant ε and the algorithm BASE as inputs. The idea is that TRAC is particularly insensitive to such choices, as supported by the OCO theory. As the result, the generic default values recommended by Cutkosky et al. (2023) are sufficient in practice. We note that those are proposed for training supervised deep learning models, thus should be agnostic to the lifelong RL applications we consider. 
experiment: 
 Does TRAC experience the common pitfalls of loss of plasticity? Does it rapidly adapt to distribution shifts? To answer these questions, we test TRAC in empirical RL benchmarks such as vision-based games and physics-based control environments in lifelong settings (Figure 3 ). Specifically, we instantiate PPO with two different optimizers: ADAM with constant learning rate for baseline comparison, and TRAC for our proposed method (with exactly the same ADAM as the input BASE). We also test ADAM PPO with concatenated ReLU activations (CReLU; Shang et al., 2016) , previously shown to mitigate loss of plasticity in certain deep RL settings (Abbas et al., 2023) . Our numerical results are summarized in Table 1 . Across every lifelong RL setting, we observe substantial improvements in the cumulative episode reward by using TRAC PPO compared to ADAM PPO or CReLU. Below are the details, with more in the Appendix. Procgen We first evaluate on OpenAI Procgen, a suite of 16 procedurally generated game environments (Cobbe et al., 2020) . We introduce distribution shifts by sampling a new procedurally generated level of the current game every 2 million time steps, treating each level as a distinct task. There is a steady loss of plasticity in agents using ADAM PPO and CReLU, characterized by their inability to maintain performance through succesive Procgen levels. In contrast, TRAC avoids this loss of plasticity, quickly achieving high performance with each new task.We evaluate game environments including StarPilot, Dodgeball, Fruitbot, and Chaser. In all of these environments, we observe in Figure 4 that both ADAM PPO and CReLU encounter a continually degrading loss of plasticity as these distribution shifts are introduced. In contrast, TRAC PPO avoids this loss of plasticity, which contributes to its rapid reward increase when adapting to new levels.In the cumulative reward across all the Procgen levels, TRAC PPO reveals normalized average improvements of 3,212.42% and 120.88% over ADAM PPO and CReLU respectively (see Table 1 ). For later levels, in all games, TRAC PPO's reward does not decline as sharply as the baselines, potentially indicating positive transfer of skills from one level to the next.One key advantage of TRAC is that it functions as an optimizer, making it orthogonal to various policy methods such as PPO, as well as other baselines like Online EWC (Schwarz et al., 2018) , IMPALA (Espeholt et al., 2018) , Modulating Masks (Nath et al., 2023) , and CLEAR (Rolnick et al., 2019) . In Appendix C, we evaluate these methods using both TRAC and ADAM on the Procgen setup. We find that in every environment, TRAC improves the performance of these algorithms.Atari The Arcade Learning Environment (ALE) Atari 2600 benchmark is a collection of classic arcade games designed to assess reinforcement learning agents' performance across a range of diverse gaming scenarios (Bellemare et al., 2013) . We introduce distribution shifts by switching to a new Atari game every 4 million timesteps, where each game switch introduces a new task. This benchmark is more challenging compared to OpenAI Procgen: it requires the agent to handle distribution shifts in both the input (state) and the target (reward).In this experiment, we assessed two online settings distinguished by games with action spaces of 6 and 9. From Figure 5 , both ADAM PPO and CReLU sometimes failed to learn in certain games. In contrast, TRAC PPO shows a substantial increase in reward over different games compared to the baselines. For example, during the first 12 million steps (3 games) in Atari 6, TRAC PPO not only achieves a significantly higher mean reward but also demonstrates rapid reward increase. Over both experiment settings, TRAC PPO shows an average normalized improvement of 329.73% over ADAM PPO and 68.71% over CReLU (Table 1 ). In rare instances, such as the last 2 million steps of Atari 6, CReLU performs comparably to TRAC PPO. This observation aligns with findings from (Abbas et al., 2023) , which noted that CReLU tends to avoid plasticity loss in continual Atari setups. Here (Figure 6 ), we notice a peculiar behavior after introducing the first distribution shift in both ADAM PPO and CReLU: policy collapse. We describe this as an extreme form of loss of plasticity. Surprisingly, TRAC PPO remains resistant to these extreme distribution shifts. As we see in the Acrobot experiment, TRAC PPO shows minimal to no policy damage after the first few distribution shifts, whereas ADAM PPO and CReLU are unable to recover a policy at all. We investigate if TRAC's behavior here indicates positive transfer in Appendix A. Across the three Gym Control environments, TRAC PPO shows an average normalized improvement of 204.18% over ADAM PPO and 1044.24% over CReLU (Table 1 ). 
discussion: 
 Related work Combating loss of plasticity has been studied extensively in lifelong RL. A typical challenge for existing solutions is the tuning of their hyperparameters, which requires prior knowledge on the nature of the distribution shift, e.g., (Asadi et al., 2024; Nath et al., 2023; Nikishin et al., 2024; Sokar et al., 2023; Mesbahi et al., 2024 ). An architectural modification called CReLU is studied in (Abbas et al., 2023) , but our experiments suggest that its benefit might be specific to the Atari setup. Besides, Abel et al. (2018a,b) presented a theoretical analysis of skill transfer in lifelong RL, based on value iteration. Moreover, related contributions in nonstationary RL, where reward and state transition functions also change unpredictably, are limited to theoretical sequential decision-making settings with a focus on establishing complexity bounds (Roy et al., 2019; Cheung et al., 2020; Wei & Luo, 2021; Mao et al., 2020) .Our algorithm TRAC builds on a long line of works on parameter-free OCO (see Section 3). To our knowledge, the only existing work applying parameter-free OCO to RL is (Jacobsen & Chan, 2021) , which focuses on estimating the value function (i.e., policy evaluation). Our scope is different, focusing on empirical RL in lifelong problems by exploring the key connection between parameterfree OCO and regularization.Particularly, we are inspired by the MECHANIC algorithm from (Cutkosky et al., 2023) , which goes beyond the traditional convex setting of parameter-free OCO to handle stationary deep learning optimization tasks. Lifelong reinforcement learning, however, introduces a layer of complexity with its inherent nonstationarity. Furthermore, compared to MECHANIC, TRAC improves the scale tuner there (which is based on the coin-betting framework; Orabona & Pál, 2016) by the erfi algorithm that enjoys a better OCO performance guarantee. As an ablation study, we empirically compare TRAC and MECHANIC in the Appendix G (Table 3 ). We find that TRAC is slightly better, but both algorithms can mitigate the loss of plasticity, suggesting the effectiveness of the general "parameter-free" principle in lifelong RL.In our experiments, we observe that TRAC's reward decline due to distribution shifts is less severe than that of baseline methods. These results may suggest TRAC facilitates positive transfer between related tasks. To investigate this further, we compared TRAC to a privileged weight-resetting approach, where the network's parameters are reset for each new task, in the Gym Control environments (see Appendix A). Our results show that TRAC maintains higher rewards during tasks than privileged weight-resetting and avoids declining to the same low reward levels as privileged weight-resetting at the start of a new task (Figure 8 ).On the choice of θ ref In general, the reference point θ ref should be good or "safe" for TRAC to perform effectively. One might presume that achieving this requires "warmstarting", or pre-training using the underlying BASE optimizer. While our experiments validate that such warmstarting is indeed beneficial (Appendix B), our main experiments show that even a random initialization of the policy's weight serves as a good enough θ ref , even when tasks are similar (Figure 4 ). (2023) , who suggested that persistent gradient steps away from a random initialization can deactivate ReLU activations, leading to activation collapse and loss of plasticity in neural networks. Our results also support Kumar et al. (2023) 's argument that maintaining some weights close to their initial values not only prevents dead ReLU units but also allows quick adaptation to new distribution shifts.Tuning L 2 regularization The success of TRAC suggests that an adaptive form of regularizationanchoring to the reference point θ ref -may suffice to counteract both mild and extreme forms of loss of plasticity. From this angle, we further elaborate the limitation of the L 2 regularization approach considered in (Kumar et al., 2023) . It requires selecting a regularization strength parameter λ through cross-validation, which is incompatible with the one-shot nature of lifelong learning settings. Furthermore, it is nontrivial to select the search grid: for example, we tried the λ-grid suggested by (Kumar et al., 2023) , and there is no effective λ value within the grid for the lifelong RL environments we consider. All the values are too small. Figure 7 : For each Gym Control environment and the initial ten tasks, we identified the best λ, which is the regularization strength that maximizes reward for each task's specific distribution shift. We also determined the best overall (well-tuned) λ for each environment. The results demonstrate that each environment and each task's distribution shift is sensitive to different λ and that TRAC PPO performs competitively with each environment's well-tuned λ.Continuing this reasoning, we conduct a hyperparameter search for λ, over various larger values [0.2, 0.8, 1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50] . Given the expense of such experiments, only the more sample-efficient control environments are considered. We discover that each environment and task responds uniquely to these regularization strengths (see bar plot of λ values in Figure 7 ). This highlights the challenges of tuning λ in a lifelong learning context, where adjusting for each environment, let alone each distribution shift, would require extensive pre-experimental analysis.In contrast, TRAC offers a parameter-free solution that adapts dynamically with the data in an online manner. The scaling output of TRAC adjusts autonomously to the ongoing conditions, consistently competing with well-tuned λ values in the various environments, as demonstrated in the reward plots for CartPole, Acrobot, and LunarLander (Figure 7 ).TRAC compared to other plasticity methods Both layer normalization and plasticity injection Nikishin et al. (2024) ; Lyle et al. (2023) have been shown to combat plasticity loss. For instance, Appendix E Figure 15 demonstrates that both layer normalization and plasticity injection are effective at reducing plasticity loss when applied to the CartPole environment using ADAM as a baseline optimizer. We implemented plasticity injection following the methodology laid out by Nikishin et al. (2024) , where plasticity is injected at the start of every distribution shift. While this approach does help in reducing the decline in performance due to plasticity loss, our results indicate that it is consistently outperformed by TRAC across all three control environments-CartPole, Acrobot, and LunarLander. Moreover, while layer normalization improves ADAM's performance, it too is outperformed by TRAC across the same control settings (Figure 15 ). Notably, combining layer normalization with TRAC resulted in the best performance gains.Near convexity of lifelong RL Our results demonstrate the rapid adaptation of TRAC, in lifelong RL problems with complicated function approximation. From the perspective of optimization, the latter requires tackling both nonconvexity and nonstationarity, which is typically regarded intractable in theory. Perhaps surprisingly, when approaching this complex problem using the theoretical insights from OCO, we observe compelling results. This suggests a certain "hidden convexity" in this problem, which could be an exciting direction for both theoretical and empirical research (e.g., policy gradient methods provably converge to global optimizers in linear quadratic control (Hu et al., 2023) ).Limitations While TRAC offers robust adaptability in nonstationary environments, it can exhibit suboptimal performance at the outset. In the early stages of deployment, TRAC might underperform compared to the baseline optimizer. We address this by proposing a warmstarting solution detailed in Appendix B, which helps increase the initial performance gap. 
conclusion: 
 In this work, we introduced TRAC, a parameter-free optimizer for lifelong RL that leverages the principles of OCO. Our approach dynamically refines regularization in a data-dependent manner, eliminating the need for hyperparameter tuning. Through extensive experimentation in Procgen, Atari, and Gym Control environments, we demonstrated that TRAC effectively mitigates loss of plasticity and rapidly adapts to new distribution shifts, where baseline methods fail. TRAC's success leads to a compelling takeaway: empirical lifelong RL scenarios may exhibit more convex properties than previously appreciated, and might inherently benefit from parameter-free OCO approaches. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The full experimental details can be found in H and in the code submission in our supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Our reward plots include standard deviation error bars around the curves. We detail this in the experimental setup section of our Appendix (H).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] .Justification: Not applicable.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] .Justification: We dot use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
: 
 . There is a steady decline in reward with each distribution shift.Spot, the agile robot dog, has been learning to walk confidently across soft, lush grass. But when Spot moves from the grassy field to a gravel surface, the small stones shift beneath her feet, causing her to stumble. When Spot tries to walk across a sandy beach or on ice, the challenges multiply, and her once-steady walk becomes erratic. Spot wants to adjust quickly to these new terrains, but the patterns she learned on grass are not suited to gravel, sand, or ice. Furthermore, she never knows when the terrain will change again and how different it will be, therefore must continually plan for the unknown while avoiding reliance on outdated experiences.Spot's struggle exemplifies a well-known and extensively studied challenge in real-world decision making: lifelong reinforcement learning (lifelong RL) Abel et al. (2024) ; Nath et al. (2023) ; Mendez et al. (2020) ; Xie & Finn (2022) . In lifelong RL, the learning agent must continually acquire new knowledge to adapt to the nonstationarity of the environment. At first glance, there appears to be an obvious solution: given a policy gradient oracle, the agent could just keep running gradient descent nonstop. However, recent experiments have demonstrated an intriguing behavior called loss of plasticity (Dohare et al., 2021; Lyle et al., 2022; Abbas et al., 2023; Sokar et al., 2023) : despite persistent gradient steps, such an agent can gradually lose its responsiveness to incoming observations. There are even extreme cases of loss of plasticity (known as negative transfer or primacy bias), where prior learning can significantly hamper the performance in new tasks (Nikishin et al., 2022; Ahn et al., 2024) ; see Figure 1 for an example. All these suggest that the problem is more involved than one might think.From the optimization perspective, the above issues might be attributed to the lack of stability under gradient descent. That is, the weights of the agent's parameterized policy can drift far away from the origin (or a good initialization), leading to a variety of undesirable behaviors. 1 Fitting this narrative, it has been shown that simply adding a L 2 regularizer to the optimization objective (Kumar et al., 2023) or periodically resetting the weights (Dohare et al., 2021; Asadi et al., 2024; Sokar et al., 2023; Ahn et al., 2024) can help mitigate the problem. However, a particularly important limitation is their use of hyperparameters, such as the magnitude of the regularizer and the resetting frequency 2 . Good performance hinges on the suitable environment-dependent hyperparameter, but how can one confidently choose that before interacting with the environment? The classical cross-validation approach would violate the one-shot nature of lifelong RL (and online learning in general; see Chapter 1 of Orabona, 2023) , since it is impossible to experience the same environment multiple times. This leads to the contributions of the present work.Contribution The present work addresses the key challenges in lifelong RL using the principled theory of Online Convex Optimization (OCO). Specifically, our contributions are two fold.• Algorithm: TRAC Building on a series of results in OCO (Cutkosky & Orabona, 2018; Cutkosky, 2019; Cutkosky et al., 2023; Zhang et al., 2024b) , we propose a (hyper)-parameter-free optimizer for lifelong RL, called TRAC (AdapTive RegularizAtion in Continual environments). Intuitively, the idea is a refinement of regularization: instead of manually selecting the magnitude of regularization beforehand, TRAC chooses that in an online, data-dependent manner. From the perspective of OCO theory, TRAC is insensitive to its own hyperparameter, which means that no hyperparameter tuning is necessary in practice. Furthermore, as an optimization approach to lifelong RL, TRAC is compatible with any policy parameterization method. • Experiment Using Proximal Policy Optimization (PPO) (Schulman et al., 2017) , we conduct comprehensive experiments on the instantiation of TRAC called TRAC PPO. A diverse range of lifelong RL environments are tested (based on Procgen, Atari, and Gym Control), with considerably larger scale than prior works. In settings where existing approaches (Abbas et al., 2023; Kumar et al., 2023; Nath et al., 2023) struggle, we find that TRAC PPO mitigates mild and extreme loss of plasticity; and rapidly adapts to new tasks when distribution shifts are introduced. Such findings might be surprising: the theoretical advantage of TRAC is motivated by the convexity in OCO, but lifelong RL is both nonconvex and nonstationary in terms of optimization.Organization Section 2 surveys the basics of lifelong RL. Section 3 introduces our parameter-free algorithm TRAC, and experiments are presented in Section 4. We defer the discussion of related works and results to Section 5. Finally, Section 6 concludes the paper. 
 
 Appendix: In our theoretical framework, we hypothesize that a robust parameter initialization, denoted as θ ref , could enhance the performance of our models, suggesting that empirical implementations might benefit from initializing parameters using a base optimizer such as ADAM prior to deploying TRAC. Contrary to this assumption, our experimental results detailed in Section 4 reveal that warmstarting is not essential for TRAC's success. Below, we examine the performance of ADAM PPO and TRAC PPO when warmstarted.Both TRAC PPO and ADAM PPO were warmstarted using ADAM for the initial 150,000 steps in all games for the Atari and Procgen environments, and for the first 30 steps in the Gym Control experiments. As seen in Figure 9 , in games like Starpilot, Fruitbot, and Dodgeball, TRAC PPO surpasses ADAM PPO in the first level/task of the online setup, with its performance closely matching that of ADAM PPO in Chaser. Importantly, TRAC PPO continues to avoid the loss of plasticity encountered by ADAM PPO, even when both are warmstarted. This makes sense since all of the distributions share some foundational game dynamics; the initial learning phases likely explore these dynamics, so leveraging a good parameter initialization to regularize in this early region can be beneficial for TRAC-we observe that forward transfer occurs somewhat in later level distribution shifts as the reward does not drop back to zero where it initially started from.Our findings indicate that warmstarting does not confer a significant advantage in the Atari games. This makes sense because a parameter initialization that is good in one game setting is likely a random parameterization for another setting, which is equivalent to the setup without warmstarting where TRAC regularizes towards a random parameter initialization. In the Gym Control experiments although warmstarted TRAC PPO manages to avoid the extreme plasticity loss and policy collapse seen in warmstarted ADAM PPO, it does not perform as well as non-warmstarted TRAC PPO. This result underscores that the efficacy of warmstarting is environment-specific and highlights the challenge in predicting when ADAM PPO may achieve a parameter initialization that is advantageous for TRAC PPO to regularize towards.From an overall perspective, warmstarting TRAC PPO in every setting still shows substantial improvement over ADAM PPO (Table 2 ). While PPO is a widely used policy gradient method in reinforcement learning, it is not the only approach applicable to lifelong RL. Other continual RL methods, such as IMPALA (Espeholt et al., 2018) , Online EWC (Schwarz et al., 2018) , CLEAR (Rolnick et al., 2019) , and Modulating Masks (Nath et al., 2023) , are designed to address challenges like catastrophic forgetting in dynamic, nonstationary environments. We incorporated these algorithm implementations adapted from the code from Nath et al. (2023) into our experiments to offer a more comprehensive evaluation. These methods vary in their mechanisms for maintaining task performance over time but may still suffer from plasticity loss in later stages of training. Mitigating plasticity loss across policy methods: Figure 12 demonstrates the performance of various continual RL methods when paired with ADAM and TRAC optimizers. The results indicate that when using ADAM, methods like IMPALA, Online EWC, CLEAR, and Modulating Masks exhibit a noticeable decline in performance over time due to plasticity loss, particularly in later levels of the Procgen environments. In contrast, pairing these methods with TRAC instead of ADAM leads to significant improvements, mitigating plasticity loss and enhancing reward performance across subsequent distribution shifts.To quantify these improvements, Figures 13, 12 present the average normalized rewards over five seeds and 120M timesteps for each method across four different Procgen environments: Starpilot, Dodgeball, Chaser, and Fruitbot. Across all environments, methods that use TRAC outperform their Adam-based counterparts, consistently maintaining higher rewards over time.On average, across the Procgen environments, TRAC led to performance improvements over ADAM by the following margins: 21.83% for IMPALA, 15.86% for Online EWC, 14.41% for CLEAR, and 10.14% for Modulating Masks.General Applicability of TRAC: It is important to highlight that TRAC is orthogonal to the learning or policy algorithms themselves. It can be seamlessly integrated into various reinforcement learning architectures by simply replacing their optimizer (e.g., ADAM or RMSPROP). Our results demonstrate that TRAC enhances performance across different algorithms and environments, consistently outperforming ADAM in mitigating plasticity loss.One method to introduce distribution changes in reinforcement learning environments is by altering the dynamics Mendez et al. (2020) , such as adjusting the gravity in the CartPole environment. In this set of experiments, we manipulate the gravity by a magnitude of ten, randomly adding noise for one distribution shift, and then inversely, dividing by ten and adding random noise for the next shift. This process continues throughout the experiment. Fruitbot. Each method (IMPALA, Online EWC, CLEAR, and Modulating Masks) is evaluated using both Adam and TRAC. TRAC consistently outperforms ADAM across all methods and environments, with improvements ranging from 10% to 21%.P A L A A d a m IM P A L A T R A C C L E A R A d a m C L E A R T R A C O n li n e E W C A d a m O n li n e E W C T R A C M a s k A d a m M a s k T R A C 0.0 0.Our observations suggest that ADAM PPO is robust to such dynamics-based distribution shifts, as shown in Figure 14 . This indicates that while ADAM PPO implicitly models the dynamics of the environment well-where changes in dynamics minimally impact performance-it struggles more with adapting to out-of-distribution observations such as seen in the main experiments (Figure 6 ) and in the warmstarting experiments (Figure 11 ).To evaluate TRAC alongside other methods that aim to mitigate plasticity loss, we compare it against LayerNorm (Lyle et al., 2023) , Plasticity Injection (Nikishin et al., 2024) , and tuning weight decay (Lyle et al., 2024) .As discussed in Section 5, we confirm that both layer normalization and plasticity injection (applied at the start of every distribution shift) (Nikishin et al., 2024; Lyle et al., 2023) are effective in reducing plasticity loss (Figure 15 ). While these methods help slow the decline in performance due to plasticity loss, TRAC consistently outperforms them across the three Gym Control environments. Importantly, because TRAC is an optimizer, it can be combined with layer normalization, and doing so resulted in the best performance gains in our Control setups. Tuning weight decay: In addition to LayerNorm and Plasticity Injection, we also evaluated the effects of tuning weight decay using PyTorch's AdamW optimizer. We conducted a hyperparameter sweep across three control environments with 15 seeds for each of the following weight decay values: 0.0001, 0.001, 0.01, 0.1, 1.0, 5.0, 10.0, 15.0, and 50.0. Figure 16 presents the average normalized reward for each weight decay value over 15 seeds and 3000 timesteps, compared to TRAC.The results indicate that while tuning weight decay with Adam does provide some benefit, these values consistently underperform in comparison to TRAC across all three control environments. Figure 16 plots the performance of the best-performing weight decay value with Adam over 10 distribution shifts in the control environments. We observe that weight decay values are highly sensitive to the specific environment and the nature of the distribution shift.Interestingly, in our initial experiments, we set the weight decay to zero, yet TRAC still outperformed Adam with various weight decay values. This suggests that while weight decay can mitigate plasticity loss to some extent, it does not match the overall effectiveness of TRAC. As discussed in the algorithm section (see Section 3), TRAC operates as a meta-algorithm on top of a standard optimizer, denoted as BASE. The crucial component of TRAC involves the dynamic adjustment of the scaling parameter S t+1 , managed by the tuner algorithm (Algorithm 2). This parameter is data-dependent and typically ranges between [0, 1]. The weight update θ t+1 is consequently defined as a convex combination of the current optimizer's weight θ BASE t and a predetermined reference point θ ref .This section presents the convergence behavior of the scaling parameter S t+1 across different environments, analyzed through the mean values over multiple seeds. In our analysis, we extend the examination to other OCO-based optimizers within the lifelong RL setup. Procgen and Atari Vision backbone For both the Atari and Procgen experiments, the Impala architecture was used as the vision backbone. The Impala model had 3 Impala blocks, each containing a convolutional layer followed by 2 residual blocks. The output of this is flattened and connected to a fully connected layer. The impala model parameters are initialized using Xavier uniform initialization.Policy and Value Networks Across all experiments-including Gym Control, Atari, and Procgen-the policy and value functions are implemented using a multi-layer perceptron (MLP) architecture. This architecture processes the input features into action probabilities and state value estimates.The MLP comprises several fully connected layers activated by ReLU. The output from the final layer uses a softmax activation.TRAC TRAC, for all experiments, was implemented using the same experiment-specific baseline architectures and baseline optimizer. For the Procgen and Atari experiments, the base ADAM optimizer was configured as the same as baseline, with a learning rate of 0.001, and for the Gym Control experiments, a learning rate of 0.01 was used. Both learning rates were tested for all experiments and found to have negligible differences in performance outcomes. Other than the learning rate, we use the default ADAM parameters, including weight decay and betas, followed by the specifications outlined in the PyTorch Documentation.5 The setup for TRAC included β values for adaptive gradient adjustments: 0.9, 0.99, 0.999, 0.9999, 0.99999, and 0.999999. Both S t and ε were initially set to (1 × 10 -8 ). Modifications were made to a PyTorch error function library, which accepts complex inputs to accommodate the necessary computations for the imaginary error function. This library can be found at Torch Erf GitHub.6 Distribution Shifts In the Atari experiments, game environments were switched every 4 million steps. The sequence for games with an action space of 6 included "BasicMath", "Qbert", "SpaceInvaders", "UpNDown", "Galaxian", "Bowling", "Demonattack", "NameThisGame", while games with an action space of 9 included "LostLuggage", "VideoPinball", "BeamRider", "Asterix", "Enduro", "CrazyClimber", "MsPacman", "Koolaid". For Procgen experiments, individual game levels were sampled using a seed value as the start_level parameter, which was incremented sequentially to generate new levels. Each new environment was introduced every 2 million steps.In the Gym Control experiments, each observation dimension was randomly perturbed by a value ranging from 0 to 2. This perturbation was constant for 200 timesteps, after which a new perturbation was applied, effectively switching the environmental conditions every 200 steps.The Procgen and Atari experiments were conducted with 8 seeds/runs, while the Gym Control experiments utilized 25 seeds/runs (with the exception of the Mechanic experiments in Table 3 which utilized 300 seeds). The exception was in the L 2 initialization experiments, which used 15 seeds/runs per regularization strength. In Figures 4, 5, 6, 7, 9, 10, 11, 15, 12, 14 , the plotted lines represent the mean of all of the mean episode rewards from the different seeds/runs, and the shaded error bands indicate the standard deviation of all of the mean episode rewards from the different seeds/runs.Compute Resources For the Procgen and Atari experiments, each was allocated a single A100 GPU, typically running for 3-4 days to complete. The Gym Control experiments were conducted using dual-core CPUs, generally concluding within a few hours. In both scenarios, an allocation of 8GB of RAM was sufficient to meet the computational demands.Answer: [Yes] Justification: We provide a python jupyter notebook in our supplementary material to walk through how to setup and reproduce the results for the computationally less expensive experiments (which have a similar setup to the computationally expensive experiments).Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Such as the inactivation of many neurons, due to the ReLU activation function(Abbas et al.,2023;Sokar et al., 2023).2 Indeed, hyperparameter selection, in general, is a well-known problem in lifelong as well as continual learning settings(De Lange et al., 2021).Formally, a dynamical system that given its state θt and input gt outputs the new state OPT(θt, gt).The key difference with gradient descent is the use of intricate (non-L2) regularizers. See(Fang et al., 2022;Jacobsen & Cutkosky, 2022) for a theoretical justification of their importance.https://pytorch.org/docs/stable/generated/torch.optim.Adam.htmlhttps://github.com/redsnic/torch_erf 


Decision: This paper introduces a parameter-free optimizer called PACE, using ideas from online convex optimization to target plasticity loss in lifelong RL. The novel approach is supported by strong theoretical motivation (although no theoretical guarantees) and extensive empirical evaluations across various environments. There were some initial concerns regarding clarity/presentation, the absence of certain baselines, and similar performance to some other methods (e.g., Mechanic), but these were mostly addressed in the rebuttal through the authors providing additional experiments, including an ablation study and more detailed statistical comparisons. There still seems to be only a slight improvement over Mechanic, but the reviewers in general agree that the paper’s contributions are valuable. The authors are advised to incorporate the reviewers’ suggestions to improve clarity/presentation, the new results, and issues raised in the discussion into the final version of their paper.

Reviews: review: paper_summary: paper_summary: The paper tackles the problem of plasticity loss in RL, where a neural network may have reduced performance on learning new tasks after a task switch. The authors develop a hyperparameter-free algorithm, PACE, using ideas from online convex optimization. It can be interpreted as an automatically-tuned weight decay adapted during training. Experiments on large-scale benchmark tasks demonstrate the effectiveness of the approach compared to different baselines.
strength_weakness: strength_weakness: Generally, I have not identified any major weaknesses for this paper.
Some areas that could be improved (but are still acceptable):
- Only a couple of baselines are run, including weight decay and using CReLU activation. 
	A suggestion would be to try layer norm since it is also parameter-free and has shown some benefits to address plasticity loss [1].
- While the algorithm is theoretically-motivated, I did not see any theoretical results included in the paper. If possible, it could be nice to have a theoretical guarantee for PACE even if it is in a different setting such as OCO. Such a result could give additional confidence that the algorithm is sound.
questions: questions: _Clarification questions_
- Fig. 2: I do not fully understand what this figure is trying to convey. In particular, how is the figure demonstrating that PACE may be beneficial in this scenario? Maybe an explanation in the caption could help.

- For the section on tuning $\ell_2$ regularization (line 280), could you include a hyperparameter sensitvity plot i.e. performance vs. hyperparameter value? I would be curious to see how the performance varies around the best values.

- Fig. 6 In the experiments, it seems like even PACE suffers from some amount of plasticity loss since the learning curve does not reach the same maximum as in the first task (more evident in CartPole or LunarLander). Do you have any hypotheses to explain this?

_Comments and suggestions_ 


- Looking at the values of $S_t$ over time (in appendix C) they are usually around 0.01-0.1, which seems fairly small and have a distinct trend of decreasing over time.
This looks inversely correlated to the parameter norm, a quantity that is often measured in the context of plasticity loss. Have you looked into this relationship? I wonder if plotting $S_t \times ||\theta^{Base}||$ shows a roughly constant effective parameter size. 

- If I understand PACE correctly, it looks like overall, it produces the next parameter $\theta_t$ through a weight $S_t$ which is used to obtain convex combination between the initial parameters and the base-optimized parameters $\theta^{Base}_t $. 
What I find counterintuitive is that the gradients for the next step are then evaluated at $\theta_t$ but used to update $\theta^{Base}_t$, which could be quite different in terms of the policy implied. 
Do you have any thoughts about this?

- A potential explanation, as discussed in [3], is that rescaling may also preserve the argmax for classification tasks (and the policy in this case) although it does affect the size of the logits. This would not explain effects on the value network though. 

- On a related note, do the neural networks feature normalization layers e.g. layer norm? 
When normalization layers are used, there is a scale-invariance introduced in the weights (well, if there are no bias units, see page 1. of [2]). I could see PACE being more effective in conjunction with normalization layers due to this rescaling operation not impacting (or having a small impact on) the function learned.  

- About "near convexity", line 298. While the algorithm components are developed for the OCO setting, do you think it is possible that, instead of the problem being "near convex", it is the algorithms that have wider applicability than previous appreciated? That is, they may work well in a more general class of problems?

[1] "Understanding plasticity in neural networks" Lyle et al. 

[2] "Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction" Lyu et al. 

[3] "On Warm-Starting Neural Network Training" Ash and Adams
limitations: limitations: These are discussed.
paper_summary: paper_summary: This work proposes a new method designed to mitigate the loss of plasticity when a model is continuously retrained on a set of tasks. The approach uses online convex optimization (OCO) via the minimisation of a regret function. The approach is compared against two baselines: PPO with ADAM (fixed learning rate) and CReLU. The benchmarks used are ProcGen, the Arcade Learning Environment (ALE) and Gym Control. The empirical tests show promising results with respect to standard reward metrics. Overall, the paper engages with a new and emerging field in machine learning that assume changing input distributions, multiple tasks in various contexts, e.g. supervised and reinforcement learning. Moreover, the paper addresses a recently discovered property of optimizers that reduce the ability of the network to continuously adapt to new tasks if continuously retrained (plasticity loss).
strength_weakness: strength_weakness: 1. The core of the novel algorithm is explained with pseudo code in Algorithm 1 and 2. I understand that the authors are very familiar with all the details of several papers by Cutkosky (at least 4 main sources), on which this work builds on
- direction-magnitude decomposition from (Cutkosky & Orabona, 2018), 
- the additive aggregation from(Cutkosky, 2019), 
- and the erfi potential function from (Zhang et al., 2024b). 
-  (Cutkosky et al., 2023) for the recommended default parameter setup.

However, it's not obvious to me how clear the methodology is given the explanation in Section 3 to machine learning experts that are not very familiar with Cutkosky's work. 

2. A clear highlight of the core technical novelty is missing in relation to previous work. I understand that this work "builds on" the cited work above, but it is generally not a sufficient statement of novelty. More effort is required to clearly spell the novel aspects with respect to previous work. 

3. The most notable limitation that mostly affects my final evaluation is that this method for lifelong reinforcement learning (LRL) **is compared to only two baselines that are not lifelong reinforcement learning algorithms**. I understand that such choice might derive from the narrow focus of preventing plasticity loss. However, a strong paper in LRL needs to engage with the field more thoroughly, describe notable existing LRL, and how the proposed methods contributes to advancing the field. Candidate baselines could include RL implementations of EWC, Experience Relay based RL methods (e.g. Rolnick's CLEAR), PackNet by Mallya and Lazebink or Modulating Masks for LRL by Ben-Iwhiwhu, just to mentioned a few. I find it odd that in the limitation section, the suggestion is the future application of the algorithm to SAC and DQN, which again are not LRL algorithms.

4. As a follow-up to the previous limitation, the paper would benefit from engaging with the LRL field from the start, not only by describing the Spot example (which sounds like the authors are the first to highlight this to the community), but also describing methods that have been developed to address that problem over the last decades, their achievements, their limitations, and why this particular new approach is needed. 

5. Limited analysis is provided on the inner-working of the algorithm (e.g. no ablation studies). 

6. Improving the clarity of the hypotheses would benefit the paper. Is the hypothesis the fact that OCO can be used to reduce plasticity loss? Or that it can be used to reduce plasticity loss better than existing algorithms that do that? Or that it can be used to implement LRL? These questions are only partially answered or addressed in the paper.
questions: questions: My questions are a direct follow-up on the limitations
1. Would you consider comparing your approach to lifelong reinforcement learning algorithms?
2. Why are all the three techniques from the parameter-free OCO required? What do they accomplish individually? Could you run ablation studies? 
3. How does the ADAM default (adaptive learning rate) perform? I understand the need to freeze the learning rate to allow the proposed algorithm perform the adaptation, but it would be nevertheless valuable to assess the performance of the standard version of ADAM.
limitations: limitations: The limitation section is mainly focused on a narrow aspects, e.g. the need for a warm-up. However, this is also in contradiction with the Appendix in which it is suggested that such a warm-up is not required. 

My conclusion is that the narrow scope of this limitations section stems from the lack of integration of this work with the broader field of lifelong reinforcement learning. Addressing the weaknesses that I pointed out, and engaging with a more thorough comparison of this approach with existing LRL approaches, a more insightful view of the pro and cons should emerge.
paper_summary: paper_summary: Many methods have been proposed recently for continual learning that can maintain their plasticity (e.g., through resetting or regularization towards initialization). There is, however, one critical limitation of those methods, which is their hyper-parameter tuning. Although those methods are designed for lifelong learning that can potentially last forever, their performance is highly dependent on the hyperparameter choices. This paper attempts to address this gap in research by introducing a tuning-free approach inspired by online convex optimization for continual learning by incorporating multiple ideas from online convex optimization research and showing that their method PACE is insensitive to hyperparameter choices and can work in a wide range of problems.
strength_weakness: strength_weakness: - Writing needs to be improved. Several parts of the paper seem incomplete, which may require significant restructuring of the paper.
  - The information on online convex optimization doesn’t fully introduce relevant concepts used in the algorithm. Thus, they either need to be removed, and the paper would rely on general intuition to motivate their method and explain how it’s generally related to OCO or to introduce the method in terms of OCO rigorously. The paper, as it stands, does not do either option.
  - The algorithm is not fully explained. For example, how the three components work together is not explained, and the paper relies entirely on citing the papers in which they incorporate their ideas in the algorithm. This prevents the paper from being self-contained. Additionally, what does the dot product between the instantaneous gradient $g_t$ and $\theta_t- \theta_{\text{ref}}$ represent? This was never explained, although it’s a crucial part of the algorithm. The following questions should be answered by reading the algorithm and its explanation: 1) the role of the erfi function and 2) why are we keeping track of the first and second moments of the dot product between the current gradient and $\theta-\theta_{\text{ref}}$? 3) how does the additive aggregation work? 
  - Writing should be more focused on what is crucial for following ideas explained in the paper. For example, the paper explained static regret and then mentioned that discounted regret is more suitable only in one sentence without a proper introduction. Similarly, the paper explains the minimax algorithms for static regret but never explains how parameter-free methods are different. I don’t thinking static regret or minmax formulation is helping in the flow of the paper, instead it may hurt it since it distracts the reader from the main message without mentioning the importance of parts for following along.
- The effectiveness of the three components of the proposed method has not been investigated. Specifically, an ablation study is needed to understand the role of each component and how they interact with each other. For example, the authors might add an ablation study where one component is removed at a time or one component is added at a time.

**Minor issues and suggestions:**
- The following claim in the conclusion section is unsubstantiated: “PACE’s success leads to a compelling takeaway: empirical lifelong RL scenarios may exhibit more convex properties than previously appreciated and might inherently benefit from parameter-free OCO approaches”. The effectiveness of your method doesn’t mean it works because the problem has convex properties. More work is needed to support such a claim and find direct evidence.
- I suggest the method may be called tuning-free, not parameter-free, to enhance readability for a wide range of audiences.
- “From the optimization perspective, the above issues might be attributed to the lack of stability under gradient descent.” -> This requires a citation. Also, what does stability mean in this context?
- line 78: $T$ is not defined. Also, it's doubly used to mean a different thing in line 120.
- Notation in equation 1 is not fully defined (e.g., what is $o(T)$?)
- The word "conservatism” is used loosely without properly defining what it means.
- I believe the “Lifelong vs. Continual” paragraph needs to be dropped. The two words mean the same thing. There is no need to use them in different contexts to mean different things. This would lead to unnecessary division in using the word. Further, I don't think Abbas et al. (2023) focused on backward transfer; instead, they focused on the loss of plasticity. 
- the axis in Figure 4 should be the “average episodic return” or "average cumulative rewards" not “average episode rewards”.
questions: questions: - What is a round? (line 121)
- No definition of discounted regret has been given. How is it defined?
- “Algorithms there mostly follow  the same principle as in the stationary setting, just wrapped by loss rescaling (Zhang et al., 2024a)." What does this mean?
- What does the distribution shift (from 1 to 10) mean in Figure 7 on the x-axis of the first plot? 
- Why is the cumulative sum of rewards used while the authors mentioned that discounted regret is suitable for lifelong RL? 
- The paragraph starting at line 126 is unclear. What does uncertainty set $\mathcal{U}$ mean or represent?
limitations: limitations: The authors adequately addressed the limitations of their work. However, I think since PACE exhibits suboptimal performance in the early stages of deployment, it might suggest that it doesn’t also adapt to a fast-changing sequence of tasks.
paper_summary: paper_summary: This paper proposes a new method for avoiding plasticity loss in 'lifelong RL', a setting in which distribution shift occurs during RL training. Here, the setting is split into a number of discrete tasks which change at a fixed interval. The method involves including a penalty on updates, rooted in convex optimization, which encourages movement back to a reference parameterisation. Upon testing, they find that performance sees drastic improvement over two baselines, and does not suffer from the effects of plasticity loss in the same way.
strength_weakness: strength_weakness: There are a few key weaknesses that I think really hold this paper back, which I will detail below. However, I am willing to increase my score based on rebuttals to the following weaknesses and questions.
- The key thing for me is that it seems Mechanic, in the appendix, performs *very* similarly to PACE (and well within the standard error, making it hard to make any claims about statistical significance - undermining the claim that Mechanic 'slightly underperforms in comparison to PACE'). While I am unfamiliar with this preexisting method, I think it would be good to have included this result into the main paper, with tests across more environments. I appreciate that the discussion mentions that Mechanic has worse guarantees, but empirical demonstration for why PACE is better is pretty necessary given that Mechanic has been included.
- I also think there are missing comparisons to other methods referenced in the paper, which are designed to mitigate plasticity loss. While I appreciate these may not come under the 'hyperparameter-free' setting, comparison to other methods like plasticity injection would be beneficial for showing why PACE is worth using instead.
- In places, the method section was quite confusing. Making the algorithms and subsequent text more concise would be useful as it took me quite a few reads to properly parse and understand.
- The way the results are compared in the table is quite disingenuous, since I believe (particularly in Atari) the results are warped by performance in a very small number of the selected games which had a higher reward scale. This is made clear by the return curves, which show the same story as the table but in an arguably more demonstrative way. It could be worth either normalising scores between each environment, or doing away with the tables entirely.
- As a small nitpick, it felt like lifelong RL is introduced far before its accompanying reference, making it sound like a new setting created for the paper rather than a term used in this context commonly.
- The opening paragraph read in quite a different tone than the rest of the paper. It comes across in a slightly unserious manner that I think detracts from the rest of the paper. Having real-world motivation of the problem setting is very good, but this could have been written better (I appreciate this could be quite a personal opinion, and so did not incorporate it into my score).
questions: questions: - As mentioned earlier, could you please offer a more rigorous comparison with Mechanic (or explain in further detail why PACE is much better, since the difference seems so far very marginal)?
- How much weight decay was used for Adam? Since this method is very closely aligned to weight decay, and the S value was usually *roughly* a constant scale, could tuning the extent of weight decay provide the same benefit as PACE? As pointed out by [1], weight decay and L2 regularisation are marginally different when using Adam. I appreciate this inroduces an additional tunable hyperparemeter, which is against the spirit of the paper.



[1] Kumar, Saurabh et al. “Maintaining Plasticity in Continual Learning via Regenerative Regularization.” (2023).
limitations: limitations: The authors discuss some limitations of their work, such as limited analysis with respect to different algorithms and a brief discussion of Mechanic, another method which performs very similarly to PACE. While I have briefly discussed additional results in regards to limitations, I feel the authors are upfront about a lot of the limitations of their work and that these are not overly limiting to the work.

Comments: title: Thank you
Thank you
Thank you for the ablation study
Ablation Study Experiment and Writing Revisions
Thank you for your response
More PACE and Mechanic seeds for Gym Control experiments
comment: Dear Reviewer,

Thank you for recognizing our additional experiments showing PACE is better than Mechanic and increasing your original score! We appreciate your constructive comments and will polish the camera-ready version to make the presentation of our contribution fair and easy to understand. We are grateful for your engagement in our discussions.
Thank you so much for your thoughtful review and increasing your assessment!
I thank the authors for providing the ablation study. This addresses my concern about the importance of the three components of PACE, and therefore, I raised my score to 6.
Dear authors,

Trying to present the main message of your paper as anything besides the algorithm seems to me to be a mischaracterisation, on the basis that your introduction features a contribution section stating that the paper has two contributions:
- Algorithm (i.e. PACE)
- Experimental (i.e. showing that PACE outperforms standard PPO)
To suggest that the paper is not an algorithmic contribution, but one of analysis, seems questionable and would require significant restructuring of the paper. GIven the contribution set out in your introduction is an algorithmic one, it seems important to ensure that your algorithm outperforms the baseline with significance.

Though your additional experiments required a large number of seeds, I agree that they do seem to demonstrate some significance for the claim that PACE > Mechanic. As such, I have decided to increase my score to reflect this, increasing to borderline acceptance. However, I maintain my low confidence since this has only been shown in a small number of the environments and the improvement seems to be **very** marginal, and as such I still find it very hard to make strong claims about the performance or usefulness of PACE. It also feels that a potentially large amount of restructuring will be required to the paper to cover the necessary changes discussed here, which may require extra reviewing.

Thank you for your engagement in discussions and running additional experiments; I appreciate that this can be very time consuming during a busy time, but I have felt this conversation has been enlightening regarding your paper.
We thank you for your continued engagement with our work!

# Ablation study
Regarding your concern about the ablation study, as previously noted in our rebuttal, the components of our parameter-free algorithm are designed to function integrally, which is foundational to maintaining its parameter-free nature.

However, we appreciate your suggestion that exploring the individual impact of each component could highlight their respective contributions.

**We conducted an ablation study to quantify the impact of removing specific components, expressed as average normalized rewards over 25 seeds:**
| Environment   | Full Configuration (Additive Aggregation & Erfi Tuner) | No Additive Aggregation | No Erfi tuner (1D Online Gradient Descent) | No Erfi Tuner or Additive Aggregation |
|---------------|--------------------------------------------------------|-----------------|---------------------------|--------------------------------------|
| **CartPole**  | 0.32                                                   | 0.19            | 0.02                      | 0.01                                 |
| **Acrobot**   | 0.72                                                   | 0.45            | 0.00                      | 0.00                                 |
| **LunarLander** | 0.54                                                | 0.49            | 0.16                      | 0.14                                 |

We ablate the erfi tuner by replacing it with a baseline 1D online gradient descent tuner, where the parameter is $s_t$.  Additionally, we ablate the additive aggregation component by employing a single tuner with $\beta$=1.

The ablation study demonstrates that the erfi tuner plays the most critical role in performance. Moreover, removing discounting diminishes the benefits of additive aggregation (which automatically selects the best discounting), though some advantages are retained in performance. 

---
# A simple revision for the clarity of writing concern
We are pleased that our detailed explanation in the rebuttal has addressed your inquiries about how PACE works. We recognize that your concern is regarding the clarity of the writing in the method section. **To address this concern, we revise a part of Section 3.** Here is a modified excerpt we will include in the camera-ready text following your suggestions:
> **PACE for Lifelong RL:** In lifelong RL, a key issue is the excessive drifting of weights $\theta_t$, which can detrimentally affect adapting to new tasks. To address this, PACE enforces proximity to a well-chosen reference point $\theta_{\mathrm{ref}}$, providing a principled solution derived from a decade of research in parameter-free Online Convex Optimization (OCO). Unlike traditional methods such as $L_2$ regularization or resetting, PACE avoids hyperparameter tuning, utilizing the unique properties of OCO to maintain weight stability and manage drift effectively.
> 
> The core of PACE, similar to other parameter-free optimizers, incorporates three techniques:
> 1. **Direction-Magnitude Decomposition**: Inspired by Cutkosky & Orabona (2018), this technique employs a carefully designed one-dimensional algorithm, the "parameter-free tuner," atop a base optimizer. This setup acts as a data-dependent regularizer, controlling the extent to which the iterates deviate from their initialization, thereby minimizing loss of plasticity, which is crucial given the high plasticity at the origin (Abbas et al., 2023).
> 2. **Erfi Potential Function**: Building on the previous concept, the tuner utilizes the Erfi potential function, as developed by Zhang et al. (2024b). This function is crafted to effectively balance the distance of the iterates from both the origin and the empirical optimum. It manages the update magnitude by focusing on the gradient projection along the direction $\theta_t - \theta_{\mathrm{ref}}$. 
> 3. **Additive Aggregation**: The tuner above necessitates discounting. Thus, we employ Additive Aggregation by Cutkosky (2019). This approach enables the combination of multiple parameter-free OCO algorithms, each with different discount factors, to approximate the performance of the best-performing algorithm. Importantly, it facilitates the automatic selection of the optimal discount factor during training.
> 
> These three components crucially work together to guarantee good regret bounds in the convex setting and are the minimum requirement for any reasonable parameter-free optimizer.
> 
These revisions follow your suggestions on introducing the concepts utilized in the algorithm and explaining the interactions among the 3 components.

We are happy to include the ablation study and writing revisions in the camera-ready version. **Do you have any other concerns or suggestions that were not addressed here?**
I thank the authors for their explanations and answers. The authors agree with me that writing needs to be improved. I noticed the authors did not respond to my second concern, which an ablation study would have addressed. Thus, based on the current state of the paper, I have maintained my original score since I think major restructuring is needed, which will require another review.
Thank you for the insightful review and for reading our rebuttal response!
Thank you for the thoughtful review and increasing your assessment! At the moment, we are not allowed to edit the original submission pdf, but we will carefully revise the camera-ready version to include an overview and results of these additional baseline methods.
After reading the rebuttal and other reviews, I remain satisfied with the paper and the additional experiments strengthen the paper further. 
I am still recommending acceptance.
Dear Reviewer,

Following up on our previous comment regarding the statistical significance of our results, we have conducted additional experiments on the Gym Control suite, which are more feasible, quicker, and efficient to run on CPUs than the other experiments. Specifically, we increased the sample size to 300 seeds for CartPole, Acrobot, and LunarLander. While our rebuttal initially reported the mean and standard deviations of the seed runs, we now provide a more stringent analysis, including the mean, standard error, and p-values, to make our claim as strong as possible. P-values were calculated using two-sample t-tests. For each control task. we tested the hypothesis that the means between Mechanic and PACE are the same ($\mu_{\text{PACE}} = \mu_{\text{Mechanic}}$)) (Null Hypothesis, H0) against the alternative that they are different (Alternative Hypothesis, H1). P-values here are helpful to quantify the probability of observing the data under the assumption that the null hypothesis is true.


### Results:

| Task             | Method    |  Mean   | Std Error | p-value |
|------------------|-----------|--------|-----------|---------|
| **LunarLander-v2**| PACE       | 0.6018 | 0.0036    | 0.0000  |
|                  | Mechanic   | 0.5755 | 0.0027    |         |
| **CartPole-v1**  | PACE       | 0.3518 | 0.0244    | 0.0021  |
|                  | Mechanic   | 0.3008 | 0.0230    |         |
| **Acrobot-v1**   | PACE       | 0.7044 | 0.0221    | 0.0000  |
|                  | Mechanic   | 0.6396 | 0.0239    |         |

**These results support our argument that PACE outperforms Mechanic in the Gym Control tasks.** The p-values here,  significantly below the commonly accepted threshold of 0.05, strongly reject the null hypothesis in favor of the alternative. This rejection supports our claim about the efficacy of PACE over Mechanic. Additionally, the relatively low standard errors associated with the mean scores for each method further indicate a high level of precision in our results, reinforcing the robustness of our findings.

Even with these results, we reiterate that our primary contribution remains the demonstration that parameter-free OCO helps mitigate the loss of plasticity in lifelong RL.

**Please let us know if this has changed your mind or if you have any other concerns/questions that were not addressed here.**
The authors made a significant effort to improve the paper, in particular with the addition of baselines. Some concerns still remain, but the modifications grant an increase of my initial assessment.

PS: I'm not able to access the new version of the paper. I'm not sure whether the authors have uploaded it.
Thank you for your quick response to our experiments!

We agree with you and acknowledge that more seeds are needed for the precise comparison of PACE and Mechanic. However, we would like to emphasize that **the main message of this paper is the effectiveness of parameter-free OCO for mitigating plasticity loss.** In this regard, PACE vs Mechanic is somewhat a secondary consideration. It's important to note that
1. Mechanic is designed for the classical DL training regime, and hasn't been applied to any lifelong learning setting before. 
2. Both PACE and Mechanic are better than existing approaches in lifelong RL (like $L_2$ regularization and plasticity injection) which require a hyperparameter tuning oracle that's impossible in practice. 

As for how exactly PACE compares to Mechanic, our current experiments show that PACE is better, although we agree with you that more seeds are needed to strengthen this argument (we'll do an even more rigoruous comparison in the camera-ready revision). This is also consistent with the OCO theory: Zhang et al (2024b) showed that the underlying 1D tuner of PACE has a better OCO regret bound (actually, optimal) than the tuner of Mechanic, although the improvement is on the log factor and multiplying constant (instead of the $\tilde O(\cdot)$). In other words, PACE is theoretically better by a slight margin, and it should be *very* hard to improve even further. 

Regarding our contribution, we would also add that our paper is not an "A+B" type of work, where one takes a method A and blindly applies it to a new domain B, hoping it "beats" the benchmark there. Instead, our paper reveals some (in our opinion) neat and surprising insights. 
- For the lifelong RL community, we show that: (1) Parameter-free OCO provides better regularization than other common approaches (e.g., weight decay). (2) Better regularization mitigates loss of plasticity. 
- For the parameter-free OCO community, we show that one could jump out of the typical application regime (DL training), and use the intrinsic connection to regularization to achieve interesting results. In particular, DL training often involves iid data, where applying OCO is somewhat an overkill. Lifelong RL concerns complicated nonstationary data generation mechanisms, which means that the full capability of OCO algorithms can be better exploited. 


In summary, our intended primary contribution lies in demonstrating that parameter-free OCO mitigates plasticity loss in lifelong RL. With the camera-ready version, we will emphasize how PACE, alongside Mechanic, substantiates this point, clarifying the benefit of parameter-free OCO in lifelong RL.


Thanks again! And please let us know if there's anything else we can do to change your mind.
Dear authors,

Thank you for your rebuttal.

# Extra Mechanic Experiments

These provided results reaffirm my point. It is important to highlight that, generally, bolding is reserved for **statistically significant results**. In your provided results, every single value for Mechanic is within the standard (error? deviation? sorry I'm not sure what your intervals are) of PACE, suggesting the results are not statististically significant. In some cases, the differences are on the order of 0.01 with error bars in the range 0.1 - ten times the difference!

# Weight Decay

Thank you for running these experiments. It does seem like PACE has improvements over Adam + WD, but it is interesting how the weight decay has improved the performance of Adam.

# Plasticity Loss Mitigation

Thank you for running these additional baselines. In Cartpole and LunarLander I have concerns about statistical significance between Adam + LN and vanilla PACE, but it is interesting how PACE + LN seems genuinely better than PACE. 

Overall, while your weight decay and plasticity loss experiments are certainly interesting, my primary concern still remains on the performance differential between PACE and Mechanic. In fact, I would say these results have made me less sure that PACE offers any benefits over Mechanic. Are there any other considerations that can differentiate between the two, such as difference in compute/speed? 

As it stands, I am not willing to change my score and still recommend borderline rejection of this paper as my principal concern remains unresolved.