Online Reinforcement Learning in Non-Stationary Context-Driven Environments
Authors: Pouya Hamadanian
Arash Nasr-Esfahany
Malte Schwarzkopf
Siddhartha Sen
Mohammad Alizadeh
Venue: ICLR 2025 Spotlight (2025)
Keywords: catastrophic forgetting
reinforcement learning
context-driven MDP
online learning
non-stationary
Primary area: reinforcement learning
TL;DR: A reinforcement learning algorithm that solves catastrophic forgetting in non-stationary exogenous context-driven environments by constraining the policy optimization on out of distribution samples.

Abstract: We study online reinforcement learning (RL) in non-stationary environments, where a time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to "catastrophic forgetting" (CF). The agent tends to forget prior knowledge as it trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice), employ brittle regularization heuristics, or use off-policy methods that suffer from instability and poor performance.

We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems environments with a variety of synthetic and real context traces, and find that it outperforms a variety of baselines in the non-stationary setting, while achieving results on-par with a "prescient" agent trained offline across all context traces.

LCPO's source code is available at https://github.com/pouyahmdn/LCPO.

Published as a conference paper at ICLR 2025
ONLINE REINFORCEMENT LEARNING IN NON-STATIONARY CONTEXT-DRIVEN ENVIRONMENTS
Pouya Hamadanian
Arash Nasr-Esfahany MIT CSAIL
Malte Schwarzkopf CS Brown University ma malteêca brown.edu
MIT CSAIL
pouyah@mi t edu
arashne@mit... ede
Siddhartha Sen
Mohammad Alizadeh MIT CSAIL al alizaacee maizadeh@mii...o edu
Microsoft Research dideeemicrosoft... com
ABSTRACT
We study online reinforcement leaming (RL) in non-stationary environments, where time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to "catastrophic forgetting" (CF). The agent tends to forget prior knowledge as trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice), employ brittle regularization heuristics, or use off-policy methods that suffer from instability and poor performance.
We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems enviromments with a variety of synthetic and real context traces, and find that it outperforms a variety of baselines in the non-stationary setting, while achieving results on-par with "prescient" agent trained offline across all context traces.
LCPO's source code is available httptt:///// httttp:://github..oom wwwp:///aoooooooooooooocoa
INTRODUCTION
Those who cannot remember the past are condemned repeat it. (George Santayana, The Life of Reason, 1905)
Reinforcement Learning (RL) has success in many domains (Mao et al. 2017 Haamoja et al. 2018a| Mao et al. 2019b Marcus et al.) 2019 Zhu et al. 2020 Haydari & Yılmaz, [0222] but real-world deployments have been rare. A major hurdle has been the gap between simulation and reality, where the environment simulators do not match the real-world dynamics. Thus, recent work has turned to applying
RL in an online fashion, i.c., continuously training and using an agent in a live environment (Zhang Gu et al. (021))
2021
While online RL difficult in and of itself, it is particularly challenging in non-stationary environments- also known as continual RL (Khetarpal et al. 2020)) -where the characteristics of the enviromment change over time. key challenge is Catastrophic Forgetting (CF) (McCloskey Cohen 1989. agent based on function approximators like Neural Networks (NNs) tends to forget its past knowledge when training sequentially on new non-stationary data. On-policy RI. algorithms (Sutton & Barto 2018) are particularly vulnerable CF in non-stationary environments, since these methods cannot retrain stale data from prior experiences.
In this paper, consider problems where the source of the non-stationarity is observed exogenous context process that varies over time and exposes the agent to different environment dynamics. Such context-driven environments (Sinclair
al. 2023 Mao al. 2018 Zhang al. 2023 Dietterich (et al.) 2018 Pan et al., (2022) appear in a variety of applications. Examples include computer systems subject to incoming workloads (Mao al. 2018 locomotion in environments with varying terrains and

---

Published as a conference paper at ICLR 2025
obstacles (Heess al. 2017b), robots subject to external forces (Pinto al. 2017). and more. In contrast to most prior work (Alegre et al. 2021 Chandak et al. 2020). we do not restrict the context process to be discrete. piecewise stationary or Markov.
Broadly speaking, there are three existing approaches to mitigate CF in online learning, One class of techniques is task-based (Rusu et al. 2016 Kirkpatrick et al. 2017 Schwarz et al., 2018 Farajtabar et al. 2019 Zeng et al. (2119). Such works assume explicit task labels that identify the different context distributions which the agent encounters over time. Task labels make it easier to prevent the training for one context from affecting knowledge learned for other contexts. In settings where task labels (or boundaries) are not available, few proposals try to infer the task labels via self self-supervised (Nagabandi al. 2019b) or Change-Point Detection (CPD) approaches (Padakandla et al.] [2200] Alegre et al. (0011) These techniques, however, are brittle when the context processes are difficult to separate and task boundaries are not pronounced (Hamadanian et al. 2022). Our experiments show that erroneous task labels lead to poorly performing agents in such environments ($5).
A second category of approaches avoids task labels by approximating task-based methods with heuristics [Schwarz et al.] 20188] Chaudhry et al. 20188] Kaplanis et al. 2018 Woo et al., (022). However, these heuristics are based on brittle assumptions about the nature and cadence of non-stationarity For example, one approach implicitly assumes each episode is a distinct task, and uses a window of past N episode to regularize learning (Woo et al. 2022). These assumptions rarely met and would likely lead to poor performance in practice, as we observe in our analysis and evaluations ($5 and D. D.2 and $D.3 in the Appendix).
The third category of approaches employs rehearsal, i.e., learning using past or generated data. For example, off-policy learning (Sutton & Barto 2018) makes it possible to retrain on past data. These techniques (e.g.. Experience Replay (Mnih et al. (013). CLEAR (Rolnick et al. 2019), etc.) store prior experience data in buffer and sample from the buffer randomly to train. Not only does this improve sample complexity, it sidesteps the pitfalls of sequential learning and prevents CF (Rolnick et al. (219)). However, off-polico methods come at the cost of increased hyper-parameter sensitivity and unstable training (Duan al.) 2016)
Gu et al. [0166] [Haamoja et al. (0188).. This brittleness is particularly catastrophic in an online setting, as we also observe in our experiments
We present LCPO ($4.1), an on-policy RL algorithm that "anchors" policy outputs for old contexts while optimizing for the current context. Unlike prior work, LCPO does not rely on task labels and only requires an
(OOD) detector, i.e., function that recognizes old experiences that occurred in a sufficiently different context than the current one. LCPO maintains a bounded buffer of past experiences, similar to off- off-policy methods (Mnih et al. 2013). But an on-policy approach, LCPO does not use stale experiences to optimize the policy. Instead, it uses past data to constrain the policy optimization on fresh data, such that the agent's behavior does not change in other contexts.
We evaluate LCPO on several environments with real and synthetic contexts ( and show that it outperforms variety of baselines across mentioned categories in the online learning setting. We also compare against a "prescient agent" that is trained offline on the entire context distribution prior to deployment. The prescient agent does not suffer from CF. Among all the online methods, ICPO is the closest to this idealized baseline. Our ablation results show that LCPO is robust to variations in the OOD detector's thresholds and works well with small experience buffer sizes.
LCPO's source code is available online at httpt::t///// //tp::////tthb.... wtt com/pooyyhnndn/ICPP
2 PRELIMINARIES
Notation. We consider online reinforcement learning in non-stationary context-driven Markov Decision Process (MDP), where the context is observed (only up to the current time step t) and exogenous. Formally, at time step & the environment has state s, E and context 2, E Z. The agent takes action a; E A based on the observed state s; and context 2,, a, 2)), and receives feedback in the form of a scalar reward re r== (0==(s,,,, 2,,22 ).). where r(:::):: :SSx Z x A +R
A RR is the reward function. The environment's 's state, the context, and the agent's s action determine the next state, $t+1, according to transition kemel, T(s [(s+115,,2200 (s,2),0)) The context 2, is an independent stochastic process, unaffected by states s, or actions a, Finally, dg defines the distribution over initial states (sp). This model is fully defined by the tuple M M= (z 100 T T.do.)).
2

---

Published as a conference paper at ICLR 2025
Non-stationary contexts. The non-stationary context (= impacts the environment dynamics and implies a non-stationary environment. We assume the context process can change arbitrarily: e.g., it can follow predictable pattem, be iid samples from some distribution, be discrete process multi-dimensional continuous process, experience smooth or dramatic shifts, be piecewise stationary, or include any mixture of the above. We have no prior knowledge of the context process, the environment dynamics, or access to an offline simulator. Examples of (observed) context processes include market demand in supply chain system, incoming request workloads in virtual machine allocation problem, customer distributions in airline revenue management (Sinclair et al. 2023), traffic information in vehicular networks (Wu et al. al.||201 (017)) terrain profiles in locomotion task (Heess al. allaaa2111111 (0177)), network traffic for video streaming (Mao et al. 2020) and congestion control [Winstein & Balakrishnan 2013), etc. etc.
We seek good long-term performance. Formally, for given policy x: : - and context process = (2 we define the lifelong return as J( = lim Ei=1 Vss for an infinite horizon MDP. Similarly, for finite horizon MDPs of length H. where episode is subject to context traces
Goal.
Hf -/= -t=1 (²) we define the lifelong
z, = (2 2 (++())))- and has an episodic return of R2 return as (TTTz)) = +om Ei=1 R. T. For a policy sequence I== { e.g., the sequence of policies resulting from a continual RL algorithm, we can define the lifelong return (II,z) similarly.
Online RL. In most RL settings, policy is trained in a separate training phase. During testing, the policy fixed and does not change. By contrast, online learning starts with the test phase, and the policy must reach and maintain optimality within this test phase. An important constraint in the online setting is that the agent gets to experience each interaction only once. There is no way to revisit past interactions and try a different action in the same context. This is a key distinction with training in a separate offline phase, such as in meta-learning (Al-Shedivat et al. 2018), where the agent can explore the same conditions many times.
Note that the policy that maximizes lifelong return π² = J has to be prescient, i.e., it needs to have upfront knowledge of the context process z. Since an online agent causal and has only observed context values up to the current time step it can never perform well as this prescient policy. Therefore, in general online RL agents will have a gap with prescient policies in terms of lifelong retum. In certain special cases the online RL can asymptotically reach the prescient policy, c.g., when the context process is Markovian the entire context-driven MDP collapses to a standard MDP with a state š =< $,, >. However, we do not intend to limit the context process in any way, and our aim it to minimize the gap between the online and prescient agents for arbitrary context processes.
3 RELATED WORK
Non-stationary RL. Non-stationary RL is a family of sub-problems, such as CF, latent context inference,
meta-learning, etc (Khetarpal et al. (020). In this work we focus on CF, and highlight the differences of CF with other well-known non-stationary RL problems below. Then, we will explore related work for CF in Machine Learning (ML) and RL. We highlight other lines of work in { in the Appendix.
Latent Context Inference. These works consider a context-driven MDP where the context 2, is unobserved. The goal is to infer an estimated context & from other signals, such as transition functions, reward functions,
(Hallak
2015
Zintgraf al. [2019] Xie et al.
2020
Caccia et al.
2020 Lee
2020
He et al. [2020] Poiani et al. 2021 Chen et al. [022] Huang et al.
[2022 Feng et al. 2022 Ren et al.
2022
Woo et al. a [2022] Bing et al. [002 Luo et al. [022 Lee et al. (023)) Once inferred, traditional
RL algorithm such as Soft Actor Critic (SAC) learns a policy π (-(((,,(t) from the state and inferred context,
and is typically compared to an 'upper-bound policy' that observes the true context T( rT{1,),=.). These
works aim to recover the unobserved context, while we focus on CF after observing the true/recovered context. In fact, the tupper-bount policies in these works are baselines we compare to in $ Combining LCPO with this line of work to solve CF in environments with latent context is interesting future work.
Catastrophic Forgetting. Three general techniques exist for mitigating CF in ML (Parisi et al. 2019
(1) regularizing the optimization to avoid memory loss during sequential training (Kirkpatrick et al. 2017
Zenke et al. 2017 Farajtabar et al. 2019: Lopez-Paz Paz & Ranzato 2022]; ; (2) training separate parameters per task, and expanding/shrinkng parameters as necessary (Rusu et al. 2016 Shmelkov et al. 2017; Li et al.
2019; (3) rehearsal, i.e,, retraining on original data or generative batches (Shin et al. 2017 Isele & Cosgun
Isele
2018
Atkinson al. 2021); combinations of these techniques (Schwarz et al. 2018 Aljundi et al. 2019
3

---

Published as a conference paper at ICLR 2025
Regularization techniques such as Elastic Weight Consolidation (EWC) and Orthogonal Gradient Descent (OGD) require task labels. Approximations have been proposed for problems without task labels (Schwarz) et al. 2018) or boundaries (Woo et al. 2022). Kaplanis et al. (2018) use biologically inspired models
of brain synapses to regularize networks.
Another class of approaches aims to infer task labels, by learning the transition dynamics of the MDP, and detecting a new environment when surprising sample is observed with respect to the learned model (Doya) al. 2002 da Silva et al. 2006 Padakandla et al. [2020] Alegre et al. [202]. The inferred labels are often used to train separate volicies/model to mitigate CF. These methods are effective when MDP transitions are abrupt and have well-defined boundaries, but are brittle and perform poorly in realistic environments
with noisy and hard-to-distingg to-distinguish non-stationarities (Hamadamian et al.) 2022).
For rehearsal, we can use learned models of the MDP replay past experiences (Xu et al. 2020 Lee et al. 2020 |ong et al. [0200 Huang et al. 2021 Janner et al. 202T). An alternative type of rehearsal is off-policy training (Haarnoja et al. 2018b van Hasselt et. 2016) (e.g., Experience Replay (Mnih et al. 2013), which
can train on stale data, naturally circumvent sequential learning and avoid CF. However, off-policy RL is em-
pirically unstable and sensitive to hyperparameters due to bootstrapping and function approximation (Sutton) & Barto| 2018), and is often outperformed by on-policy algorithms in online settings (Duan et al. [0066] Gu ct al. [0166] Haarojj et al. 2018b). CLEAR (Rolnick et al. 2019) is an off-policy RL algorithm explicitly designed to overcome CF with fast adaptations. Similarly, PT-DQN (Anand & Precup 2023) learns permanent Q-network to remember past tasks while learning a transient Q-network for fast adaptation.
Constrained Optimization. ICPO's constrained optimization formulation is structurally similar to Trust Region Policy Optimization (TRPO) (Schulman al. 2015), despite our different problem setting.
4 LOCALLY POLICY OPTIMIZATION
Our goal is to learn policy π(--) that takes action a ( in context-driven MDP characterized by an exogenous non-stationary context process.
4.1 ILLUSTRATIVE EXAMPLE
Consider a simple environment with a discrete context. In this grid-world problem depicted in Figure] the agent can move in 4 directions in 2D grid, and incurs base negative reward of -1 per step until it reaches the terminal cxit state (no penalty in the last step) or fails to reach the exit within 20 steps. The grid can be in two modes; 1) 'No Trap' mode, where the center cell is empty, and 2) 'Trap Active' mode, where walking into the center cell incurs a reward of -10. When in 'No Trap' mode, the optimal path passes through the center cell, and the best episodic return is -3. the 'Trap Active' mode, the center cell's penalty forces the optimal path go left at the blue cell for optimal episodic return of -5. This environment mode is our discrete context and the source of non-stationarity in this simple example. The agent observes its current location and the context, i.c., whether the trap is on the grid (2₁ 1) not (2 = 0) in every episode (beginning from the start square).
Advantage Actor Critic (A2C). We use the A2C algorithm to train a policy for this environment, while its context changes every so often. Figure Ic FiguretIedepictss depicts the episodic return across time and Figures and depict the total variation distance between the optimal and learned policy when the policy input is 'No Trap' mode (Figure [d] or 'Trap Active' mode (Figure [Ic. This distance represents how close the learned policy is to the optimal in either context. The agent initially attains optimality for the 'No Trap' mode, but once the context changes at epoch 4K it immediately forgets it. Note that during epochs 4K- the A2C agent is only trained on samples from the 'Trap Active' mode, and its output for the 'No Trap' mode is drifting. When the context changes back to the 'No Trap' mode at epoch 16K, the agent behaves sub-optimally (epochs 16K-18K) before relearning. Figure [e shows that A2C also forgets the optimal 'Trap Active' policy during the final 4K epochs.
Key Insight. Since the policy observes the current context 2,, should be able to distinguish between different environment modes. Therefore, if the agent could surgically modify its policy the current state-context pairs 5t (=1),,,2, and leave outputs for other state-context pairs a-1|,,,2' * 2)) unchanged, would eventually learn good policy for all contexts. In fact, tabular RL achieves this trivially in this
4

---

Published as a conference paper at ICLR 2025
Episodic Return
Fintah
-10
Tabular A2C
LCPO
-15
No Trap
Trap Active
No Trap
12
16
Start
Start
Epochs (xIR)
(a)
e-Blue Coll ) Stepe)
Buu Call (agt))
0.75
Drv(Rt Tep Activr
0.75
Drv (Fl an Tra
0.25
0.25
0.00
0.00
No Trap
Trap Active
No Trap
No Trap
Trap Active
No Trap
0
12
16
20
12
16
20
Epochs (x1x)
Epochs (x1K)
(e)
Figure 1: A 3x3 grid-world problem with two modes and the optimal path visualized in blue, (a) In the 'No Trap' mode, the center square is safe pass through. (b) In the "Trap' mode, the agent must avoid the trap with a longer path. (c) Episodic return across time in the grid environment. (d and e) Total variation distance between learned and optimal policy outputs for the (d) 'No Trap' mode, and the (e) TTrap Active' mode at the blue cell (lower is better). Tabular A2C and LCPO remember the optimal decision for either context during shaded regions and instantly attain optimal returns when the environment switches.
finite discrete state-context space. To illustrate, we apply tabular version of A2C: i.e., the policy and value networks are replaced with tables with separate rows for each state-context pair (18 total rows). Figures [d and Ic demonstrate that the tabular RL policy for each context remains unchanged when does not actively interact with that context. This is because when an experience is used to update the table, it only updates the row pertaining its own state and context, and does not change rows belonging other contexts. Under sufficient conditions, tabular RL can provably converge to the optimal policy for such environments. Due to space constraints, we state the theorem and its proof in iB the Appendix.
Can we achieve a similar behavior with neural network function approximators? In general, updating neural network (say, policy network) for certain state-context pairs will change the output for all state-context pairs, leading to CF. But if we could somehow "anchor" the output of the neural network on distinct prior state-context pairs (analogous the cells in the tabular setting) while we update the relevant state-context pairs, then the neural network would not "forget".
LCPO. Achieving the aforementioned anchoring does not require task labels. We only need to know if two contexts and zj are different. In particular, let the batch of recent environment interactions < ,,, > be B, and let all previous interactions (from possibly different contexts) be B,- Suppose we have difference detector W (B,,B,,) that can be used to sample experiences from B, that not from the same distribution as the samples in the recent batch B,, i.e., the difference detector provides out-of-distribution (OOD) samples with respect to B,.. Then, when optimizing the policy for the current batch B,, we can constrain the policy's 's output on experiences sampled via W W,B,B,,)) to not change (see $4.2 for details). We name this approach Locally Constrained Policy Optimization (LCPO). The result for LCPO is presented in Figures ld] and Te While it does not retain its policy as perfectly as tabular A2C, does sufficiently well to recover near instantaneously upon the second switch at epoch 16K.
Change-Point Detection (CPD) vs. OOD Detection. CPD (and task labeling in general) requires stronger assumptions than OOD detection. The context process has to be piecewise stationary to infer task labels and context changes must happen infrequently to be detectable. Furthermore, online CPD is sensitive to outliers. In contrast, OOD is akin to defining a distance metric on the context process and be well-defined any context process. Consider the context process shown in Figure 2 We run this context process through a CPD algorithm (Alegre et al. 2022) for two different sensitivity factors o F and represent each detected change-poine with red vertical line. A slight increase in sensitivity leads
5

---

Published as a conference paper at ICLR 2025
-2
10M
20M
Figure 2: A sample context process 2,, and detected change-points at two thresholds. Teasing meaningful task boundaries is difficult for this process, but defining OOD metric is intuitive.
34 detected change-points, and these change-poing to assign task lahels for this smooth process and there aren't t clearly separated segments that can be defined tasks. However, an intuitive OOD detection method is testing the distance between 2; and 2j is larger than some threshold, i.c. 2 2j > 1. Altogether, OOD is considerably easier in practice compared to CPD. Note that although the grid-world example- and discrete context environments general is good fit for CPD, this environment was purposefully simple to explain the insight behind LCPO.
also reasonable. There is obvious way
4.2 METHODOLOGY
Consider parameterized policy my with parameters 0. Our task choose a direction for changing I such that improves the expected return on the most recent batch of experiences B,, while the policy 'anchored" on prior samples with sufficiently distinct context distributions, W (,).
Algorithm I.PPO Training
I: initialize parameter vectors lg empty buffer B,
2: for each iteration do
3:
B, + Sample mini-batch of new interactions
4:
S. 4 (Ba.
5:
v VaLto (0:
6:
if S. is not empty then
7:
g(z):=
8:
V. mn.
9:
while Osa U. violates constraints do
10:
11c 5- 2
00 + 4
12:
else
0g 4 0u +
14:
B. - B BBr
In supervised learning, this anchoring is straightforward to perform, e.g., by adding regularization loss that directs the neural network to output the ground truth labels for OOD samples (Caruanal 1997). In the case of an RL policy, however, do not know the ground truth (optimal actions) for anchoring the policy output. Moreover, using the actions we took in prior contexts as the ground truth is not possible, since the policy may have not converged at those times. Anchoring those actions may cause the policy to relearn suboptimal actions from an earlier period training. To avoid these problems, LCPO solves change for OOD samples. Formally, we
constrained optimization problem that forces the policy consider the following optimization problem:
min Lisr (l:B(): L LPG (..
(I)
s.t. (BL(BO.eW(B...B)), Canchor
We use the standard definition of policy gradient loss, that optimizes policy maximize returns (Schulman et al. 2018 Mnih et al.) 2016 Sutton & Barto 2018):
H
(po(O:B))=Brwe B,
r
We use automatic entropy regularization (Haarojj et al. 2018c), to react to and explore in response to novel contexts. The learnable parameter 0, is adapted such that the entropy coefficient is ef and the

---

Published as a conference paper at ICLR 2025
entropy remains close to target entropy H. This worked well in experiments but LCPO could use any exploration method designed for non-stationary context-driven environments.
(0, =
F. B (logr (1,)))
(3)
We use KL-divergence a measure policy change, and for simplicity we use 1 (B,. as a shorthand for BB,. (B,.. 1 (To, (s),,)] Here, On denotes the current policy parameters, and we are solving the optimization over determine the new policy parameters.
Buffer Management. To avoid storing all interactions in Ba, use reservoir sampling (Vitter 1985); we randomly replace old interactions with new ones with probability n, n, the total interactions thus far. Reservoir sampling guarantees that the interactions in the buffer are uniformly random subset of the full set of interactions. For pseudo-code see $C.1 in the Appendix.
Ha, where ni is the bufter size and
Difference detector: To realize W (Ba, Br), we treat as an OOD detection task. A variety of methods can be used in practice ($5). e.g.. we can compute the Mahalanobis distance (Mahalanobis 2018 the normalized distance of each experience's ' context with respect to the average context in B,- and deem any distance above certain threshold to be OOD. To avoid a high computational overhead when sampling from W BB), we sample larger batch from Ba, and keep the state-conter pairs that are OOD with respect to B,. If not enough different samples exist, we do not apply the constraint for that update. For pseudo-code and further implementation details about the OOD detector, see CC2 and CC.3
Solving the constrained optimization. To solve this constrained optimization, we approximate the optimization goal and constraint, and calculate a search direction accordingly (pseudocode in A Algorithm|[ Tp. Our problem structurally similar to TRPO (Schulman et al.) 2015). though the constraint quite different. Similar TRPO, we model the optimization goal with a first-order approximation, i.e. Lzo((0;) = fo + (0 0p) Ve (0:-)lo.- and the constraint with a second order approximation DKI(00,0:)) (0 0o) V VKL (0. 0:))), (0-6o). The optimization problem can therefore be written as
min (0-0))" 7T
s.t. OT) A(D 00) S Canchor
where A, 80, on; solved using the conjugate gradient method followed by line search (Schulman et al. 2015 Achiam
and セ= V Liot (0:)),, This optimization problem can be
al. 2017).
Bounding policy change. The above formulation does not bound policy change the current context, which could destabilize learning. We could add a second constraint, i.e., TRPO's 's constraint, (Kz(00,8:B, S Crecent (note that this constraint is different from that in Equation I the samples come from B, instead of ((B,, B,), )). However, having two second order constraints is computationally expensive. Instead, we guarantee the TRPO constraint in the line search phase (lines 10 in Algorithm T). where we repeatedly decrease the gradient update size until both constraints are met.
5 EVALUATION
We evaluate LCPO across six environments: four from Gymnasium Mujoco (Towers et al.) 2023). one from Gymnasium Classic Control (Towers al. 2023), and straggler mitigation task from computer systems (Mao et al. 2019a Hamadanian et al. 2022). These environments are subject to synthetic or real context processes that affect their dynamics. Our experiments aim to answer the following questions: (1) How does LCPO compare to baselines, and can it perform as well as the pre- pre-trained prescient policies
($5.1)? (2) How does the accuracy of the OOD sampler W ((--) affect LCPO (..2) ? (3) How does the
maximum buffer size = Bal affect LCPO (3,.3) ? We include further ablations of LCPO and baselines Appendices SE.1
D.5 and D.
Baselines. We consider the following approaches for comparison: Regularization-based (1) Online
EWC (Kirkpatrick et al. 2017: Chaudlar et al.) 2018 Schwwarz et al. 2018), (2) Sliding OGD (Farajtabar al. 2019
Woo et al. 2022) and Benna-Fusi DQN (BFQDN) (Kaplanis al. 2018), Task Inference:
7

---

Published as a conference paper at ICLR 2025
(4) Model-Based Changepoint Detection (MBCD) AAlegre et al. 2021). Rehearsal: (5) Model-Basee Based
Policy Optimization (MBPO) (aanner et al., 202T), (6) CLEAR (Rolnick et al. 2019), (7) PT-DQN (Anand
& Precup 2023), (8) SAC (Haamoja et al. 20188) and (9) Double Deep Q Network (DDQN) (Hasselt
et al. 2016) On-policy RL: (10) A2C (Mnih et al.
2016) and (11) TRPO (single-path) (Schulman et al.
2015), ) both using Generalized Advantage Estimation (GAE) (Schulman et al. 2018), Prescient RL: (12)
as described in $2) the best of policies trained with A2C (Mnih ct 2016)) TRPO (single-path) (Schulman
al. 2015 DDQN (Hasselt et al. (016) and SAC (Haarnoja et al. 2018 For more details about these
base fine, refer to $D in the Appendix
Experiment Setup. We use 25 random seeds for gymnasium for slower schemes) and 10 random seeds for the straggler mitigation experiments, and use the same hyperparameters for LCPO in all environments and contexts. Gym environments were modified to accept discrete action space policies, as even prescient policies struggled to learn stable continuous space policies in the presence of contexts (See |F3). Hyperparameters and neural network structures are noted Appendices $F4 and $ These experiments were conducted on machine with 2 AMD EPYC 7763 CPUs (256 logical cores) and 512 GiB of RAM. With 32 concurrent runs, experiments finished in ~ hours. This figure does not include runtime devoted to tuning the baselines.
Environment and Contexts We consider six environments: Modified versions of Pendulum-vi from the classic control environments, InvertedPendulum-vv. InvertedDoubeeeendluun---
Hopper-v4 and Reacher-v4 from the Mujoco environments (Towers et al.] (023). and a straggler mitigation environment (Hamadanian et al. 2022). In the gym environments, the context is exogenous "wind" process that creates external force on joints and affects movements. We append the external wind vectors from the last time-steps to the observation, since the agent cannot observe the externa context that is going to be applied in the next step, and a history of prior steps helps with the prediction. We create 4 synthetic context sequences with the Monstein-Uhlenbeck process (Uhlenbeck & Ornstein 1930). piecewise Gaussian models, or hand-crafted signals with additive noise. These context processes cover smooth, sudden, stochastic, and predictable transitions at short horizons. All context traces are visualized in Figure in the Appendix. Context traces I and 2 are 20 million, and context traces 3 and 4 are 8 million steps long. All baselines were allowed a 'warm-up' period of million time steps, and episodes were truncated at 200 steps. For the straggler mitigation environments, authors in (Hamadanian et al. 2.22). that are from a production web framework cluster at Microsoft, collected from single day in Fehruary 2018. These workloads are visualized in Figure 8b
workloads provided by the
in the appendix.
OOD detection. We set the buffer size n, to 1% of all samples, which is ni, S. 200K. To sam- ple OOD state-context pairs W W(B,, B,,), we use distance metrics and thresholds. For gym environ- ments where the context is a wind process, we use L2 distance, i.c., if w. = E,w_B, [w] is the av- erage wind vector observed in the recent batch B,, we sample minibatch of states in B, where W(B.,
r). There exist Nomain-speciff. models for workload distribu- /uy/, E B.: l/w,- -,y, 12 tions in the straggler mitigation environment, but we used Mahalaaobis distance as is well-acoepted and general approach for outier detection in prior work (Lee et 20188 (Podolskiy et al., (02T)) Concretely, we fit a Gaussian multivariate model to the recent batch B,, and report minibatch states in B, with a Mahalanobis distance further than from this distribution (see CC22 in the Appendix for more details).
5.1 RESULTS
To evaluate across different gymnasium environments and traces, we score agents with Normalized Return, i.e., for each environment and context process we report a scaled score function where 0 and are the minimum and maximum lifelong return across all agents. We prefer agents with higher scores over different environments and traces. Figure 3a| provides a summary of all gymnasium experiments (full details in Tables 5 and 6 in the Appendix). LCPO maintains a lead over basclines, is close to the best-performing prescient policy, all while learning online and sequentially. We present a detailed analysis of baselines' performance in $ and we summarize these findings below. We also report the wallclock time for each scheme in $22 in the appendix: LCPO is ~ 11.x as demanding as A2C.
Online EWC and Sliding OGD employ heuristics circumvent the necessity of task labels in the original techniques. Conceptually, they implicitly assume past episodes are separate "tasks". Empirically these heuristics are not successful at solving CF. As for BFDON, Kaplanis et al. (2018) note that while the
8

---

Published as a conference paper at ICLR 2025
Bax: Prescient
CLEAR
SAC
MBPO
1C00 /
MRCD
LCPO mun
LCPO
0.6
LCPO 4.0
LCPO (1.7)
Stidng
A2C CPO (MHI))
DOON
PT-DQN
LCPO R
12.0
LCPO 8² 1.0
AZC
1.00
103
CDF (96)
CDF (%6
70
00
20
0
0.7
Normalized Return
Normalized
(a)
(b)
Figure 3: CDF of normalized lifelong returns, where 0/1 denote the lowest/highest returns among agents. Shaded regions denote 95% confidence intervals. (a) LCPO outperforms all online agents, and remains the closest prescient policies. (b) LCPO affected by the OOD threshold but still outperforms baselines.
architecture was successful in simple environments, it failed with more complex and challenging ones. In our experience, this architecture did not provide any benefits compared to vanilla DDQN.
MBCD struggles to tease out meaningful task boundaries. In some experiments, it launches anywhere between policies just by changing the random seed. This observation is line with MBCD's sensitivity in $4.1 and prior work (Hamadanian et al. 2022).
MBPO performed poorly, even though we verified that its learned model of the environment is highly accurate. CLEAR (Rolnick et al. (019)) and PT-DQN (Anand & Precup 2023) are highly hyperparameter sensitive due to how they address catastrophic forgetting. While we tuned both extensively for the Pendulum-vl environment, we did for all baselines, they fail catastrophically in other environments. SAC and Deep Q Network (DQN) struggle to outperform A2C in the online case. This falls line with prior observations (Hamadanian et al. (2022) and can be attributed to the instability and hyperparameter sensitivity of off-policy RL (Duan et al. 2016 Gu et al.) 2016 [Harnoja et al. (0188) and the quick adaptation that fully online algorithm such A2C provides (Sutton al. 2007). J. In fact, despite not having been designed for non-stationn non-stationary RL, A2C is the most successful baseline.
Table 1:
Tail latency (negative reward) and 95th percentile confidence ranges for different algorithms and contexts in the straggler mitigation environment.
Online Learning
LCPO Agg
LCPO
LCPO Cons
Online BWC
Best Prescicut
MBCD
MBPO
A2C
TRPO
DDQN
SAC
Mod
Workload
1076
1701
2531
2711
1854
984
+10
+16
+7
+112
119
+232
#710
+464
#47
2245
(TRPO)
589
617
586
678
891
724
604
864
633
644
509
Wockload
+44
+:62
+227
4:38
4:54
4:22
100
105
+7
+27
(A2C)
For the straggler mitigation environment, Table Dalleeprreeene presents the latency metric (negative reward) over two workloads. Recall that this environment uses real-world traces from production cloud network. The overall trends are similar to the gymnasium experiments, with LCPO outperforming all other baselines. This table includes three variants of LCPO, that will be discussed further in $5.2
5.2 SENSITIVITY OOD METRIC
LCPO applies constraint to OOD state-context pairs, dictated by the OOD sampler (B., We vary the OOD threshold o-which the OOD method uses sampling- and monitor the normalized return for the gym environments in Figure and the straggler mitigation environments in Table [ In the gym environments, lower value for a yields tighter margin of difference before a sample is deemed OOD. LCPO is affected by a,, with the smallest threshold o² =0.25 performing the best. However, LCPO still maintains a lead over the A2C baseline across or variations. We also experiment with a handicapped OOD metric that observes state-context vector I2 =< 8t, >> withou the ability separate state and context.

---

Published as a conference paper at ICLR 2025
We use the Mahalanobis distance OOD metrie MMahalanobis] 2018 at several thresholds RAD for this experiment. Despite the handicap,, the LCPO +Mahalanobis surpasses the LCPO +L2 agent that we have used in this evaluation. This is not surprising, the L2 distance is less robust than Mahalanobis distance, but easier to interpret. In the straggler mitigation environment ICPO Agg. ICPO Med and ICPO Cons use o 5, 6 and 7, and higher value for a yields more conservative OOD samples (i.e., fewer samples are detected as OOD). The difference between these three is significant: The model in LCPO Agg allows for x more samples to be considered OOD compared LCPO Cons. Table [ provides the normalized return for LCPO with varying thresholds, along with baselines. Notably, all variations of LCPO achieve similar results.
5.3 SENSITIVITY TO BUFFER SIZE
LCPO uses Reservoir sampling (Vitter 1985) to maintain a limited number of samples ri.- We eval- uate how sensitive LCPO to the buffer size in Figure 4 (full results in Table 8 in the Appendix), The full experiment has 8-20M samples. LCPO maintains its high performance, even with as little as ni = 500 samples, but drops below this point (statistically significant in over one third of experi- ments). This is not surprising, as the context traces do not change drastically at short intervals, and even 500 randomly sampled points from the trace should be enough have representation over all of the trace. However, with more complicated and digeediiensional contexts, a higher buffer size would likely be necessary.
LCPO nu 20 LCPO Ty
AXC
LCPO nos 200K
LCPO n1 26
100
CDF (%)
75
50
25
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Normalizze Return
Figure 4: CDF of normalized returns of LCPO in gym environments with various buffer sizes. Shaded regions denote 95% confidence intervals. LCPO loses performance with nı 500.
DISCUSSION AND LIMITATIONS
Network Capacity. In general, online learning methods with bounded parameter counts will reach the function approximator's (neural network's) s) maximum representational capacity. LCPO is not immune from this, as we do not add parameters with more context traces. However, neither are prescient agents. To isolate the effect of this capacity and CF, we compare against prescient agents, rather than single agents trained on individual tasks or context traces (He et al. 2020). This ensures a fair evaluation that does not penalize online learning for reaching the capacity ceiling. If the maximum capacity has been reached, it may be beneficial remove significantly old samples from B, allow LCPO to forget such contexts, thereby favoring flexibility instead of stability.
Exploration. LCPO focuses on mitigating catastrophic forgetting in non-stationary RI.. An orthogonal challenge in this setting is efficient exploration, i.c., to explore when the context distribution has changed but only once per new context. Our experiments used automatic entropy tuning for exploration (Haamoja ct al. 2018b); while empirically effective, this was not designed for non-stationary problems. LCPO may benefit from a better exploration methodology such as curiosity- exploration (Pathak et al. 2017).
Efficient Buffer Management. We used Reservoir Sampling |itter 1985), which maintains a uniformly random buffer of all observed state-context samples far. Future work could explore strategies that selectively store or drop samples based on their context, e.g., to maximize sample diversity.
7 CONCLUSION
We proposed and evaluated LCPO, a simple approach for online learning in non-stationary context-driven environments. LCPO requires two conditions: (I) the non-stationarity must be induced by an exogenous observed context process; and (2) a similarity metric is required that can inform us if two contexts come from noticeably different distributions (OOD detection). This is less restrictive than prior approaches that require either explicit or inferred task labels. Our experiments showed that LCPO outperforms baselines on several environments with real and synthetic context processes.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For Theorem B.2 we include the proof and assumptions in $ We include detailed accounts of environ- ments, context traces, baselines, hyperparameters, software and hardware in $5 the main text and $ and $G in the Appendix. We include implementation details and pseudo algorithms in $4.2 and $ in the Appendix. A link to the source code also provided S
ACKNOWLESSEMMETS
We thank our reviewers for insightful comments. This work was supported by NSF grants 1751009 and an award from the CSAIL-MSR Trustworthy AI collaboration.

---

Published as a conference paper at ICLR 2025
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization, 2017.
Maruan Al-Shedivar, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Contin- uous adaptation via meta-learning in nonstationary and competitive environments, 2018.
Lucas N Alegre, Ana LC Bazzan, and Bruno C da Silva. Minimum-delay adaptation in non non-stationary reinforcement learning via online high-confidence change-point detection. In Proceedings of the 20th
International Conference Autonomous Agents and MultiAgent Systems, pp. 97-105, 2021.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free free continual learning, 2019.
Nishanth Anand and Doina Precup. Prediction and control in continual reinforcement learning, 2023. URL
httpps://axxvv.orgg//oooo
11669
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting. Neurocomputting, 428:291-307, Mar 2021. ISSN 0925-2312. doi: 10. 10166].eeucom22220) 11.050. URL htttpp://dx.oo..orggg/../o
neucom 2020 11. 0.50
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning, 2020. URL http::/wwwwwwwwwwp//o//////aa//a
06350
Zhenshan Bing, Lukas Knak, Fabrice Oliver Robin, Kai Huang, and Alois Knoll. Meta-reinforcement learning in broad and non-parametric environments, 2022.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam Hadj Laradji. Irina Rish, Alexandre Lacoste, David Vázquez, et al. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances Neural Information Processing Systems, 33: 33:16532-16545, 2020.
Rich Caruana. Multitask leaming. Machine learning, 28:41-75, 1997.
Yash Chandak, Georgios Theocharous, Shiv Shankar, Martha White, Sridhar Mahadevan, and Philip Thomas. Optimizing for the future in non-stationary mdps, 2020.
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian Walk for Incremental Learning: Understandingg Forgetting and Intransigence, pp. 556-572. Springer doi: 10. 0011007978----30-01555333
International Publishing, 2018 ISBN 9783030012526.
URL
http::/dx.do// org 10. 1011100070---0000-2220
Xiaoyu Chen, Xiangming Zhu, Yufeng Zheng, Pushi Zhang, Li Zhao, Wenxue Cheng, Peng Cheng, Yongqiang Xiong, Tao Qin, Jianyu Chen, et al. An adaptive deep rl method for non-stationary en- vironments with piecewise stable context. Advances Neural Information Processing Systems, 35: 35449-35461, 2022.
Bruno C. da Silva, Eduardo W. Basso, Ana L. C. Bazzan, and Paulo M. Engel. Dealing with non- stationary environments using context detection. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pp. 217-224, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1145/1 143844. 113872. URL Shtthtt:: : //doi org/
1145/1143844.1 114387113872
Thomas G. Dietterich, George Trimponias, and Zhitang Chen. Discovering and removing exogenous state variables and rewards for reinforcement learning. In International Conference on Machine Learning. 2018. htt.::ttss://.oo.t.
semant. cscholaase ar, org/ Cwarr///ooreeeee (oo-ppuII:
Kenji Doya, Kazuyuki Samejima, Ken-ichi Katagiri, and Mitsuo Kawato. Multiple model-based reinforce- ment learning. Neural computation, 14(6):347713699 2002.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pp. 1329-1338. PMLR, 2016.
12

---

Published as a conference paper at ICLR 2025
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018.
Mehrdad Farajtabar, Navid Azizan, Alex Mou, and Ang Li. Orthogonal gradient descent for continual learning, 2019, URL wttps:ttt//....m. http:://aaxvvvvoooo www.oog/g/////oooooo////ooo/.aae abs absaabasl990 11t//191...00000 07104
Fan Feng, Biwei Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for non-stationary reinforcement learning. Advances in Neural Information Processing Systems, 35:31957-319711 2022.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning, 2019.
Shixiang Gu, Timothy Lillicrap. Zoubin Ghahramani, Richard E Tumne, and Sergey Levine. Q-prop: Sample-efficient policy gradient with off off-policy critic. arXiv preprint arXiv:6611.02244 2016.
Zhouyou Gu, Changyang She, Wibowo Hardjawana, Simon Lumb, David McKechnie, Todd Essery, and Branka Vucetic, Knowledge-asistedd deep reinforcement learning in 5g scheduler design: From theoretical framework implementation. IEEE Journal on Selected Areas Communicatioon,, 39(7): 2014-2028, 2021.
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Compos- able deep reinforcement learning for robotic manipulation. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6244-6251, 2018a. doi: 10.1109/CRA.201 1099/ 8,8460756.
Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International conference on machine learning pp. 1861-1870. PMLR, 2018b.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications, 2018c. |tL /taptttss//axxi....ot wttp://.rrggaab///i/eeee aatpaaasss//...e 05905
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes, 2015. URL
httppe///arxxv.orrr///////550.0 0tts//aaab/11020.22999 02259
Pouya Hamadanian, Malte Schwarzkopf, Siddartha Sen, and Mohammad Alizadeh. Demystifying
reinforcement learning in time-varying systems. arXiv preprint arXiv:2201.05560, 2022. doi:
10.4850AARXIV.200..000
URLhttpa http:///axivvv.o org wttpor///////////// abs / 2201. 05560
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double g-eamning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pp. 2094-2100.
AAAI Press, 2016.
Ammar Haydari and Yasin Yılmaz. Deep reinforcement learning for intelligent transportation systems: survey. IEEE Transactions on Intelligent Transportation Systems, 23(1):11-32, 2022. doi: 10.1 TTTS. 20TS.2020.000861..
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei Alex Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. In 4th Lifelong Machine Learning Workshop at ICML 2020, 2020. URL
net Aow.mmmdddeeeeaddillll
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments, 2a.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, M. Ali Eslami, Martin Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments, 2017b. httthttss://rxxtt http://awwx.....///..a.a/e/////a absp://b/lii 1707. 02286
Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang, Adarl: What, where, and how to adapt in transfer reinforcement learning, 2022.
Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti. Continual model-based reinforce- ment learning with hypemetworks, 2021.
13

---

Published as a conference paper at ICLR 2025
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization, 2021.
Christos Kaplanis, Muray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses, 2018.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: review and perspectives. arXiv preprini arXiv:2012. 13490, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiaoo Ramalho, Agnieszka Grabska-Barwinkke. et al. Overcoming catastrophic forgetring in neural networks. Proceedings of the national academy of sciences, 114(13): 3521-3526, 2017.
Qingfeng Lan and A. Rupam Mahmood. Elephant neural networks: Born be a continual learner, 2023. URL
org. abs//2311 01365
Qingfeng Lan, Yangchen Pan, Jun Luo, and A. Rupam Mahmood. Memory-efficient reinforcement learning with value-based knowledge consolidation, 2023. URL hattps://arxiv.org)) /bs//
2205. 10868
Hyunin Lee, Yuhao Ding, Jongmin Lee, Ming Jin, Javad Lavaei, and Somayeh Sojoudi. Tempo adaptation in non-stationary reinforcement learning, 2023.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks, 2018. URL http::t:/.......rrrgtt/
03886
Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization model-based reinforcement learning, 2020.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. 2019.
Vincent Liu, Raksha Kumaraswamy, Lei Le, and Martha White. The utility of sparse representations for control in reinforcement learning, 2018. URL htttts:://aaatt.. org/ abs wwss/811...... 06626
Vincent Liu, Han Wang, Ruo Yu Tao, Khurram Javed, Adam White, and Manth White. Measuring and mitigating interference reinforcement learning, 2023. URL htt.ttpps///rx...rrrr/aaa//// 04887
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning, 2022.
Fan-Ming Luo, Shengyi Jiang, Yang Yu, ZongZhang Zhang, and Yi-Feng Zhang. Adapt to environment sudden changes by learning a context sensitive policy. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7)7637-7666, Jun. 2022. doi: 10. 10.11609/aii..2 v3617.20734 URL https::/ojjs aaai. org. wewpg//iixx..omm httpp////////aaaae//eeeeeeeeee/e
730
Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Sankhya: The Indian Journal of Statistics, Series A (2008-), 80:$1-S7, 2018.
Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. Neural adaptive video streaming with pensieve. In Proceedings of the Conference of the ACM Special Interest Group Data Communication, pp. 197-210, 2017.
2017.
Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan. Zili Meng, and Mohammad Alizadeh. Learning scheduling algorithms for data processing clusters, 2018. URL wwtp::hpp////cc /ttp:::://xxxxvo/////// abs/ 1 81 0. 963
14

---

Published as a conference paper at ICLR 2025
Hongzi Mao, Malte Schwarzkopf, Hao He, and Mohammad Alizadeh. Towards safe online reinforcement learning in computer systems. In NeurIPS Machine Learning for Systems Workshop. Curran Associates New York, NY, 2019a.
Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Mohammad Alizadeh.
Learning scheduling algorithms for data processing clusters. In Proceedings of the ACM Special Interest Group on Data Communication, SIGCOMM '19, pp. 270-288, New York, NY, USA, 2019b. Association for Computing Machinery. ISBN 9781450359566. 111.01..111313111111 https::/doi. http://....0r/111111144333333333
doi: 10.1145/3341302.33420000 URL
3342080
Hongzi Mao, Shannon Chen, Drew Dimmery. Shaun Singh, Drew Blaisdell, Yuandong Tian, Mohammad Alizadeh, and Eytan Bakshy. Real-world video adaptation with reinforcement learning, 2020.
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul. Neo: A learned query optimizer. arXiv preprint arXiv:1904.03711
2019.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989. ISSN 0079-7421. doi: scieece/ secieecc//rrrleel//////////71110
URL httt::::/ww..cciereeeii...c
con/
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Picter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning, 2019a.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning Continual adaptation for model-based rl, 2019b.
Sindhu Padakandla, Prabuchandran K. J., and Shalabh Bhatnagar. Reinforcement learning algorithm for non-stationary enviromments. Applied Intelligence, 50(11 (011))33500366666 jun 2020. doi: 10. 10070/000499
020-01758-5. URL httt:::ttt//to :/o::://dooiiee http://ii.ooogg..oooo//oo org/10 10010000222000000000000-----1110
E. S. PAGE. CONTINUOUS INSPECTION SCHEMES. Boonetrika,4111221100000 06 1954. ISSN 0006-3444, doi: 10. (0.1099/biome/441112..000
htt.:httss//oi..rrtt
ometi/me//// /4
1-2. 100
Minting Pan, Xiangming Zhu. Yunbo Wang, and Xiaokang Yang. Iso-dream: Isolating noncontrollable visual dynamics in world models. In Alice H. Oh, Alekh Agarwal. Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URI httt:tttts///pereerrr
net. forum? (....BB.SSVgggg
Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach to learning
sparse representations online, 2021. URL httpt::/aaaxii.ooooooaa/// abs////b//// http///sss//111//// 1911. 08068
German Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 11::54-71, 71, 2019. ISSN 0893-6080. doi: http:///toooooo///..oooo//eeeeee
01.22.. URL https::///// www. www.wwwwwwww.eieee..t.cmm com/
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, performanem deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus Buc, E. Fox, and R. Garnett (eds.).
15

---

Published as a conference paper at ICLR 2025
Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curann Associates, Inc., 2019. URL
imperrrtiveeeee Sttp::/ce..ppparroommooeoaaaaaaa
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darell. Curiosity-driven exploration by self-supervised prediction, 2017. /ttl:tttp/////x..oorr//// http://///aaa/10..55.000. 05363
Lemrll Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning, 2017. URL
02702
Alexander Podolskiy, Dmitry Lipin, Andrey Bout, Ekaterina Artemova, and Irina Piontkovskaya. Revisiting mahalanobis distance for transformer-base out-of-domain detection, 2021. hRtttss/://axxx
org/ bzg/abs/21021 03778
Poiani, A Tirinzoni, M Restelli, al. Meta-reinforcemmett leaming by tracking task non-stationarity. In IJCAI, pp. 2899-2905, International Joint Conferences on Artificial Intelligence, 2021.
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free
deep rl for model-based control. 2020.
Antonin Raffin. RI baselines3 zoo. httpes///gtchb..coo///kk/////--- 2020.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Emnsstu,, and Noah Dormann. Stable-haselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021. /ttt:ttt:// wttpppppsppppppp....pppp. 222122622222-6222---- html
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta-reinforrcement learning via probabilistic context variables, 2019.
Hang Ren, Aivar Sootla, Taher Jafferjee, Junxiao Shen, Jun Wang, and Haitham Bou Ammar. Reinforce- ment learning presence of discrete markovian context evolution, 02 2022.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experience replay for continual learning, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv: 2016.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization, 2015. DRLLattpss//aaxx..c////////2
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
1737706377777777 06347
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- High-dimensional continuous control using generalized advantage estimation, 2018
Jonathan Schwarz, Jelena Luketina, Wojciech M. Czamecki, Agnieszka Btabska-Barwinske Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay, 2017.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting, 2017.
Sean R. Sinclair, Felipe Frujeri, Ching-An Cheng, Luke Marshall, Hugo Barbalho, Jingling Li, Jennifer Neville, Ishai Menache, and Adith Swaminathan. Hindsight learning for mdps with exogenous inputs. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations, 2021.
16

---

Published as a conference paper at ICLR 2025
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Richard S. Sutton, Anna Koop, and David Silver. On the role tracking in stationary environments. In Proceedings the 24th International Conference on Machine Learning, ICML '07, pp. 871-878, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781 595937933. doi:
0M.144//1733496.1733666 httpa//////////a////a//////////. 11151111111112222111 1273606
URL htt.hhtps:/// /tt_httpss:///o..orrre
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization, 2019.
Yee Whye Teh, Victor Bapst. Wojciech Marian Czamecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning, 2017. URL
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL httLhttps//// /ttp://eoooooooooo//// wwtp:/coooo//111100000000
G. E. Uhlenbeck and L. S. Omstein. On the theory of the brownian motion. Phys. Rev, 36:823- 841, Sep 1930. doi: 10.1 1033/hhyseev336.22. URL.https://link aps. haps.org//oo/1...11 PhysRev. 36.366.4 823
Hado van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. Learning values across many orders of magnitude, 2016.
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mahematical Software (TOMS), 11(1):37-57, 1985.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Chen-Yu Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: optimal black-box approach, 2021.
Keith Winstein and Hari Balakrishnan. Tep ex machina: computer-generatte congestion control. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, SIGCOMM '13, pp. 123-134, New York, NY, USA. 2013. Association for Computing Machinery. ISBN 9781450320566. doi: 10.1 10.1145/2486001 2486600.. htt.sttsss:////ttt 1ttp://oo....1111111111/../
2486000
Honguk Woo, Gwangpyo Yoo, and Minjong Yoo. Structure bearning-based task decomposition for reinforcement learning in non-stationary environments. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8557-8665, Jun. 2022. doi: 10.1609/aai.. ww........6....0... URL https
//ojs. //ojs.eaai.n
aaai orr.org///iooo php/ C..p/AAAI//A www.//artice//iaee...aaa..aa.aee 20844
Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: Architec- ture and benchmarking for reinforcement learning in traffic control. arXiv preprint arXiv: 10, 2017.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non-stationarity, 2020.
Mengdi Xu, Wenhao Ding, Jiacheng Zhu, Zuxin Liu, Baiming Chen, and Ding Zhao. Task-agnostic online reinforcement learning with an infinite mixture of gaussian processes, 2020.
Zhenghai Xue, Qingpeng Cai, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. State regularized policy optimization on data with dynamies shift, 2024. URL
abs/2306. 03552
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization, 2020.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 18))3664-72, 2019.
17

---

Published as a conference paper at ICLR 2025
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual leamingg through synaptic intelligence, 2017.
Huanhuan Zhang, Anfu Zhou, Yuhan Hu, Chaoyue Li, Guangping Wang, Xinyu Zhang, Huadong Ma, Leilei Wu, Aiyun Chen, and Changhui Wu. Loki: Improving long tail performance of leamningbbsse real-time video adaptation by fusing rule-based models. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, MobiCom '21, pp. 775-788, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383424, doi: (0.11455944799933488339 URL https:// http::///woo..ggg////ooooo 0tt.://0...153347995533333333 3483259
Jinpeng Zhang, Yufeng Zheng, Chuheng Zhang. Li Zhao, Lei Song, Yuan Zhou, and Jiang Bian. Robust situational reinforcement learning in face of context disturbances. In Proceedings of the
40th International Conference Machine Leariing (ICML), September 2023. URL https:
/www www..mmoocoooommmmmm.m.........t www.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning, 2020. URL https?
/ //arxiv.oom org/ abs/2004 12570
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference Machine Learning, pp. 7693-7702, PMLR, 2019.
18

---

Published as a conference paper at ICLR 2025
APPENDIX A RELATED WORK: CONTINUED
Meta-learning. Here, in the general case, the environment can switch between a set of possible MDPs without an explicit signal, At the 'meta-train' phase, the policy is allowed to train on a part or all of the MDPs. At the 'meta-test' phase, the policy must make decisions in continual RL setup where the MDP may abruptly change, and it has to adapt to the current MDP with few-shot adaptation (Al-Shedivat et al., 2018 Nagabandi et al. 2019bJa) or context inference (Rakelly et al. 2019). In our problem setup, we assume access to the environment beforehand.
Multi-task RL. In this problem, there are several (e.g., 10 or 50) different tasks with different MDPs. The goal of this line of work is to leam shared policy for all tasks that approaches the performance of learning separate policies per task (Yang et al. 22200] [Sodhani et al. [202]) These tasks may come with contextual information about the task that can be used in the policy (Sodhani al. (02T). This problem
not continual RL, does not experience CF, and the learner is allowed to explore all tasks at the same time.
Often the goal is postitive transfer learning, i.e., learning faster on all tasks parallel than learning tasks separately (Thh et al. 2017).
Another type of work focuses on speeding up the learning process for set of new tasks by pre-training on set of old tasks (Xue et al. 2024). This line of work bears similarities to multi-task RL and meta-leaming.
Assuming there are no explicit signals for environment contexts, Wei & Luo (202 provide a regret-optimal black-box RI. algorithm.
Interfernce in stationary RI. Investigating interference in vanilla RL (non-stationarity in sample distribution) an adjacent and interesting line of work (Bengio al. 2020 Pan al., [021 Liu et al.) 2023; 2018). The type of non-stationarity discussed in these works is different than what we study. Here, non-stationarity refers to the moving target of bootstrap losses, such as Q-learning, due to shifting policies that change future data distributions. Non-stationarity in our problem setup means the MDP itself is shifting, irrespective of the changes the policy makes. The second type of non-stationarity cannot be resolved even with "perfect" RL algorithms that deal with interference in stationary MDPs.
However, there may be ideas that are transferable. For example, this line of work suggests that techniques such as GAE (Schulman et al. (2018) and target networks reduce interference (Bengio et al. 2020 Liu
ct al. 2023). Another interesting avenue is experimentation with sparse learning (Pan et al. 202 Liu et al. 2018
Buffer limitation interference. A line of work, related to interference, deals with a type of non- stationarity induced by having small buffers (e.g.. 32 samples, compared to the typical thousands to millions) in off off-policy algorithms. These works aim to mimic an off off-policy agent with unbounded buffers, and do focus on context-driven non-stationarity. They learn policies that perform closely to unbounded agents via techniques such as following old target Q networks (Lan et al. (202)) or utilizing sparse -parse-gradient activation functions (Lan Mahmood 2023). Note that we do compare to DDQN and SAC with unbounded buffers.
APPENDIX B PROOF OF OPTIMALITY IN TABULAR CONTEXT-DRIVEN RL
Below, we show how the non-stationary environment in $2 can be learned with vanilla RL algorithms, the state, action and context spaces finite, context switches occur at episode boundaries. First, we prove Lemma I and then we prove the main theorem Theorem B.2
Lemma B.1 Given monotonically decreasing sequence
(a that satisfies the following conditions:
oc
Σ a 000
of <00
(5)
Consider any subsequence {8,)32m where for any 1 j. there exists
such that
19

---

Published as a conference paper at ICLR 2025
Prove that:
Bj
0c
37
<00
(6)
==1
j=1
Proof. First, note that since (a: _ is monotonically decreasing, have:
aNxi M
(7)
For the first equality, we have for all k>0:
30
NNxi
1=1
(8)
Nxitk
i=
Thus, we have:
N
Nx
8
8
N
ONit
(9)
k=1iml
2 Σ
04
i=N+1
Therefore:
N
3
=0
(10)
N
N
1
The second bound trivially proven:
8
Nx
(11)
i=1
0
Theorem B.2. Consider context-driven MDP as defined in $2 Under the following set of assumptions,
prove that the Q-learning algorithm (Wakins & Dayan 1992) converges to the optimal policy:
Rewards are bounded bounded rewards Ir 2r/20 R.
2. We have a monotonically decreasing sequence of learning rates {au) where 02 <1, and
20
Σ
00=00
00
Σ
0
<000
(12)
3. The state, action and context spaces are finite [5).|A|,12]44 <00.
4. There exists N, such that for any context E Z, z occurs at least once in any consecutive subsequence of the context trace of size N.
5. The context only changes episode terminations.
20

---

Published as a conference paper at ICLR 2025
LCPO (Algoritmm 1)
unin (i, B.) Loo(d; B.) (.))B..)
12 (. (-)) fancher
Polizy To(
W( Rh)
Observe
Interact
(sr.a)
All Sam Samples ples Buffer
Out Of Datribution Metric
Recent Sampl es Buffer
eg. L2, Mahalanobis,
Reservoir
Context-Driven Non-Statiooary
Sampling w/ mi stots
Environment
(oc 4))
Receive reward
Figure 5: Architecture of LCPO.
Proof. As contexts only switch at episode boundaries, the Q-leaming updates for each context E 2 are isolated from Q-values of other contexts. Thus, for any context = E 2, we have a separate Q-learning session. If lg denotes the of episode indices where the context trace equal to z, If prove
20.=00
Σ SEL.
a <00
(13)
iEL.
the original Q-learning proof of convergence (Watkins & Dayan 1992.
To prove this, note that from assumption 5 we can surmise that that for any 1 S there exists N (xx(-1) is j such that i E I.. Therefore based on Lemma B. the subsequence of learning rates satisfy the Watkins Q-learning condition.
APPENDIX C LCPO IMPLEMENTATION
Here, we will discuss the implementation details LCPO. Figure depicts the overall architecture of LCPO.
C.I RESERVOIR SAMPLING
As discussed in $4.2 LCPO needs to maintain a buffer Bg of all samples observed so far. However, this buffer will grow with time and incur extensive memory costs. To limit the buffer size while maintaining distribution of all samples observed so far, we utilize Reservoir Sampling (Vitter 1985).
The pseudo-code for the implementation can be found in Algorithm 2 Reservoir sampling operates by maintaining a bounded list of samples B,, While the number of samples in the list B, has not reached max capacity nig, all samples are admitted. Once the buffer full, random index is sampled in the range of to n, 1, where n1, is the number of samples observed so far. If the index is smaller than ni, the element at index replaced with the new sample. If it is larger, the sample thrown away. This strategy bounds the size of Ba, but maintains a uniform distribution of samples from the true list of all samples observed so far.
C.2 OOD FUNCTION
LCPO requires a definition for OOD samples, i.c., samples that come from contexts far away from recent samples B.. Intuitively, the optimal policy conditioned this OOD context should be significantly different from the policy optimal conditioned on the current context.
Such metrics can be based domain insight, where expert who is familiar with how the context value changes the MDP would define a function to detect OOD samples. Altemattvely, may be generic OOD criteria commonly used in the literature, such as 1.2 thresholding, Mahalanobis distance, etc. In this case,
21

---

Published as a conference paper at ICLR 2025
Algorithm Reservoir Sampling
1: Input: max capacity
2: B, O Initialize empty buffer
3: <-00 Initialize sample count
4: for each new sample x do
5:
12, + Increment the sample count
6:
if' ny > n4, then 1r,Unif(0,m, Sample random index smaller than n
7:
8:
if < ny then
9:
Replace element index in Ba with
10:
clse
Throw new sample away
12:
else
Ba - Ba (x)
Append x Ba
the OOD function needs be tuned be meaningful, which is common challenge in OOD detection work. An interesting line of future work is utilize MDP transitions in the warm-up period for tuning the threshold online.
Concretely, LCPO requires an OOD function w2z, B,) that denotes whether is OOD with respect to B, or not. the L2 thresholding used for the gym environments subject to the wind context in $5] we calculate average over Br, i.e., Mru Eu B. (w]. and define (w, [B,) := ||w Mwll2 threshold σ.. For the Mahalanobis distance (Mahalanobis) 2018) measure used for the straggler mitigation experiments in $5 calculate average and standard deviation over B,, ie., Mw = Eww B, [w] and
for some
En E Em [(w )wr) and define w/ (ww,B, == (w- (w)) " Ew (w)) > o² for some threshold a.
C.3 OOD SAMPLING
Finally when want to sample batch of size b from W W,B,,B,), forming the full set B,) and then sampling randomly from it has a computational cost that scales with [Ba]. To avoid this cost, we instead sample experiences from B, and keep the ones that are OOD. If this results in at least OOD experiences, we return the samples. If not, we conclude that there aren't enough OOD samples B. with respect to B,- pseudo-code provided in Algorithm 3
Analytically we can model the sampling with a binomial distribution, where p= Ez- [x,,B,,) is the fraction of samples B, that are OOD with respect B,. We will successfully get batch of OOD samples with the probability 1 - F (bbss.p) where F(:..)) is the binomial cumulattvv distribution function. In all experiments in we have set 94% when p= 0.22. In other words, if at least 22% of the samples in B, are OOD, we are highly likely to be able to collect batch of size b OOD samples, and unlikely if the rate 18% or below.
5b. With b 200, the success probability is 5% when p= 0.18 and
Algorithm OOD Sampler
Input: All samples buffer Ba, Recent samples buffer B,, OOD function w(z,B,), max sample
count, b batch size
2: Initialize empty list Brut + O
3: i+-
4: while [Bout] << and i< do
5:
2~ B, Sample from Ba
6:
if (2, Sample OOD then
7:
Bout Bout + {z) Add sample list
8:
i i+1 Increment
9:
if
Bott b then
10:
return Bout
else
12:
return ()
22

---

Published as a conference paper at ICLR 2025
APPENDIX D BASELINES
D.I ONLINE EWC
EWC Kinkpatrick et al. 2017) regularizes online learning with a Bayesian approach assuming task indices, and online EWC (Chaudhry et al. online EWC to our problem, we update the importance vector and average weights using a weighted moving average in every time step. The underlying learning approach is SAC (Haarnoja al. 20186).
et al. 2018) generalizes to task boundaries. To adapt
EWC applies a regularization loss
N 00
Lewe
(14)
the training loss, where N are the number of tasks, 02 is the converged parameter set for task &t and Fk is the diagonal of the Fisher Information Matrix (FIM) of task k on the converged model for task k. Online EWC applies an approximation of this regularization loss
Lewe a: (0. 8
(15)
using rumning average F2 of the diagonal of the Fisher Information Matrix (FIM), and running average of the parameters 0r. The running average F is updated with weighted average =7= = where Fj is the diagonal of the FIM respective to the recent parameters and samples Similarly, the running average 07 uses the same parameter 83.
Online EWC may constrain the policy output to remain constant on samples in the last ~ f-1 epochs, but it has to strike a balance between how fast the importance metrics are updated with the newest FIM (larger 3) and how long the policy has to remember its training (smaller (3). This balance will ultimately depend on the context trace and how frequently they evolve. We tuned a and 3 on Pendulum-vl for all contexts, trying E f0.05.0 1.1.1...1.1. 1,0,100...0...1 100,100,10000 1000000) and 8 11 E {1M M.M. 10M) (M denotes million). The returns are visualized in Figure 6 with full details in Table 2 There is no universal 33 that works well across all contexts and online EWC would not perform better than LCPO even if tuned to each context trace individually. We ultimately chose 8 33M samples to strike a balance across all contexts, but struggled to surpass SAC on other environments.
Prescient
LCPO
SAC
Online EWC
Context
Context
-00
-00
Lifelong Return
-22
Combext 2
Context
-000
350
400
-600
0.05 1M
3M
3M
IM
10M
0.05 1M
3M
IM
3M
10M
0.S. 3M
0.6. 10M
1.0. 3M
1a. 10M
0.05,
0.05
0.1.
0...
0.6.
1.0.
1.0.
1..
0.05.
0.5
0.1
0.1.
0.1
0.S.
1.0.
Figure 6: Pendulum-vl lifelong returns and 95% confidence bounds of Online EWC with 12 hyperparam-
eter trials. Hyperparameters are labeled as (a,ô-1), where a is the regularization strength and 3 is the averaging weight. The optimal online EWC hyperparameter is sensitive to the context, but LCPO is better even if online EWC is tuned per context.
D.2 SLIDING OGD
OGD (Farajtabar et al. 2019) projects gradients for new tasks to vector spaces that are orthogonal to previous tasks' loss functions. OGD needs task labels, and Woo et al (2022) circumvent this by making projecting.
We deviate from the original notations (A,2) Rusu al. 2016 since they could be confused with the MDP
discount factor y and GAF discount factor X (Schulman al. 2018
23

---

Published as a conference paper at ICLR 2025
Table 2: Average and 95th percentile confidence ranges for lifelong returns for online EWC variants in the Pendulum-vl environment with external wind processes.
Online EWC
0.5
1M
3M
10M
1M
3M
10M
1M
3M
10M
1M
3M
10M
Connect
-209
-207
-206
-199
-203
-204
-201
-204
-207
-208
-212
-200
Piace
+13.0
19.59
16.6
=5447
47.28
48806
47.57
4:11.
1:367
49969
13387
1:449
Content
4489
-545
-544
-477
-527
-525
-540
-531
-541
-545
-510
-539
174.1
#26
1:47.6
1:659
1:49.0
148.5
171.2
Connect
-380
-356
-372
-422
-349
-393
-340
-330
-320
-320
-316
Trace
+82.4
1108
=70.0
100
472.4
+:11
+114
1:661
+6855
4:29.1
+:461
#44.7
Contect
-396
-400
-007
-413
-397
-006
-404
-12
-409
-416
Trace 4
16.1
136.7
15.6
221.1
223.0
1:1.9
+11.
15.
1185
4:3.5
+20.1
2.1.7
OGD (Farajtabar al. 2019) circumvents CF by applying parameter updates that are orthogonal the losses of past tasks. To do this, after a task has finished training, OGD calculates the gradient vector w.r.t samples for that task and stores those gradients. When the next task training begins, gradient updates are projected to orthogonal spaces w.r.t the saved vectors from before. This procedure requires task labels and convergence. Sliding OGD (Woo et al. (2022) avoids needing task labels by using the gradient updates in the past N episodes for projection. In other words, Sliding OGD implicitly assumes that each of the past N episodes were "tasks" that have already finished training. This assumption is incorrect in our problem setup, the context trace may change slowly. As result, the gradient vectors of past episodes belong to the same "task" as the current episodes, and this projection fully hinders training.
D.3 BENNA FUSI DQN
This approach applies a biologically plausible model for synapses neural network weights in a deep Q network (Kaplanis et al., 2018). Conceptually, the weights are regularized with their past values in multiple different time scales. Kaplanis al. (2018) note that while the Benna-Fuss DQN architecture was successful in simple environments, it failed with more complex and challenging ones. In our experience, this architecture did not provide any benefits compared to vanilla DDQN.
D.4 MBCD
This work handles piecewise stationary environments by inferring Stange--pinnt and task labels (Alegre et al.] (202T). It trains models to predict environment state transitions, and launches new policies when the current model is inaccurate predicting the state trajectory based on the CUSUM algorithm (PAGE 1954.
The underlying learning approach is SAC (Haarnoja et al. 2018b).
MBCD's s sensitivity for detecting environment changes is tunable hyperparameter; we tuned it by trying 6 values in logarithmic space spanning 10t to 10th on the evaluation context traces with Pendulum-vl. and chose the best performing hyperparameter on the test set. MBCD still endlessly spawned new policies for other environments, and therefore limited the maximum number of models to 7. Despite this, MBCD fails to perform well over the diverse set of contexts. MBCD struggles to tease out meaningful task boundaries. In some experiments, it launches anywhere between to 7 policies just by changing the random seed. This observation is in line with MBCD's sensitivity in $4.1
D.5 MBPO
MBPO (Janner al.) 2021) is model-based approach that trains ensemble of experts to learn the environment model, and generates samples for an SAC (Haarnoja et al. 2018b) algorithm. If the model is accurate, it can fully replay prior contexts, thereby avoiding catastrophic forgetting.
MBPO performed poorly. If the fault is the accuracy of the learned environment models, could be improved with approaches such online meta-learnigg (Finn et al. 2019) or goal-oriented model-based learning (Pong et al. 2020).. We investigated the learned models for the Pendulum-vl experiments manually. We found these models to be very accurate, since the environment dynamics are simple.
24

---

Published as a conference paper at ICLR 2025
To concretely verify that the accuracy of MBPO learned models is not the reason it underperforms, instantiated an MBPO agent with access to the ground truth environment dynamics and context traces (but not future context traces), which we call Ideal MBPO. We compare the performance of Ideal MBPO vs. standard MBPO in 3] The performance of these two agents is widely similar. This confirms the fact that the learning algorithm itself is the problem, and not the learned models.
Table 3: Average and 95th percentile confidence ranges for lifelong returns for different algorithms and conditions in the Pendulum-vl environment with external wind processes. An MBPO agent with access to the ground truth model performs similarly to the MBPO model. Schemes with superiority beyond 95% confidence are highlighted in bold.
Online Learning
LCPO
MBPO
MBPO
A2C
Prescient
-190
-325
-320
-201
Context Trace
-187 (SAC)
+0.45
+47.2
#44.2
#3.92
-355
-843
-870
-377
Context Trace
-376 (DDQN)
+2.57
+24.2
#36.4
$8.05
-240
-003
-637
-307
Context Taac
-203 (SAC)
#:441
+:213
=900
4:26
-378
-553
-555
-399
Context Trace
337 (SAC)
13.02
.1.90.9
117.76
The reason is the way that MBPO samples experiences for training. At every iteration, MBPO samples batch of actual interactions from its experience buffer and generates hypothetical interactions from them. These hypothetical interactions amass in a second buffer, which is used train an SAC agent. During the course of training, the second buffer uccumulates more interactions generated from samples from the start of the experiment compared to recent samples. This is not an issue when the problem is stationary, but in non-stationary RI. subject to an context process, this leads to over-emphasis of the context processes encountered earlier in the experiment. As such, MBPO fails to even surpass SAC. Prior work has observed the sensitivity of off-policy approaches to such sampling strategies (Isele & Cosgun 2018 Hamadanian ct al. 2022).
D.6 CLEAR
This approach aims to mitigate CF with off-policy learning and maintain quick adaptation with on- policy leaming (Rolnick et al. 2019). They use IMPALA and V-trace (Espeholt al. 2018) on recent batches for on-policy and stale batches for off-policy RL.
CLEAR (Rolnick et al. 2019) aims to mix the quick adaptation of on-policy RL and the CF resilience of off-policy RL, but in practice this fusion makes it very hyperparameter sensitive. The V-trace algorithm was originally intended to correct for lagging policies in distributed RL architecture (Espeholt al.) 2018). V-trace uses clipped importance sampling to correct for the drift between the logging and training policies, which reduces variance but biases the loss. With small lags between workers, the bias small. If V-trace used in a scenario where the logging and training policy are very different, such as in CLEAR (Rolnick) al. 2019), the bias becomes significant enough to hinder training. CLEAR attempts circumvent this by inducing regularization loss on the actor and critic. Yet, this regularization will count against improving the RL policy for better returns, and will be brittle. The correct balance between policy improvement and this regularization will ultimately depend on the environment and context trace. While we tuned CLEAR extensively for the Pendulum-vi environment, as we did for all baselines, fails catastrophically in other environments.
D.7 PT-DQN
PT-DQN (Anand & Precup 2023) learns two separate networks with two different goals; (1) a permanent network that is updated infrequently and slowly, and aims to learn a generalized estimate of Q-values in various tasks, and (2) transient network that leams and forgets aggressively, and aims to quickly leam the optimal policy for the current task. Anand & Precup (2023) prove PT-DQN asymptotically converges to the optimum predictors in piecewise stationary prediction tasks with tabular input spaces.
25

---

Published as a conference paper at ICLR 2025
Note that PT-DQN avoids catastrophic forgetting indirectly by slowly updating permanent Q-network. The hope is that it strikes the right balance in learning this network slowly enough such that earlier contexts not forgotten, but updates it frequently enough such that new knowledge is not lost. This trade-off is brittle; how fast the transient Q-network should forget and relearn, and how slowly the permanent Q-network should be updated highly depends on (1) how quickly the context process changes, and (2) by how much these changes affect the transition dynamics of the MDP. As was also observed with CLEAR ($D.6), balances of this nature are brittle, due to the indirect nature of how these techniques address catastrophic forgetting.
Thus, PT-DQN is understandly hyperparameter sensitive. Indeed, tune PT-DQN on Pendulum-vl v1 environment, (as done for all baselines), we carried out three rounds of grid-search on five hyperparameters, totalling 440 different combinations. Despite PT-DON being competitive with DDQN on Pendulum-vl, the performance is unpredictable in other environments.
D.8 OFF-POLICY RL
Off-policy RL is potentially capable of overcoming CF due to replay buffers, at the cost unstable training. We consider DDQN (Hasselt et al. 2016) and SAC (with automatic entropy regularization, similar to LCPO) (Haarnoja et al. 2018b).
D.9 ON-POLICY RI.
On-policy RL is susceptible CF, but more stable in online learning compared off-policy RL algorithms, and the fast adaptation of these algorithms can also help them 'track' the optimal policy as the environment changes (Sutton et al. 2007). We compare with A2C (Mnih et al., (016) and TRPO (single-path) (Schulman) et al. 2015), with GAE (Schulman et al. (018)) applied. Note that TRPO vine is not possible in online learning, as it requires rolling back the environment world.
D.10 PRESCIENT RL
To establish an upper-bound on the best performance that online learner achieve, we train prescient policies, as discussed in 2 We allow prescient policies to have unlimited access to the contexts and environment dynamics, i.e., they able replay any particular environment and context as many times as necessary. Since prescient policies can interact with multiple contexts in parallel during training, they do suffer from CF. In contrast, all other baselines (and LCPO) are only allowed to experience contexts sequentially as they occur over time and must adapt the policy on the go. We report results for the best of four prescient policies with the following model-free algorithms: A2C (Mnih al.] (0166) TRPO (single-path) (Schulman et al., 2015), DDQN (Hasselt et al. 2016 and SAC (Harnoja et al. 2018b).
APPENDIX E LCPO VARIANTS
In 4.2 we discussed our main approach to solving the constrained optimization problem below:
min LPC(O:B,)
(16)
s.l. D (0. < Canchor
The goal is to optimize the policy gradient loss, i.e., maximize returns current input, while minimizing policy change on past observed inputs with the anchoring constraint. We outlined solution this problem, with pseudo-code in Algorithm [] that is essentially a second order constrained optimization plus line search phase:
min (0-00)² UPG 8
(17)
s.1. (0 (0-6) TT) A(0-0n) 0n) ) S Canchor
where A,- a DK.(Ou.eW((..BB)) and uPG VoLro((;)lor However, there is another way de, solve this problem, that discuss as follows.
26

---

Published as a conference paper at ICLR 2025
E.1 LCPO-P
An alternative way to uphold the anchoring constraint is to directly add it as term in the loss function. Let define:
ancchor Lanchor(0,00;:B)) (.,
(18)
(B..BW(B..B.)) Be,p- (.E.., (s,2),
Where we use the Cross Entropy loss to incentivize policy anchoring. Then, optimize the following total loss:
min tK (000 (0,0o;
(19)
This approach is even less compute intensive than LCPO, but is not possible in vanilla policy gradient. This because the gradient direction from Lanchor is zero when 0= and will not affect the optimization. Therefore, we have to repeat the gradient update several times before this term has effect. The optimization setup in Proximal Policy Optimization (PPO) (Schulman et al. 2017) allows for several gradient steps with one batch of data, and therefore apply the above loss in the PPO optimization step. We dub this approach LCPO-P (P stands for proximal).
Table 4: Average and 95th percentile confidence ranges for lifelong returns for LCPO and LCPO-P and conditions the Pendulum-vl environment with external wind processes. Schemes with superiority beyond 95% confidence are highlighted in bold.
LCPO
LCPO-P
Prescient
-190
-211
-201 (A2C)
Context Trace
-187 (SAC)
1:045
+3.92
+3.92
-355
-388
-377 (A2C)
Context Trace
-376 (DDQN)
+2257
77.33
4:805
-240
-271
-262 (CLEAR)
Context Traco
-203
1:.41
+21
+:2.2
Context Trace
-378
-305
-399 (A2C)
-357 (SAC)
+3002
+9932
#7.76
We compare L.CO and LCPO-P in Table We tuned is on the Pendulum-vl environment, similar to all baselines, and finalized on k= 10. Despite this, LCPO-P fail to outperform the best baseline for each context trace even on Pendulum-vl. The stark contrast between LCPO and LCPO-P is due to the difficulty in tuning LCPO's optimization constraint guarantees that the policy is anchored past contexts, while the loss term added LCPO-P motivates for this change to be small. Although the setup in LCPO-P is the Lagrangian dual of the setup LCPO, they only equivalent when is tuned properly per each gradient step.
APPENDIX F LOCOMOTION TASKS IN GYMNASIUM
Context Trace
Context Trace 2
Context Trace
Context Trace 4
X-xiss
Applied force
Y-axis
Samples (x1 Million)
Figure 7: External wind force, per axis and context trace.
27

---

Published as a conference paper at ICLR 2025
F.1 FULL RESULTS
Tables and 6 presents the lifelong return for all agents, environments and context traces. Table 7 presents the lifelong return for multiple LCPO agents with different OOD thresholds a. Table 8 presents the lifelong return for multiple LCPO agents with different buffer sizes na.
Table 5: Average and 95th percentile confidence ranges for lifelong return for different algorithms and conditions environments with external wind processes. Schemes with superiority beyond 95% confidence highlighted in bold. Continued in Table
Online Learning
Prescient Policies
LCPO
Online EWC
Sliding OGD
CLEAR
MBCD
MBPО
A2C
TRPO
DDQN
SAC
-190
-204
-1195
-215
-207
-325
Trace
:10.45
12.37
14.38
10.42
233
1:472
2008
-197
-188
-187
Pendalum
Trace
4:2.77
-355
4:226
-525
#325
-1232
1:158
-229
7:13
-577
4:242
-84
-007
-399
-376
-380
-240
-345
-1181
-262
-349
-603
Trace
1:4.1
1:29.
23.78
1:220
2:28
4:213
224
-320
-228
-203
-378
412
-1233
.111
-430
-553
Trace
3383
-381
-384
-357
+:302
1:652
-4.1
22.45
155
+:00
Trace
18.7
3,47
2.21
2.28
6.17
2.88
24.4
25.7
24.1
24.8
22.85
+0.32
=001
10.10
=6.10
+1.7
Inverse Pendukim
1.55
1.58
1.52
Trace
+0.54
+0.03
=0.00
+0.03
1106
+:0.01
7.86
8.51
7.18
7.75
15.4
5.56
4.24
2.45
14.2
14.4
#:0.22
+:088
-0.02
10.36
0077
+:0.3
16.5
14.3
92.4
5.43
4.64
2,87
46.5
37.6
Trace
1:0.19
1:.88
1:017
1:0.8
102
112
53.7
40.1
114
38.8
38.5
34.2
77.3
68.5
Inverse Douhle Pendalum
Trace
#:354
#22.5
=0.10
::0.46
13.5
1:19
142
165
88.0
95.5
56.2
30.1
30.9
28.9
44.2
37.2
Trace
#:2.28
+1003
0003
+:020
33.27
1:9.2
51.4
64.0
46.8
49.5
65.1
36.1
36.4
27.1
45.8
42.7
Trace
+:0.98
+1.80
0006
1:0.66
2:338
1:1.55
63.5
74.0
47.4
Trace
=3.33
89.2
+3.22
37.7
2007
41.4
+0.80
29.3
4.66
75.0
12866
59.5
91.9
96.9
90.7
78.0
153
Trace
240
215
155
184
+5558
+5505
0042
+:0.54
333.2
+:128
144
28.2
12.0
287
76.0
66.0
Trace
Hopper
13..4
14.98
20.53
10.25
1533
118.
Trace
508
10.4
27.0
19.3
10.31
3,62
281.
240
1:49
264
520
492
327
308
296
22.2
12.8
2.94
127
205
Trace 4
+22.9
#:847
=038
+0.10
27.3
+:37.5
319
297
189
189
-26.5
-16.3
4622
-1780
-169
-1862
Trace
-8.00
-7.59
-7.59
-143
+00.11
10060
1224
1958
122
+:2258
-19.8
-16.4
-6733
-1695
-339
-4308
Trace
-8.33
3366
-7.41
-153
Reacher
+:0.11
1:0.77
=74.0
1731
4:4
1:376
-27.3
-15.4
-6722
-709
-596
-2058
Trace
4:0.9
+:023
+161
2:57
+-4173
-9.63
-8.42
-8.1
-15.7
-29.4
.16.6
.1392
121
.2344
Trace
+:0.29
+0.44
102
1000
111
4.298
-9.63
-8.57
-7997
-15.7
F.2 COMPUTATIONAL. LOAD
LCPO is about 1.5 x more computationally demanding than the leading baseline A2C. Table 9 depicts the total runtime per each environment interaction in the experiments in $5.1
28

---

Published as a conference paper at ICLR 2025
Table 6: Average and 95th percentile confidence ranges for lifelong return for different algorithms and conditions in environments with external wind processes. Schemes with superiority beyond 95% confidence are highlighted in bold. Continued from Table
Ouline Learning
Prescient Policies
LCPO
A2C
TRPO
BFDQN
PT-DON
DDQN
SAC
A2C
TRPO
DDQN
SAC
-190
-201
-222
.346
-205
-219
-207
Trace
-208
-197
-187
#:045
+3.92
+4.14
+15.0
#2.80
#3.88
+:.12
-355
-377
-448
-709
-85
-756
-636
Penhhlum
Trace
+:2.77
+:805
+:1.3
+64.9
+103
+623
-007
-399
-376
-380
Trace
1.4.1
-240
1.326
-307
163.4
-511
175.1
-766
1.795
-596
1.7622
-612
1.10
-402
224
-320
-228
-203
-378
-488
Trace
1.300
1.776
1.114
1.53.3
1.32.0
1.7.99
1.5.52
383
381
334
-357
Trace
1.2.85
18.7
1.2.1
7.57
1..84
4.61
11.13
5.79
1:2.27
13.6
1.338
12.7
1.0.67
3:97
24.8
Inverse Pendifum
5.31
2.36
1.73
1.74
2.24
1.60
1.59
Trace
7,86
8.51
7.18
7.75
1.0.4
Trace
15.4
7.94
4.28
15.5
16.5
15.1
14.3
+:0.2
+0.46
+2.36
+1.58
+0.17
#0.14
+0005
92.4
93.7
75.9
79.6
75.5
92.1
50.5
Trace
1:019
+0021
10.6
+288
#3.18
1.18
1..4
102
112
114
118
86.3
48.7
76.6
80.0
75.6
Inverse Doable Penshlum
Trace
+3.54
+9.47
+139
+7.99
+1.91
#0.95
+5.67
142
165
88.0
95.5
Trace
56.2
54.3
30.0
25.6
43.6
42.4
38.8
5144
64.0
46.8
49.5
#2.28
+..7
+3.60
+0.08
+1.52
1.17
+2.37
Trace
1.0.9
1.2.37
1.3.93
1:4.9
1.0.99
1:0.26
1.0.23
63.5
74.0
63.6
47.4
89.2
87.9
75.9
67.6
93.9
93.9
78.7
Trace
1.0.33
1.053
1:1.8
14.37
1.0.6
10.18
1:063
244
203
164
93.5
86.0
136
154
Trace
1.7.25
1.3.44
1.14.1
240
215
155
184
Happer
Trace
1.3.44
144
1.3.0
97.9
1.911
65.8
1..85
23.2
1:743
30.0
1.4.1
68.1
1.5.69
81.7
128
131
86.6
96.7
508
484
443
137
52.2
220
360
Trace
+:254
1338
117.1
+12
+6.63
#6.15
+:39.2
520
492
296
263
186
105
117
161
258
Trace
319
297
189
189
+:2.59
+5.50
144
16.14
113.7
48002
144.
Trace
-26.5
-118
-2346
.9.1
2492
-7.84
-370
-8.00
77.59
-7.59
-14.3
=:0.21
+445.9
-:471
20.15
1198
+0.03
160
-19.8
.4728
Reacher
Trace
1:0.11
3346
#:15
+3.52
+17.
#0.07
+:20
-8.33
3366
-7.41
-15.3
-27.3
-133
-71.7
-10.1
-9116
-8.56
-258
Trace
1:0.89
1:615
1800
10.32
1:2335
10.04
1:140
-29.4
-117
-1780
-9.17
-23.2
-9.21
-157
Trace
1:56.3
1272
10.45
1:596
-9.63
-8.57
-7.97
-15.7
F.3
ACTION SPACE
Pendulum-vl and Mujoco environments by default have continuous action spaces. We observed instability while learning policies with continuous policy classes even for the prescient policies, and were concemeed about how this can affect the validity of our online experiments, which are considerably more challenging. As the action space is tangent to problem, atoms, spaced equally from the minimum to the maximum action in each dimension. This stabilized training greatly, and is not surprising, as past work (Tang & Agrawal 2019) supports this observation. The reward metric, continuous state space and truncation and termination conditions remain unchanged.
discretized each dimension of the action space 15
We provide the achieved episodic return for all baselines in $ in Table 10 over 10 seeds for the Pendulum- vI envioronment, which we can compare to SB3 (Rathin et al. 2021) and RL-Zoo (Raftin 2020) reported figures. These experiments finished in approximately 46 minutes. A2C, DDQN and SAC were trained for
29

---

Published as a conference paper at ICLR 2025
Table 7: Average and 95th percentile confidence ranges for lifelong return in LCPO with different OOD metrics and thresholds and other agents for different conditions in environments with external wind processes. L.2 stands for the L.2-disance OOD metric and MHD stands for the Mahalanobis distance OOD metric. Schemes with superiority beyond 95% confidence are highlighted in bold (LCPO agents are only compared against baselines).
LCPO (1.2)
I.POO (1.2)
=0255
d²=0.3 0.5
===1
=²=2
d²
Min
Mum 12
Best Baseline
Best Prescient
Trace
+:0.22
-189
1:066
-190
1045
-190
1062
-194
10.32
-199
10.50
-188
20.53
-188
-201 (A2C)
13.92
-187 (SAC)
-349
.354
.355
.359
.361
-366
-346
-377 (A2C)
Pandalum
+2.12
22.68
#2.57
#2.64
+3.03
11.6
2249
#8.05
-376 (DDQN)
Trace
-242
-242
-240
.255
-231
-233
-33
.22 (CLEAR)
-203 (SAC)
+5.34
#3.94
44.13
+4.39
=4.00
#2.20
Trace 4
-375
-377
-378
-380
-385
-369
-373
-399 (A2C)
-357 (SAC)
+2.76
#:412
+3.02
#2.99
22778
+3.45
#3.07
#7.76
(PT:DDO))
Trace
25.7 (TRPO)
+1.72
+3.03
#2.85
13.13
+0,16
22.64
#2.96
#2.27
Inverse Pendalum
4.85
4.24
5.31
5,35
2.79
5.12
5.28
(A2C)
(TRPO)
1:0.0
10.71
10.54
10.56
10071
10068
2062
1004
15.1
15.6
15.4
15.4
15.4
16.3
16.3
1A,0
Trace 3
16.5 (TRPO)
1:036
10.13
+022
20.23
10023
10.03
20.4
10466
(A2CC)
Trace +0024
112 (TRPO)
+0.19
+0.19
+0.19
+0019
1:225
+0339
#0,21
114
118
118
223
221
118 (A2C)
Inverse Double Pendalum
Trace 1:5.26
1:.26
#3.54
#:526
+5.26
1:5.39
66.5
#9447
165 (TRPO)
Trace
53.4
$4,2
42.28
56.2
22.23
54,9
+2.23
54.9
4:295
62.9
4..4
63.3
$4.3 (A2C)
1.72
64.0 (TRPO)
Trace 3
65.1
64.9
64.9
62.5
64.5
(A2C)))))
74.0 (TRPO)
+:0.2
10.1
+098
20.91
-0.91
14.07
#341
#2.37
89.4
89.4
89.7
(DDQN) 180.18
Trace +:0.7
96.9 (TRPO)
+0.24
+0.33
20.24
-0024
+0.13
20.14
247
250
244
235
222
245
252
203 (A2C)
(A2C)
16696
18.38
15.58
-2.81
+3.49
1:102
28.1
17.73
144
144
144
139
138
147
146
97.9 (A2C)
Trace 2
131 (TRPO)
Happer
43.48
#:320
+3.44
#3.14
+:3.50
+:3.59
33.79
$3007
508
484 (A2C)
Trace 3
(20C)
+4.01
+2.46
#2.54
#244
#2446
1:8.78
=6.00
13.8
302
298
296
292
291
300
298
263 (A2C)
Tiace 4
319 (A2C)
+:3.6
4:322
+2.59
#2004
+4.09
+:396
-288
#:5.59
-26.7
-2655
-265
-24.5
-21.6
-25,1
-25.1
-7.84 (DDQN) 1:003
Trace
-7.59 (DDQN)
1:0188
+0.19
#021
+0.32
=0019
4:012
-0.09
Trace 2
22.55
.22.6
.198
-1988
-1.88
.3.9
-3.8
-7.78 (DDQN) 1007
-7.1 (DDQN)
Roucher
1:0.1
10.13
20.11
20.12
20.12
+0.23
-0.22
-2544
.8.56 (DDQN) 10.04
Trace +1.07
8.41 (DDQN)
+1.07
+0.89
+0.62
20.67
+0.82
20.56
-30.6
-30.4
-29.4
-27.7
-26.0
-24.6
-24.2
-9.17 (BFDQN) 10.55
-7.7 (DDQN)
1:0.29
10.36
1.029
-0037
-0.34
110.19
-0.23
8000 epochs, and TRPO was trained for 500 epochs. Evaluations 1000 episodes. As these results show, the agents exhibit stable training with a discretized action space.
F.4 EXPERIMENT SETUP
We Gymnasium (v0.29.1. MIT license) and Mujoco (v3.1.1, Apache-2.0 license). Our baseline and LCPO implementations use the Pytorch (Paszke et al. 2019) (31.13.1, BSD-style license) library. Table I comprehensive list of all hyperparameters used training and the environment.
All baselines were tuned on Pendulum-vl via a multi mhalti-phased grid search, similar to that in DD. General parameters such discount horizon were copied from the base RI. algorithm each baseline using (e.g..
30

---

Published as a conference paper at ICLR 2025
Table 8: Average and 95th percentile confidence ranges for lifelong return in LCPO with multiple buffer sizes and other agents for different conditions in environments with external wind processes. Schemes with superiority beyond 95% confidence are highlighted in bold (LCPO agents are only compared against baselines).
76
20.M
Na
200K
76
Thy my=20
Best Baseline
Best Prescient
Trace
-191
-190
-190
-193
-201 (A2C)
-187 (SAC)
+0.65
+0.45
+0.6
+0.99
#3.92
-355
-356
3377 (A3C)
Pendalunt
Trace
+2.00
+2.57
+2.01
+3.57
#8.05
-376 (DDQN)
-240
-24
-240
-259
-262 (CLFAR)
Trace
+:683
+4.1
+9.10
+10.7
+:220
-203-(SAC)
-379
-378
-379
-400
-399 (A2C)
Trace
1.3288
1.3.88
1.7.76
(SAC)
Trace
18.4
18.7
20.8
14.7
13.6 (PT-DQN)
25.7 (TRPO)
12554
12.85
11.72
1.348
1.2.77
Inverse Pandalum
4.69
5.31
4.91
4.61
(A2C) 10.45
Trace
10.66
10.54
10.83
1.0.7
8.1 (TRPO)
15.4
15.4
15.4
15.4
15.0 (A2C)
Trace
16.5 (TRPO)
1.023
1:0.22
1.0.2
1:0.3
1:0.46
Trace
92.3
92.4
92.4
92.4
93.7 (A2C)
(2R
+0222
#0.19
+0.19
+9.19
=:021
117
118 (A2C)
loverse Double Pendalun
Trace
+:530
13.54
4:5.29
1:5.26
1:9.47
165 (TRPO)
54.6
56.2
54.7
54.9
54.3 (A2C)
Trace
(TRPO)
#:2.9
+2.28
+2.19
+:2.33
11772
Trace
64.9
65.1
65.0
64.9
63.4 (A2C)
(TRP
+:090
+00.8
+0.92
+0091
+:2.37
89.4
89.2
89.4
89.4
93.9 (DDQN)
Trace
10.24
1:0.33
10024
1:0.44
110.18
96.9 (IRPO)
238
233
Trace
(AZC)
+:4445
+:5.88
+5.37
+3.22
+7.73
141
144
139
135
97.9 (A2C)
Trace
131
Happer
1:4.13
13.44
1.3.76
1:2.54
1:3.07
509
508
500
508
484 (A2C)
Trace
2:4888
12.54
1:2.42
1:2.46
1:13.8
520 (AZC)
296
291
263 (AZC)
Trace
(A2C))
+22.48
+2.59
+:423
+5.55
#5.59
-2.66
2665
-26.5
-2644
-7.84 (DDON)
(DION))
10.15
10.21
1.0.9
1:0.66
1:0.03
.1988
-198
-1988
-19.8
-7.78 (DDQN) +:000
Trace
-741 (DDON)
Reacher
10.12
10.11
+0.12
+0012
-27.8
-27.3
-27.8
-28.1
-8.56 (DDQN) 1:0.04
Trace
83.4 (DDQN)
+0.84
+:0.9
+0097
+:0.3
-29.5
-29.4
-29.3
-9.17 (BFDQN)
Trace
10.31
:10.29
-10.73
+008
1:0.45
-797 (DIXNN)
Table 9: Average and 95% confidence interval for training time per environment interactions, in psec.
Sliding OGD
Online EWC
LCPOO
A2C
TRPO
DDQN
SAC
CLEAR
BFDQN
MBPO
MBCD
FT-DQN
0.78
0.52
0.45
045
0.50
0.65
0.43 +0.01
0.31
2,19
5.62
3.97
0.42
40.03
+0.03
+0.03
+0.03
40.02
+0.01
4001
2004
+0.32
+0002
2001
online EWC is using SAC, and copied SAC-specific parameters directly). Several LCPO hyperparameters were copied from TRPO, SAC and A2C (namely, entropy target, entropy learning rate, damping coefficient, rollout length, X. 7) and the rest (Conchors Crecent and base entropy) were tuned with informal search with a separate context trace (not in the evaluation set) in Pendulum-vl. The OOD threshold o was not tuned with search.
31

---

Published as a conference paper at ICLR 2025
Table 10: Average episodic returns and 95th percentile confidence ranges for different algorithms in the Pendulum-vl environment with discretized and continuous action space.
Episodic
TRIO
DDQN
MBPO
Discrete
-664+5
-166+-8
-199+2
-146+-2
-161+-3
SB3 Raffin et al. 20211
RL-Zoo
Raffin 2020
-203
-224
-176
Table 11: Training setup and hyperparameters for gymnasium environments with external wind.
Group
Hy perpaaaraeee
Value
Hidden layers
(64, 64)
Hidden layer activation
Relu
Neural nerwork
Output layer activation
Actors: Softmax, Critics DDQN: Identity mapping
Optimizer
(8) 0.9. 0.999) (Kingma 2017)
Learning rate
Actor: 0.0004, Critie DDQN: 0.001
Weight decay
10-1
25 in main experiments (5, excopt for MBPO and MBCD seeds in ablations (55
Random scads
D.
RL training (general)
À (for GAE in A2C and TRPO)
0.9
0.99
A2C
Rollout epoch
200
Rollout per epoch
3200
TRPO
Damping coefficient
Stepaize
Rollout per epoch
200
Batch Size
512
Initial fully random period
1000 spochs
DDQN
e-greedy schedale
to 5000 epochs
Polyak
0.01
Bulfer
samples 200 88M
Rollout per epoch
200
Baich Size
512
Initial fully random period
1000 epochs
Base Entrupy
0.1
Entropy Target
0.1ln(15)
Log Entropy Leuring Rate
le:3
Polyak
Buffer
samples 20M -8M)
Rollout per epoch
200
Base Entropy
0.03
Entropy Target
0.1In(15)
Log-Entropy Leaming Rate
Io-3
LCPO
Buffer Size
1% of samples (2XXJ 8DK)
Damping cocfticient
0.1
Conchor
Crecent
0.1
PPO Clipping
0.2
LCPO.P
PPO Ieerations (Max)
30
PPO Max KL
0.01
10
32

---

Published as a conference paper at ICLR 2025
Group
Hyperparameter
Value
h
1000 (defauli 1003300)
MBCD
max_sdd
(default
(ensemblle size)
NN hidden layers
(64, 64, 64)
M (model rollouns)
512
MBPO
N (ensemblle size)
5
k (rollout length)
Gi (gradient steps)
averaging weight
0,00007 (oquivalent ~.3M samples rollou-2000)
Online EWC
scaling
learning rale
le.4
Sliding OGD
N (window sine)
1000 spisodes
Value Clone Coefficient
le.2
Policy Clone Coefficient
1e:3
CLRARR
Entropy Coefficient
Se.3
Depth
BEDQN
Benna Fusi Buffer Length
2000
le:3
Permancut Learning Rate
le-5
Transient Learning Rate
Io-3
PT-DQN
Target Update Period
2000 steps
Pernament Update period K
steps
Transient Forget Factor
0.999
APPENDIX G STRAGGLER MITIGATION
Context Trace
Context Trace
5o
200
Ensirommntt
Anrival
Processing Time
100
25
Agrnnt
200
Time (hrs)
(a) Illustration of request proxy with hedging.
(b) Request arrival rate and processing time per input.
Figure 8: Request arrival rate and processing time per input.
G.1 FULL RESULTS
Figure plots the tail latency across experiment time the straggler mitigation environment for both contexts.
33

---

Published as a conference paper at ICLR 2025
(TIPP))
(APC)
(ADC)
Context 1
Context
latency
1000
500
1500
2000
Time (hrs)
Figure 9: Tail latency with 95th percentile confidence intervals as training progresses (lower is better). We consider an initial learning period of 3.5 million samples. ICPO remains close to the prescient throughout contexts, while baselines suffer from non-stationarity
G.2 EXPERIMENT SETUP
We use the straggler mitigation environment from prior work (Hamadanian et al. 2022) with a similar
configuration except with 9 actions (timeouts of 600 and 1000 added). Similar to $F.4 our implemen- tations of baselines and LCPO use the Pytorch (Paszke et al. (019) (v1.13.1, BSD--ttye license) library. The environment code and dataset not public and released is a comprehensive list of all hyperparameters used in training and the environment.
with proprietary license. Table T2
All baselines were tuned separate workload. LCPO hyperparameters were copied from the gymnasium experiments, except for base entropy which was tuned with an informal search with separate workload (not in the evaluation set).
Table 12: Training setup and hyperparameters for straggler mitigation experiments.
Group
Hyperparameter
Value
netwoek: (32, 16)
Hidden layers
p network: (32, 32)
Hidden layer activation function
Noural network
Ouipul layer uctivation function
Aciors: Softmux, Critis and DDQN: Identily mupping
Optimizer
Adam (8, 0.9,, /9g 0.900) Kingrna & Ba 2017
Leuring rall
0.001
Weight decay
10
Ramdon
training (geuerall
(for GAE A2C and TRPO)
0.95
7
0.9
A2C
Rollout epoch
4608
Rollout epoch
10240
TRPO
Damping coefficient
0.1
Stepsize
0.01
Rollout per epoch
128
Batch Size
512
Initial fully mindam period
1000 epachs
DDQN
ggreedy schedule
1 to 0 in 5000 epocts
Polyak
0.01
Buffer size N
All samplex 21
34

---

Published as a conference paper at ICLR 2025
Group
Hyperparameter
Value
Rollout epoch
128
fully mindom period
1000 epochs
SAC
Base Entropy
0.01
Entropy Target
0.
Log. Entropy Learning Rute
le.3
Polyak
0.005
Buffer size N
All samplex 22
per epoch
Entropy
Entropy Tirget
0.1lu(9)
Log. Entropy Learning Rute
le.3
LCPO
Buffer Size
210K
Damping cnefficient
0.1
Canchor
0.0001
Grecent
0.1
(default
MBCD
max_sid
(detault was 0.5)
N (ensemale sizze
5
NN hädden layes
(64, 64, 64)
M (model pollouts)
512
MBPО
N (ensemble size)
5
(rollout length)
(gradient steps)
averaging weight (3
0,00007 (oquivalent 2M samples at rollout=128)
Online EWC
scaling factor a
35


Reviews:
[{"review_id": "AcTfiMGws0", "reviewer": "Reviewer_JEGP", "summary": "The paper proposes Locally Constrained Policy Optimization (LCPO), a policy gradient algorithm for non-stationary reinforcement learning where the context changes are observed and exogenous. LCPO’s updates are constrained to preserve the policy trained on the past contexts while it maximizes returns on newer samples. Two replay buffers are maintained in order to store the samples belonging to the past contexts and the current context. Whenever the distribution of samples is different in these two buffers, which is detected using an OOD threshold-based detector, then a “regularization” update is performed to preserve past learnings. The proposed approach is tested on a range of environments with a whole suite of baselines to demonstrate its benefits.", "questions": "**Decision:**\n\nAlthough the paper has some weaknesses, the positives outweigh them. The paper is well-executed and it is a good contribution to continual reinforcement learning. Therefore, I recommend an **acceptance.**\n\n**Areas of improvement:**\n\n* The paper should discuss how [1] and [2] are related to the proposed approach. [1] uses a KL constraint on the current policy to be closer to the global policy, and [2] learns a global value function as a baseline estimate for all contexts on top of which the learning happens. It seems like the global policy and the global value functions induce the regularization effect on the past that the paper proposes.\n* In Sec 4, the episodic returns are -4 and -6 and not -3 and -5 as mentioned in the paper.\nThe ablation experiments and the results presented in the appendix can be summarized as one-line bullet points in the main paper.\n\n**Questions:**\n\n* Deep neural networks are known to overfit to the early samples [3], wouldn’t regularizing with respect to the past data exacerbate this process?\n* How is the context information different from a task ID?\n* In Sec 4, does the A2C use both context and state information to learn the policy?\n* Is it possible to approximate the proposed objective in the PPO style as opposed to the TRPO style to avoid using second-order information?\n* What happens if the reverse KL is used in the constraint proposed in Eq. 1?\n* What is a warm-up period and why is it needed in the experiments?\n\n**References:**\n\n[1] Teh, Yee, et al. \"Distral: Robust multitask reinforcement learning.\" Advances in neural information processing systems 30 (2017).\n\n[2] Anand, Nishanth, and Doina Precup. \"Prediction and control in continual reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Nikishin, Evgenii, et al. \"The primacy bias in deep reinforcement learning.\" International conference on machine learning. PMLR, 2022.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and easy to read: the introduction is thorough, the related works section is exhaustive, preliminary and the LCPO sections provide adequate details, and the methodology section is well-presented.\n* The specific kind of non-stationary problem that the paper considers is well-described. And, the proposed solution fits like a glove to this problem. The approach is also intuitive and parallels the TRPO algorithm.\n* The authors evaluate their approach against numerous baselines on several tasks, which provides substantial confidence in the presented results and conclusions. The ablation studies are also welcome, as they offer more insights into the proposed approach.\n* The illustrative examples discussed throughout the paper offer an intuitive understanding of various complex ideas – the authors have done a great job of including them.", "weaknesses": "* The biggest weakness is that the paper considers a specific kind of non-stationarity in the study. Although exogenous processes could be common in real-world scenarios, many other kinds of non-stationarities exist and it is unclear whether the proposed approach works well in those cases. Besides, the OOD detector is an important piece in the algorithm and its effectiveness depends on the threshold value – a hyperparameter that is environment-dependent and unknown beforehand.\n* The approach also assumes that the context is observed, although it is not used as a task detector or task boundary detector. In many practical cases, the context information is unavailable and it is unclear how one could use the proposed approach in that case.\n* The proposed constrained optimization problem is approximately solved, which in itself is fine, but when paired with the fact that the approach doesn’t use TRPO-style local constraint, the training procedure could be unstable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LRKUWY7Ntv", "reviewer": "Reviewer_ksTz", "summary": "The paper addresses \"catastrophic forgetting\" in continual reinforcement learning with non-stationary dynamics. The dynamics depends on the current context which is observable but can change arbitrarily. A problem in these kind of settings is that the agent over time forgets how to behave in prior contexts. To address this problem the paper proposes a new on-policy algorithm that aims to keep policy updates such that the policy for prior contexts does not change. The proposed approach does not need task labels but only an out-of-distribution detector.", "questions": "The proposed algorithm places a constraint that aims to keep the new policy relevant for old samples. This constraint replaces the TRPO policy update constraint. The original TRPO formulation results in an approximate natural gradient due to which in TRPO the gradient update does not in principle depend on the parameterization of the (neural network) policy. This property is lost here? Do we get other properties?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Well written paper. Clear definitions and derivations. The paper explains well all the design decisions in the proposed approach.\n\nThe main algorithmic idea of sampling states from the history buffer that differ from the samples currently collected from the environment, and, then using these samples to constrain the policy update is simple and based on the experimental results appears very effective.\n\nI like that the resulting algorithm is an almost straight forward modification of TRPO but for a completely different problem. Note that while being simple the main idea is only obvious in hindsight.", "weaknesses": "I could not find major weaknesses in the paper.\n\nThis is not a weakness but just a comment:\nChoosing of the out of distribution (OOD) detector is one obvious design decision when running the method in a specific environment. Section 6.2 discusses different thresholds for the OOD detector. Although OOD detection has been discussed in the supervised learning literature, the paper could provide a more thorough discussion on which kind of OOD detectors could be beneficial in which kind of RL environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r1W5u3aBCQ", "reviewer": "Reviewer_dCoM", "summary": "The paper introduces an online RL method for environments with non-stationary (but observable) context, which affects the dynamics/reward. The main difficulty in such settings is catastrophic forgetting; while training the agent on an unobserved context, it forgets the optimal behavior in past contexts. The authors aim to solve this issue by constraining the agent to maintain current behavior for previously-observed contexts, which is possible due to an OOD detector -- basically a distance measure over contexts. To test the proposed method, the authors use a single toy problem and a large amount of benchmarks, the results show that the method indeed surpasses other baselines.", "questions": "1. Have you tried learning the OOD proxy? e.g., using weighted norm with tunable parameters?\n\n2. You do discuss the differences between your proposed OOD detector and CPD, but have you tried comparing the \"whole package\"? i.e. using LCPO with CPD.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the paper is well-written, easy to understand and comprehensive. Notably, the empirical sections present the advantages of the proposed algorithm. For my opinion, with a few minor changes, it passes the acceptance threshold.\n\n### Experiments\nThe experiments are extensive in terms of both benchmarks and baselines, and show good performance for the proposed method.\n\n### Illustrative Example:\nI think this empirical analysis helps to understand what's happening, and shows the value of the method.", "weaknesses": "**Paper exceeds the 10-page limit**\n\n### Presentation\n1. The references within the paper seem to have improper notation (using § instead of Sec.) \n\n2. I am not sure about the contribution of the opening sentence in the introduction, as this is not the first paper introducing CF, and I do not see any specific relation to the proposed method.\n\n3. Since the method uses a few components, adding a block diagram could help, but is not mandatory in the main body. Such block diagram should show the interconnections at each time step between the buffer, new sampled data, OOD detector, and the agent (policy).\n\n### Related Work:\nWhile not exactly the same, maybe contextual MDPs could be related.\n\n### Minor comments:\n\n1. line 66: \"These assumptions are rarely met and lead to poor performance in practice.\" -- is there any reference to support this claim? If it cannot be backed, you can alter the sentence to be more soft: \"...and would likely lead to...\".\n\n2. line 268: \"proof in §B in the Appendix.\" -- rephrase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qYj8a4eUZI", "reviewer": "Reviewer_QiEy", "summary": "This paper investigates online reinforcement learning (RL) in non-stationary environments, \nwhere the non-stationarity arises from a time-varying exogenous context process affecting the dynamics.\nIn particular, they study the catastrophic forgetting (CF) problem.\nThey propose a new method named locally constrained policy optimization (LCPO),\nwhich combats CF by constraining policy updates for a particular context on prior data from other contexts.\nThey evaluate their method in the classic control tasks from MuJoCo and computer systems environments.", "questions": "- How can you improve the clarity of the writing?\n- How am I supposed to read Figure 3. What does the y-axis represent?\n- Why have you used textual citations everywhere?\n- What does the bolding in tables represent?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Whilst this paper is slightly out of my expertise, I think the authors are addressing an important problem as\nhandling non-stationarity in online RL is important.\nTo the best of my knowledge, the method presented in this paper seems original.\n\nI particularly liked Figure 1. I would suggest the authors move it earlier in the paper.", "weaknesses": "Overall, I found the paper quite hard to follow. \nThe method seems fairly straightforward but the paper lacks a clear and easy-to-follow structure I would expect from an ICLR paper.\nI am confused as to why Section 4 \"Locally-Constrained Policy Optimization\" and Section 4 \"Methodology\" are different sections.\nI would recommend the authors combine these sections as they both explain the method. \nIf needed, use sub-sections to structure the content.\n\nFurther to this, the figures are not explained thoroughly enough in the text. What does the y-axis in Figures 3 and 4 represent? I assume it is the cumulative distribution function but this should be stated in the paper. \nFurther to this, how should the reader interpret the results?\nThe authors should explain how to interpret the plots, including what the steepness of curves represents, what does it mean\nwhen curves are intersecting, etc?\nI suggest the authors check out [rliable](https://github.com/google-research/rliable/tree/master) and use this to create figures.\nI am also not sure at what environment step these figures were created for.\nAgain, this should be clear from the text.\n\nWhy are there confidence intervals for Figure 3a but not for Figures 3b or 4?\nIn Section 6.3, the authors claim that buffer sizes less than $n_b=500$ results in a drop in performance.\nI am not sure you can make this claim without showing confidence intervals in Figure 4.\n\nThe authors have incorrectly cited throughout the paper. \nThe authors should take time to become familiar with when to use textual citations and when to use parenthetical citations.\nIf using the `natbib` package, the authors can use `\\citet` for textual citations and `\\citep` for parenthetical citations.\nThe authors almost always cite such that they should be using parenthetical citations but they use textual citations throughout.\n\nWhat does the bolding in the table represent?\nDoes it represent statistical significance under a t-test?\nThis should be clear from the caption.\n\n# Minor comments\n- \"Mujoco\" should be \"MuJoCo\"\n- Line 191 - the policy is stochastic but on line 112 it's a deterministic mapping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)