Online Reinforcement Learning in Non-Stationary Context-Driven Environments

## Abstract

Abstract We study online reinforcement learning (RL) in non-stationary environments, where a time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to “catastrophic forgetting” (CF). The agent tends to forget prior knowledge as it trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice) or use off-policy methods that suffer from instability and poor performance. We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems environments with a variety of synthetic and real context traces, and find that it outperforms state-of-the-art on-policy and off-policy RL methods in the non-stationary setting, while achieving results on-par with an “oracle” agent trained offline across all context traces.

## 1 Introduction

— Those who cannot remember the past are condemned to repeat it. (George Santayana, The Life of Reason, 1905)

Reinforcement Learning (RL) has seen success in many domains (Mao et al., ; Haarnoja et al., ; Mao et al., ; Marcus et al., ; Zhu et al., ; Haydari & Yılmaz, ), but real-world deployments have been rare. A major hurdle has been the gap between simulation and reality, where the environment simulators do not match the real-world dynamics. Thus, recent work has turned to applying RL in an online fashion, i.e. continuously training and using an agent in a live environment (Zhang et al., ; Gu et al., ).

While online RL is difficult in and of itself, it is particularly challenging in non-stationary environments (also known as continual RL (Khetarpal et al., )), where the characteristics of the environment change over time. A key challenge is Catastrophic Forgetting (CF) (McCloskey & Cohen, ). An agent based on function approximators like Neural Networks (NNs) tends to forget its past knowledge when training sequentially on new non-stationary data. On-policy RL algorithms (Sutton & Barto, ) are particularly vulnerable to CF in non-stationary environments, since these methods cannot retrain on stale data from prior experiences.

In this paper, we consider problems where the source of the non-stationarity is an observed exogenous context process that varies over time and exposes the agent to different environment dynamics. Such context-driven environments (Sinclair et al., ; Mao et al., ; Zhang et al., ; Dietterich et al., ; Pan et al., ) appear in a variety of applications. Examples include computer systems subject to incoming workloads (Mao et al., ), locomotion in environments with varying terrains and obstacles (Heess et al., ), robots subject to external forces (Pinto et al., ), and more. In contrast to most prior work, we do not restrict the context process to be discrete, piece-wise stationary or Markov.

Broadly speaking, there are two existing approaches to mitigate CF in online learning. One class of techniques are task-based (Rusu et al., ; Kirkpatrick et al., ; Schwarz et al., ; Farajtabar et al., ; Zeng et al., ). They assume explicit task labels that identify the different context distributions which the agent encounters over time.
Task labels make it easier to prevent the training for one context from affecting knowledge learned for other contexts.
In settings where task labels (or boundaries) are not available, a few proposals try to infer the task labels via self-supervised (Nagabandi et al., ) or Change-Point Detection (CPD) approaches (Padakandla et al., ; Alegre et al., ). These techniques, however, are brittle when the context processes are difficult to separate and task boundaries are not pronounced (Hamadanian et al., ). Our experiments show that erroneous task labels lead to poorly performing agents in such environments (§[6](#S6)).

A second category of approaches use off-policy learning (Sutton & Barto, ), which makes it possible to retrain on past data. These techniques (e.g., Experience Replay (Mnih et al., )) store prior experience data in a buffer and sample from the buffer randomly to train. Not only does this improve sample complexity, it sidesteps the pitfalls of sequential learning and prevents CF. However, off-policy methods come at the cost of increased hyper-parameter sensitivity and unstable training (Duan et al., ; Gu et al., ; Haarnoja et al., ). This brittleness is particularly catastrophic in an online setting, as we also observe in our experiments (§[6](#S6)).

We present LCPO (§[4](#S4)), an on-policy RL algorithm that “anchors” policy outputs for old contexts while optimizing for the current context. Unlike prior work, LCPO does not rely on task labels and only requires an Out-of-Distribution (OOD) detector, i.e., a function that recognizes old experiences that occurred in a sufficiently different context than the current one. LCPO maintains a bounded buffer of past experiences, similar to off-policy methods (Mnih et al., ). But as an on-policy approach, LCPO does not use stale experiences to optimize the policy. Instead, it uses past data to constrain the policy optimization on fresh data, such that the agent’s behavior does not change in other contexts.

We evaluate LCPO on several environments with real and synthetic contexts (§[6](#S6)), and show that it outperforms state-of-the-art on-policy, off-policy and model-based baselines in the online learning setting. We also compare against an “oracle agent” that is trained offline on the entire context distribution prior to deployment. The oracle agent does not suffer from CF. Among all the online methods, LCPO is the closest to this idealized baseline. Our ablation results show that LCPO is robust to variations in the OOD detector’s thresholds and works well with small experience buffer sizes.
The source code of our LCPO implementation is available online at [https://github.com/LCPO-RL/LCPO](https://github.com/LCPO-RL/LCPO).

## 2 Preliminaries

#### Notation.

We consider online reinforcement learning in a non-stationary context-driven Markov Decision Process (MDP), where the context is observed and exogenous.
Formally, at time step $t$ the environment has state $s_{t}\in\mathcal{S}$ and context $z_{t}\in\mathcal{Z}$. The agent takes action $a_{t}\in\mathcal{A}$ based on the observed state $s_{t}$ and context $z_{t}$, $a_{t}=\pi(s_{t},z_{t})$, and receives feedback in the form of a scalar reward $r_{t}=r(s_{t},z_{t},a_{t})$, where $r(\cdot,\cdot,\cdot):\mathcal{S}\times\mathcal{Z}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function. The environment’s state, the context, and the agent’s action determine the next state, $s_{t+1}$, according to a transition kernel, $T(s_{t+1}|s_{t},z_{t},a_{t})$. The context $z_{t}$ is an independent stochastic process, unaffected by states $s_{t}$ or actions $a_{t}$. Finally, $d_{0}$ defines the distribution over initial states ($s_{0}$). This model is fully defined by the tuple $\mathcal{M}=(\mathcal{S},\mathcal{Z},\mathcal{A},\{z_{t}\}_{t=1}^{\infty},T,d_{0},r)$. The goal of the agent is to optimize the *return*, i.e. a discounted sum of rewards $R_{0}=\sum_{t=0}^{\infty}\gamma^{t}r_{t}$.
This model is similar to the input-driven environment described in (Mao et al., ).

#### Non-stationary contexts.

The context $z_{t}$ impacts the environment dynamics and thus a non-stationary context process implies a non-stationary environment. We assume the context process can change arbitrarily: e.g., it can follow a predictable pattern, be i.i.d samples from some distribution, be a discrete process or a multi-dimensional continuous process, experience smooth or dramatic shifts, be piece-wise stationary, or include any mixture of the above. We have no prior knowledge of the context process, the environment dynamics, or access to an offline simulator.

While task labels can be considered contexts, contexts in general are not reducible to task labels since there may be no clear boundary between distinct context behaviors. Examples include time-varying market demand in a supply chain system, incoming request workloads in a virtual machine allocation problem, customer distributions in airline revenue management, etc (Sinclair et al., ).

Our framework is compatible with both infinite-horizon and episodic settings, and we consider both types of problems in our evaluation. In episodic settings, each new episode can differs from prior ones because the context has changed. An example is a robotic grasping task where the position and shape of the object changes in each episode.

#### Online RL .

In most RL settings, a policy is trained in a separate training phase. During testing, the policy is fixed and does not change. By contrast, online learning starts with the test phase, and the policy must reach and maintain optimality within this test phase. An important constraint in the online setting is that the agent gets to experience each interaction only once. There is no way to revisit past interactions and try a different action in the same context. This is a key distinction with training in a separate offline phase, where the agent can explore the same conditions many times. Naturally this means that the online learner must explore in the test phase, yielding temporarily low returns, and then enter an exploitation phase. This complicates the definition of a successful online learner. In this work, we primarily seek long-term performance, i.e., that the policy eventually attains high returns. For simplicity, we assume a short grace period in each online learning experiment that allows for exploration without penalty, and we rate the agent’s performance after the grace period.

## 3 Related Work

#### CF in Machine Learning (ML) and Deep RL .

Outside of RL, three general approaches exist for mitigating CF; (1) regularizing the optimization to avoid memory loss during sequential training (Kirkpatrick et al., ; Kaplanis et al., ; Farajtabar et al., ); (2) training separate parameters per task, and expanding/shrinking parameters as necessary (Rusu et al., ); (3) utilizing rehearsal mechanisms, i.e. retraining on original data or generative batches (Isele & Cosgun, ; Rolnick et al., ; Atkinson et al., ); or combinations of these techniques (Schwarz et al., ). For a detailed discussion of these approaches to CF, refer to (Parisi et al., ).

The first two groups require task labels, and often require each individual task to reach convergence before starting the next task. The RL answer to rehearsal is off-policy training (e.g., Experience Replay (Mnih et al., )). However, off-policy RL is empirically unstable and sensitive to hyperparameters due to bootstrapping and function apprroximation (Sutton & Barto, ), and is often outperformed by on-policy algorithms in online settings (Duan et al., ; Gu et al., ; Haarnoja et al., ).

#### Non-stationarity in RL .

A long line of work exists on non-stationary RL that explore sub-problems in the space, and a general solution has not been yet been proposed (Khetarpal et al., ). The most notable class of approaches aim to infer task labels, by learning the transition dynamics of the MDP, and detecting a new environment when a surprising sample is observed with respect to the learnt model (Doya et al., ; da Silva et al., ; Padakandla et al., ; Alegre et al., ; Nagabandi et al., ). These methods are effective when MDP transitions are abrupt and have well-defined boundaries, but prior work observes they are brittle and perform poorly in realistic environments with noisy and hard-to-distinguish non-stationarities (Hamadanian et al., ). In contrast to these papers, our work does not rely on estimating task labels and does not assume piece-wise stationarity. For further discussion on non-stationary RL, refer to (Khetarpal et al., ).

#### Latent Context Non-stationary.

Zintgraf et al. (); Caccia et al. (); He et al. (); Poiani et al. (); Chen et al. (); Feng et al. (); Ren et al. () study learning amidst non-stationarity due to latent contexts or hidden tasks.
In our setup, on the other hand, the non-stationarity in the environment is caused by *fully-observed* exogenous context.

#### Constrained Optimization.

LCPO’s constrained optimization formulation is structurally similar to Trust Region Policy Optimization (TRPO) (Schulman et al., ), despite our problem being different than TRPO.

## 4 Locally-Constrained Policy Optimization

Our goal is to learn a policy $\pi(\cdot,\cdot)$ that takes action $a_{t}\sim\pi(s_{t},z_{t})$, in a context-driven MDP characterized by an exogenous non-stationary context process.

#### Illustrative Example.

Consider a simple environment with a discrete context. In this grid-world problem depicted in [Figure 1](#S4.F1), the agent can move in 4 directions in a 2D grid, and incurs a base negative reward of $-1$ per step until it reaches the terminal exit state or fails to within $20$ steps. The grid can be in two modes; 1) ‘No Trap’ mode, where the center cell is empty, and 2) ‘Trap Active’ mode, where walking into the center cell incurs a return of $-10$. This environment mode is our discrete context and the source of non-stationarity in this simple example. The agent observes its current location and the context, i.e. whether the trap is on the grid or not in every episode (beginning from the start square).

Figure: (a)
Refer to caption: /html/2302.02182/assets/x1.png

#### Advantage Actor Critic (A2C) .

We use the A2C algorithm to train a policy for this environment, while its context changes every so often. When in ‘No Trap’ mode, the closest path passes through the center cell, and the best episodic return is $-3$. In the ‘Trap Active’ mode, the center cell’s penalty is too high, and we must go left at the blue cell instead, to incur an optimal episodic return of $-5$. [Figure 2(a)](#S4.F2.sf1) depicts the episodic return across time and [Figure 2(b)](#S4.F2.sf2) depicts what fraction of the time the policy makes the correct decision to go up in the blue cell when the context is in ‘No Trap’ mode. As observed, the agent initially learns the correct decision for that cell when training in the ‘No Trap’ mode, but once the context (mode) changes at epoch 4K, it immediately forgets it. When the mode changes back to the initial ‘No Trap’ mode at epoch 16K, the agent behaves sub-optimally (epochs 16K-18K) before recovering.

Figure: (a)
Refer to caption: /html/2302.02182/assets/x3.png

#### Key Insight.

Since the context $z_{t}$ is observed, the agent should be able to distinguish between different environment modes (even without an explicit task label). Therefore, if the agent could surgically modify its policy on the current state-context pairs $<s_{t},z_{t}>$ and leave all other state-context pairs unchanged, it would eventually learn a good policy for all contexts. In fact, tabular RL achieves this trivially in this finite discrete state-context space. To illustrate, we apply a tabular version of A2C: i.e., the policy and value networks are replaced with tables with separate rows for each state-context pair (18 total rows). As expected, in this case, the policy output, depicted in [Figure 2(b)](#S4.F2.sf2), does not forget to go up in the blue cell in the ‘No Trap’ mode after epoch 4K. This is because when an experience is used to update the table, it only updates the row pertaining to its own state and context, and does not change rows belonging to other contexts.

Can we achieve a similar behavior with neural network function approximators? In general, updating a neural network (say, a policy network) for certain state-context pairs will change the output for all state-context pairs, leading to CF. But if we could somehow “anchor” the output of the neural network on distinct prior state-context pairs (analogous to the cells in the tabular setting) while we update the relevant state-context pairs, then the neural network would not “forget”.

#### LCPO.

Achieving the aforementioned anchoring does not require task labels. We only need to know if two contexts $z_{i}$ and $z_{j}$ are different.
In particular, let the batch of recent environment interactions $<s_{t},z_{t},a_{t},r_{t}>$ be $B_{r}$ and let all previous interactions (from possibly different contexts) be $B_{a}$. Suppose we have a difference detector $W(B_{a},B_{r})$ that can be used to sample experiences from $B_{a}$ that are not from the same distribution as the samples in the recent batch $B_{r}$, i.e., the difference detector provides out-of-distribution (OOD) samples with respect to $B_{r}$. Then, when optimizing the policy for the current batch $B_{r}$, we can constrain the policy’s output on experiences sampled via $W(B_{a},B_{r})$ to not change (see §[5](#S5) for details). We name this approach Locally Constrained Policy Optimization (LCPO). The result for LCPO is presented in [Figure 2(b)](#S4.F2.sf2). While it does not retain its policy as perfectly as tabular A2C while in the ‘Trap Active’ mode, it does sufficiently well to recover near instantaneously upon the second switch at epoch 16K.

Figure: Figure 3: A sample context process $z_{t}$, and detected change-points for two thresholds. It is difficult to tease meaningful task boundaries from this process, but defining an OOD metric is intuitive.
Refer to caption: /html/2302.02182/assets/x5.png

#### Change-Point Detection (CPD) vs. OOD Detection.

CPD (and task labeling in general) requires stronger assumptions than OOD detection. The context process has to be piece-wise stationary to infer task labels and context changes must happen infrequently to be detectable. Furthermore, online CPD is sensitive to outliers. In contrast, OOD is akin to defining a distance metric on the context process and can be well-defined on any context process.
Consider the context process shown in [Figure 3](#S4.F3). We run this context process through a CPD algorithm (Alegre et al., ) for two different sensitivity factors $\sigma_{mbcd}$, and represent each detected change-point with a red vertical line. Not only does a slight increase in sensitivity lead to 34 detected change-points, but these change-points are also not reasonable. There is no obvious way to assign task labels for this smooth process and there aren’t clearly separated segments that can be defined as tasks. However, an intuitive OOD detection method is testing if the distance between $z_{i}$ and $z_{j}$ is larger than some threshold, i.e., $|z_{i}-z_{j}|>\tau$.
Altogether, OOD is considerably easier in practice compared to CPD. Note that although the grid-world example – and discrete context environments in general – is a good fit for CPD, this environment was purposefully simple to explain the insight behind LCPO.

## 5 Methodology

Consider a parameterized policy $\pi_{\theta}$ with parameters $\theta$. Our task is to choose a direction for changing $\theta$ such that it improves the expected return on the most recent batch of experiences $B_{r}$, while the policy is ‘anchored’ on prior samples with sufficiently distinct context distributions, $W(B_{a},B_{r})$.

In supervised learning, this anchoring is straightforward to perform, e.g. by adding a regularization loss that directs the neural network to output the ground truth labels for OOD samples (Caruana, ). In the case of an RL policy, however, we do not know the ground truth (optimal actions) for anchoring the policy output. Moreover, using the actions we took in prior contexts as the ground truth is not possible, since the policy may have not converged at those times. Anchoring to those actions may cause the policy to relearn suboptimal actions from an earlier period in training.

To avoid these problems, LCPO solves a constrained optimization problem that forces the policy to not change for OOD samples. Formally, we consider the following optimization problem:

$$ $\displaystyle\min_{\theta}~{}~{}\mathcal{L}_{tot}(\theta;B_{r})\triangleq\mathcal{L}_{PG}(\theta;B_{r})$ $\displaystyle+\mathcal{L}_{e}(\theta;B_{r})$ (1) $\displaystyle s.t.~{}~{}D_{KL}(\theta_{0},\theta;W(B_{a},B_{r}$ $\displaystyle))\leq c_{anchor}$ $$

We use the standard definition of policy gradient loss, that optimizes a policy to maximize returns (Schulman et al., ; Mnih et al., ; Sutton & Barto, ):

$$ $\displaystyle\mathcal{L}_{PG}(\theta;B_{r})=\mathbb{E}_{r_{t}\sim B_{r}}\left[\sum_{t=0}^{H}-\gamma^{t}r_{t}\right]$ (2) $$

We use automatic entropy regularization (Haarnoja et al., ), to react to and explore in response to novel contexts. The learnable parameter $\theta_{e}$ is adapted such that the entropy coefficient is $e^{\theta_{e}}$, and the entropy remains close to a target entropy $\bar{\mathcal{H}}$. This worked well in our experiments but LCPO could use any exploration method designed for non-stationary context-driven environments.

$$ $\displaystyle\mathcal{L}_{e}(\theta;B_{r})=e^{\theta_{e}}.\left(\mathbb{E}_{s_{t},z_{t}\sim B_{r},a_{t}\sim\pi}\left[\log\pi(a_{t}|s_{t},z_{t})\right]\right)$ (3) $$

We use KL-divergence as a measure of policy change, and for simplicity we use $D_{KL}(\theta_{0},\theta;W(B_{a},B_{r}))$ as a shorthand for $\mathbb{E}_{s,z\sim W(B_{a},B_{r})}[D_{KL}(\pi_{\theta_{0}}(s,z)||\pi_{\theta}(s,z))]$. Here, $\theta_{0}$ denotes the current policy parameters, and we are solving the optimization over $\theta$ to determine the new policy parameters.

Figure: Algorithm 1 LCPO Training

#### Buffer Management.

To avoid storing all interactions in $B_{a}$ we use reservoir sampling (Vitter, ), where we randomly replace old interactions with new ones with probability $\frac{n_{b}}{n_{s}}$, where $n_{b}$ is the buffer size and $n_{s}$ is the total interactions thus far. Reservoir sampling guarantees that the interactions in the buffer are a uniformly random subset of the full set of interactions.

#### Difference detector.

To realize $W(B_{a},B_{r})$, we treat it as an out-of-distribution detection (OOD) task. A variety of methods can be used for this in practice (§[6](#S6)). For example, we can compute the Mahalanobis distance (Mahalanobis, ), i.e. the normalized distance of each experience’s context, with respect to the average context in $B_{r}$, and deem any distance above a certain threshold to be OOD. To avoid a high computational overhead when sampling from $W(B_{a},B_{r})$, we sample a mini-batch from $B_{a}$, and keep the state-context pairs that are OOD with respect to $B_{r}$. We repeat this step up to five times. If not enough different samples exist, we do not apply the constraint for that update.

#### Solving the constrained optimization.

To solve this constrained optimization, we approximate the optimization goal and constraint, and calculate a search direction accordingly (pseudocode in [Algorithm 1](#alg1)). Our problem is structurally similar to TRPO (Schulman et al., ), though the constraint is quite different. Similar to TRPO, we model the optimization goal with a first-order approximation, i.e. $\mathcal{L}_{tot}(\theta;\cdot)=\mathcal{L}_{0}+(\theta-\theta_{0})^{T}\nabla_{\theta}\mathcal{L}_{tot}(\theta;\cdot)|_{\theta_{0}}$, and the constraint with a second order approximation $D_{KL}(\theta,\theta_{0};\cdot)=(\theta-\theta_{0})^{T}\nabla^{2}_{\theta}D_{KL}(\theta,\theta_{0};\cdot)|_{\theta_{0}}(\theta-\theta_{0})$. The optimization problem can therefore be written as:

$$ $\displaystyle\min_{\theta}~{}~{}(\theta-\theta_{0})^{T}$ $\displaystyle v$ (4) $\displaystyle s.t.~{}~{}(\theta-\theta_{0})^{T}A(\theta-$ $\displaystyle\theta_{0})\leq c_{anchor}$ $$

where $A_{ij}=\frac{\partial}{\partial\theta_{i}}{\frac{\partial}{\partial\theta_{j}}}D_{KL}(\theta,\theta_{0};W(B_{a},B_{r}))$, and $v=\nabla_{\theta}\mathcal{L}_{tot}(\theta;\cdot)|_{\theta_{0}}$. This optimization problem can be solved using the conjugate gradient method followed by a line search as described in TRPO (Schulman et al., ).

#### Bounding policy change.

The above formulation does not bound policy change on the current context, which could destabilize learning. We could add a second constraint, i.e. TRPO’s constraint, $D_{KL}(\theta_{0},\theta;B_{r})\leq c_{recent}$, but having two second order constraints is computationally expensive.
Instead, we guarantee the TRPO constraint in the line search phase (lines 9–10 in [Algorithm 1](#alg1)), where we repeatedly decrease the gradient update size until both constraints are met.

## 6 Evaluation

We evaluate LCPO across six environments: four from Gymnasium Mujoco (Towers et al., ), one from Gymnasium Classic Control (Towers et al., ), and a straggler mitigation task from computer systems (Hamadanian et al., ). These environments are subject to synthetic or real context processes that affect their dynamics. Our experiments aim to answer the following questions: (1) How does LCPO compare to state-of-the-art on-policy, off-policy, model-based baselines and pre-trained oracle policies (§[6.1](#S6.SS1))?
(2) How does the maximum buffer size $n_{b}=|B_{a}|$ affect LCPO (§[6.2](#S6.SS2))?
(3) How does the accuracy of the OOD sampler $W(\cdot,\cdot)$ affect LCPO (§[6.3](#S6.SS3))?

#### Baselines.

We consider the following approaches.
(1) Model-Based Changepoint Detection (MBCD) (Alegre et al., ): This approach is the state-of-the-art technique for handling piece-wise stationary environments without task labels. It trains models to predict environment state transitions, and launches new policies when the current model is inaccurate in predicting the state trajectory. The underlying learning approach is Soft Actor Critic (SAC) (Haarnoja et al., ).
(2) Model-Based Policy Optimization (MBPO) (Janner et al., ): A model-based approach that trains an ensemble of experts to learn the environment model, and generates samples for an SAC (Haarnoja et al., ) algorithm. If the model is accurate, it can fully replay prior contexts, thereby avoiding catastrophic forgetting.
(3) Online Elastic Weight Consolidation (EWC) (Chaudhry et al., ; Schwarz et al., ): EWC regularizes online learning with a Bayesian approach assuming task indices, and online EWC generalizes it to task boundaries. To adapt online EWC to our problem, we update the importance vector and average weights using a weighted moving average in every time step. The underlying learning approach is SAC (Haarnoja et al., ).
(4) On-policy RL: On-policy RL is susceptible to CF, but more stable in online learning compared to off-policy RL algorithms. We compare with A2C (Mnih et al., ) and TRPO (single-path) (Schulman et al., ), with Generalized Advantage Estimation (GAE) (Schulman et al., ) applied. Note that TRPO vine is not possible in online learning, as it requires rolling back the environment world.
(5) Off-policy RL: Off-policy RL is potentially capable of overcoming CF due to replay buffers, at the cost of unstable training. We consider Double Deep Q Network (DDQN) (Hasselt et al., ) and SAC (with automatic entropy regularization, similar to LCPO) (Haarnoja et al., ).
(6) Oracle RL: To establish an upper-bound on the best performance that an online learner can achieve, we train *oracle policies*. We allow oracle policies to have unlimited access to the contexts and environment dynamics, i.e. they are able to replay any particular environment and context as many times as necessary. Since oracle policies can interact with multiple contexts in parallel during training, they do not suffer from CF. In contrast, all other baselines (and LCPO) are only allowed to experience contexts sequentially as they occur over time and must adapt the policy on the go. We report results for the best of four oracle policies with the following model-free algorithms: A2C (Mnih et al., ), TRPO (single-path) (Schulman et al., ), DDQN (Hasselt et al., ) and SAC (Haarnoja et al., ).

#### Experiment Setup.

We use 5 random seeds for gymnasium and 10 random seeds for the straggler mitigation experiments, and use the same hyperparameters for LCPO in all environments and contexts. Hyperparameters and neural network structures are noted in Appendices §[A.4](#A1.SS4) and §[B.2](#A2.SS2). These experiments were conducted on a machine with 2 AMD EPYC 7763 CPUs (256 logical cores) and 512 GiB of RAM. With 32 concurrent processes, the experiments finished in $\sim$364 hours.

#### Environment and Contexts.

We consider six environments: Modified versions of Pendulum-v1 from the classic control environments, InvertedPendulum-v4, InvertedDoublePendulum-v4, Hopper-v4 and Reacher-v4 from the Mujoco environments (Towers et al., ), and a straggler mitigation environment (Hamadanian et al., ). In the gym environments, the context is an exogenous “wind” process that creates external force on joints and affects movements. We append the external wind vectors from the last 3 time-steps to the observation, since the agent cannot observe the external context that is going to be applied in the next step, and a history of prior steps helps with the prediction. We create 4 synthetic context sequences with the Ornstein–Uhlenbeck process (Uhlenbeck & Ornstein, ), piece-wise Gaussian models, or hand-crafted signals with additive noise. These context processes cover smooth, sudden, stochastic, and predictable transitions at short horizons. All context traces are visualized in [Figure 7](#A1.F7) in the appendix. For the straggler mitigation environments, we use workloads provided by the authors in (Hamadanian et al., ), that are from a production web framework cluster at AnonCo, collected from a single day in February 2018. These workloads are visualized in [Figure 9](#A2.F9) in the appendix.

#### OOD detection.

We set the buffer size $n_{b}$ to 1% of all samples, which is $n_{b}\leq 200K$. To sample OOD state-context pairs $W(B_{a},B_{r})$, we use distance metrics and thresholds. For gym environments where the context is a wind process, we use L2 distance, i.e. if $\overline{w_{r}}=\mathbb{E}_{w\sim B_{r}}[w]$ is the average wind vector observed in the recent batch $B_{r}$, we sample a minibatch of states in $B_{a}$ where $W(B_{a},B_{r})=\{w_{i}|\forall w_{i}\in B_{a}:\lVert w_{i}-\overline{w_{r}}\rVert_{2}>\sigma\}$. There exist domain-specific models for workload distributions in the straggler mitigation environment, but we used Mahalanobis distance as it is a well-accepted and general approach for outlier detection in prior work (Lee et al., ; Podolskiy et al., ). Concretely, we fit a Gaussian multivariate model to the recent batch $B_{r}$, and report a minibatch of states in $B_{a}$ with a Mahalanobis distance further than $\sigma$ from this distribution.

### 6.1 Results

#### Results.

To evaluate across gymnasium environments, we score online learning baselines with Normalized Return, i.e., a scaled score function where 0 and 1 are the minimum and maximum return across all agents. [Figure 4](#S6.F4) provides a summary of all gymnasium experiments (for full details refer to [Table 2](#A1.T2) in the Appendix). LCPO maintains a lead over baselines, is close to the best-performing oracle policy, all while learning online and sequentially. MBCD struggles to tease out meaningful task boundaries. In some experiments, it launches anywhere between 3 to 7 policies just by changing the random seed. This observation is in line with MBCD’s sensitivity in §[4](#S4).

MBPO performed poorly, even though we verified that its learned model of the environment is highly accurate.(^1^11We tried a variant of MBPO that uses the actual environment rather than a learned model to produce counterfactual interactions. It performed similarly to MBPO, with further details in §[A.2](#A1.SS2). Thus MBPO would not benefit from more complicated online learning algorithms such as online meta-learning (Finn et al., ).)
The reason is the way that MBPO samples experiences for training. At every iteration, MBPO samples a batch of actual interactions from its experience buffer and generates hypothetical interactions from them. These hypothetical interactions amass in a second buffer, which is used to train an SAC agent. During the course of training, the second buffer accumulates more interactions generated from samples from the start of the experiment compared to recent samples. This is not an issue when the problem is stationary, but in non-stationary RL subject to an context process, this leads to over-emphasis of the context processes encountered earlier in the experiment. As such, MBPO fails to even surpass SAC. Prior work has observed the sensitivity of off-policy approaches to such sampling strategies (Isele & Cosgun, ; Hamadanian et al., ). In our experiments, we also observe that the on-policy A2C usually outperforms the off-policy SAC and Deep Q Network (DQN).

**Table 1: Tail latency (negative reward) and 95th percentile confidence ranges for different algorithms and contexts in the straggler mitigation environment.**
|  | Online Learning |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | LCPO | LCPO | LCPO | MBCD | MBPO | Online EWC | A2C | TRPO | DDQN | SAC | Best |
|  | Agg | Med | Cons | Oracle |  |  |  |  |  |  |  |
| Workload 1 | 1070 | 1076 | 1048 | 1701 | 2531 | 2711 | 1716 | 3154 | 1701 | 1854 | 984 |
| $\pm$10 | $\pm$16 | $\pm$7 | $\pm$112 | $\pm$197 | $\pm$232 | $\pm$710 | $\pm$464 | $\pm$47 | $\pm$245 | (TRPO) |  |
| Workload 2 | 589 | 617 | 586 | 678 | 891 | 724 | 604 | 864 | 633 | 644 | 509 |
| $\pm$43 | $\pm$62 | $\pm$27 | $\pm$38 | $\pm$54 | $\pm$22 | $\pm$109 | $\pm$105 | $\pm$7 | $\pm$27 | (A2C) |  |

Figure: Figure 4: Normalized episodic returns, where 0 and 1 denote the lowest and highest returns among agents. LCPO outperforms all online agents, and remains the closest to oracles.
Refer to caption: /html/2302.02182/assets/x6.png

Figure: Figure 5: Pendulum-v1 episodic returns and 95% confidence bounds of Online EWC with 12 hyperparameter trials. Hyperparameters are labeled as $\{\alpha,\beta^{-1}\}$, where $\alpha$ is the regularization strength and $\beta$ is the averaging weight. The optimal online EWC hyperparameter is sensitive to the context, but LCPO is better even if online EWC is tuned per context.
Refer to caption: /html/2302.02182/assets/x7.png

Online EWC applies a regularization loss $\mathcal{L}_{ewc}=\alpha\lvert\lvert\theta_{t}-\theta_{t-1}^{*}\rvert\rvert^{2}_{F_{t-1}^{*}}$ to the training loss, using a running average $F_{t}^{*}$ of the diagonal of the Fisher Information Matrix (FIM), and a running average of the parameters $\theta_{t}^{*}$. The running average $F_{t}^{*}$ is updated with a weighted average $F_{t}^{*}=(1-\beta)F_{t-1}^{*}+\beta F_{t}$, where $F_{t}$ is the diagonal of the FIM respective to the recent parameters and samples.(^2^22We deviate from the original notations $\{\lambda,\gamma\}$ (Rusu et al., ), since they could be confused with the MDP discount factor $\gamma$ and GAE discount factor $\lambda$ (Schulman et al., ).) Similarly, the running average $\theta_{t}^{*}$ uses the same parameter $\beta$.

Online EWC may constrain the policy output to remain constant on samples in the last $\approx\beta^{-1}$ epochs, but it has to strike a balance between how fast the importance metrics are updated with the newest FIM (larger $\beta$) and how long the policy has to remember its training (smaller $\beta$). This balance will ultimately depend on the context trace and how frequently they evolve. We tuned $\alpha$ and $\beta$ on Pendulum-v1 for all contexts, trying $\alpha\in\{0.05,0.1,0.5,1.0,10,100,1000\}$ and $\beta^{-1}\in\{1M,3M,10M\}$. The returns are visualized in [Figure 5](#S6.F5) (with full details in [Table 4](#A1.T4) in the Appendix). There is no universal $\beta$ that works well across all contexts and online EWC would not perform better than LCPO even if tuned to each context trace individually. We ultimately chose $\beta^{-1}=3M$ samples to strike a balance across all contexts, but it struggled to even surpass SAC on other environments.

For the straggler mitigation environment, [Table 1](#S6.T1) presents the latency metric (negative reward) over two workloads. Recall that this environment uses real-world traces from a production cloud network. The overall trends are similar to the gymnasium experiments, with LCPO outperforming all other baselines. This table includes three variants of LCPO, that will be discussed further in §[6.3](#S6.SS3).

### 6.2 Sensitivity to Buffer Size

LCPO uses Reservoir sampling (Vitter, ) to maintain a limited number of samples $n_{b}$. We evaluate how sensitive LCPO is to the buffer size in [Figure 6](#S6.F6) (full results in [Table 3](#A1.T3) in the Appendix). The full experiment has 8–20M samples. LCPO maintains its high performance, even with as little as $n_{b}=500$ samples, but drops below this point. This is not surprising, as the context traces do not change drastically at short intervals, and even 500 randomly sampled points from the trace should be enough to have a representation over all the trace. However, it is likely that with more complicated and high-dimensional contexts, a higher buffer size would be necessary.

Figure: Figure 6: Normalized returns of LCPO in gymnasium environments with various buffer sizes. LCPO loses performance with $n_{b}<500$.
Refer to caption: /html/2302.02182/assets/x8.png

### 6.3 Sensitivity to OOD metric

LCPO applies a constraint to OOD state-context pairs, as dictated by the OOD sampler $W(B_{a},B_{r})$. We vary the Mahalanobis distance threshold $\sigma$, the threshold that the OOD method uses in sampling, and monitor the normalized return for the straggler mitigation environment. LCPO Agg, LCPO Med and LCPO Cons use $\sigma=5$, $6$ and $7$, and a higher value for $\sigma$ yields more conservative OOD samples (i.e., fewer samples are detected as OOD). The difference between these three is significant: The model in LCPO Agg allows for $26.7\times$ more samples to be considered OOD compared to LCPO Cons. [Table 1](#S6.T1) provides the normalized return for LCPO with varying thresholds, along with baselines. Notably, all variations of LCPO achieve similar results.

## 7 Discussion and Limitations

#### Network Capacity.

In general, online learning methods with bounded parameter counts will reach the function approximator’s (neural network’s) maximum representational capacity. LCPO is not immune from this, as we do not add parameters with more context traces. However, neither are oracle agents. To isolate the effect of this capacity and CF, we compare against oracle agents, rather than single agents trained on individual tasks or context traces (He et al., ). This ensures a fair evaluation that does not penalize online learning for reaching the capacity ceiling.

#### Exploration.

LCPO focuses on mitigating catastrophic forgetting in non-stationary RL. An orthogonal challenge in this setting is efficient exploration, i.e. to explore when the context distribution has changed but only once per new context. Our experiments used automatic entropy tuning for exploration (Haarnoja et al., ). While empirically effective, this approach was not designed for non-stationary problems, and LCPO may benefit from a better exploration methodology such as curiosity-driven exploration (Pathak et al., ).

#### Efficient Buffer Management.

We used Reservoir Sampling (Vitter, ), which maintains a uniformly random buffer of all observed state-context samples so far. Future work could explore more efficient strategies that selectively store or drop samples based on their context, e.g., to maximize sample diversity.

## 8 Conclusion

We proposed and evaluated LCPO, a simple approach for learning policies in non-stationary context-driven environments. Using LCPO requires two conditions: (1) the non-stationarity must be induced by an exogenous observed context process; and (2) a similarity metric is required that can inform us if two contexts come from noticeably different distributions (OOD detection). This is less restrictive than prior approaches that require either explicit or inferred task labels. Our experiments showed that LCPO outperforms state-of-the-art baselines on several environments with real and synthetic context processes.

## 9 Broader Impact

Our work advances the field of reinforcement learning. Online learning in non-stationary environments is an important hurdle for deploying RL agents in the real world. There are many potential societal consequences, none which we feel must be specifically highlighted here.

## Appendix A Locomotion tasks in Gymnasium

Figure: Figure 7: External wind force, per axis and context trace.
Refer to caption: /html/2302.02182/assets/x9.png

### A.1 Full results

[Table 2](#A1.T2) presents the episodic return for all agents, environments and context traces.
[Table 3](#A1.T3) presents the episodic return for multiple LCPO agents with different buffer sizes $n_{b}$.
[Table 4](#A1.T4) presents the episodic return for online EWC agents with various hyperparameter values in the Pendulum-v1 environment.

**Table 2: Average and 95th percentile confidence ranges for episodic return for different algorithms and conditions in environments with external wind processes.**
|  |  | Online Learning |  | Oracle Policies |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | LCPO | Online EWC | MBCD | MBPO | A2C | TRPO | DDQN | SAC |  | A2C | TRPO | DDQN | SAC |
| Pendulum | Context Trace 1 | -190 | -197 | -207 | -325 | -203 | -224 | -216 | -210 |  | -208 | -197 | -188 | -187 |
| $\pm$1.29 | $\pm$7.63 | $\pm$23.4 | $\pm$47.2 | $\pm$10.2 | $\pm$11.2 | $\pm$16.9 | $\pm$22.7 |  |  |  |  |  |  |  |
| Context Trace 2 | -357 | -523 | -577 | -843 | -373 | -512 | -755 | -581 |  | -407 | -399 | -376 | -380 |  |
| $\pm$6.38 | $\pm$58.8 | $\pm$71.3 | $\pm$24.2 | $\pm$24.2 | $\pm$45.1 | $\pm$369 | $\pm$74.2 |  |  |  |  |  |  |  |
| Context Trace 3 | -237 | -314 | -349 | -603 | -330 | -511 | -577 | -384 |  | -224 | -320 | -228 | -203 |  |
| $\pm$14.6 | $\pm$45.8 | $\pm$28.9 | $\pm$213 | $\pm$111 | $\pm$205 | $\pm$194 | $\pm$45.2 |  |  |  |  |  |  |  |
| Context Trace 4 | -383 | -415 | -430 | -553 | -395 | -741 | -414 | -416 |  | -383 | -381 | -384 | -357 |  |
| $\pm$7.73 | $\pm$22.0 | $\pm$15.4 | $\pm$90.9 | $\pm$14.0 | $\pm$137 | $\pm$27.2 | $\pm$22.8 |  |  |  |  |  |  |  |
| Inverse Pendulum | Context Trace 1 | 18.2 | 3.45 | 6.17 | 2.88 | 6.46 | 2.07 | 14.0 | 4.50 |  | 24.4 | 25.7 | 24.1 | 24.8 |
| $\pm$12.3 | $\pm$1.47 | $\pm$6.10 | $\pm$1.78 | $\pm$8.35 | $\pm$0.34 | $\pm$5.11 | $\pm$3.66 |  |  |  |  |  |  |  |
| Context Trace 2 | 4.63 | 1.50 | 2.10 | 1.52 | 3.54 | 2.39 | 1.51 | 1.53 |  | 7.86 | 8.51 | 7.18 | 7.75 |  |
| $\pm$2.20 | $\pm$0.08 | $\pm$1.06 | $\pm$0.01 | $\pm$2.38 | $\pm$1.95 | $\pm$0.07 | $\pm$0.01 |  |  |  |  |  |  |  |
| Context Trace 3 | 15.1 | 6.36 | 14.2 | 14.4 | 14.5 | 7.81 | 15.0 | 14.7 |  | 15.5 | 16.5 | 15.1 | 14.3 |  |
| $\pm$0.87 | $\pm$3.12 | $\pm$0.77 | $\pm$0.23 | $\pm$2.11 | $\pm$8.63 | $\pm$0.56 | $\pm$0.16 |  |  |  |  |  |  |  |
| Context Trace 4 | 92.2 | 4.78 | 46.5 | 37.6 | 93.7 | 79.5 | 93.3 | 53.8 |  | 102 | 112 | 53.7 | 40.1 |  |
| $\pm$0.48 | $\pm$1.15 | $\pm$4.85 | $\pm$0.98 | $\pm$0.71 | $\pm$32.4 | $\pm$1.80 | $\pm$6.09 |  |  |  |  |  |  |  |
| Inverse Double Pendulum | Context Trace 1 | 110 | 38.2 | 77.3 | 68.5 | 110 | 79.7 | 82.2 | 83.8 |  | 142 | 165 | 88.0 | 95.5 |
| $\pm$5.84 | $\pm$5.46 | $\pm$13.5 | $\pm$19.3 | $\pm$15.6 | $\pm$60.1 | $\pm$3.17 | $\pm$2.94 |  |  |  |  |  |  |  |
| Context Trace 2 | 59.8 | 30.1 | 44.2 | 37.2 | 54.2 | 30.5 | 40.8 | 36.7 |  | 51.4 | 64.0 | 46.8 | 49.5 |  |
| $\pm$5.03 | $\pm$1.35 | $\pm$3.27 | $\pm$9.92 | $\pm$4.25 | $\pm$14.8 | $\pm$5.48 | $\pm$7.03 |  |  |  |  |  |  |  |
| Context Trace 3 | 66.3 | 37.4 | 45.8 | 42.7 | 59.7 | 54.6 | 61.4 | 46.8 |  | 63.5 | 74.0 | 63.6 | 47.4 |  |
| $\pm$2.47 | $\pm$2.88 | $\pm$2.38 | $\pm$1.55 | $\pm$11.4 | $\pm$5.87 | $\pm$0.52 | $\pm$0.39 |  |  |  |  |  |  |  |
| Context Trace 4 | 88.6 | 39.1 | 75.0 | 59.5 | 88.0 | 76.4 | 94.1 | 80.0 |  | 91.9 | 96.9 | 90.7 | 78.0 |  |
| $\pm$1.55 | $\pm$12.6 | $\pm$4.65 | $\pm$2.86 | $\pm$1.87 | $\pm$6.16 | $\pm$0.52 | $\pm$0.68 |  |  |  |  |  |  |  |
| Hopper | Context Trace 1 | 245 | 31.4 | 153 | 130 | 199 | 157 | 131 | 172 |  | 240 | 215 | 155 | 184 |
| $\pm$24.0 | $\pm$21.6 | $\pm$33.2 | $\pm$12.8 | $\pm$10.3 | $\pm$31.0 | $\pm$27.6 | $\pm$8.75 |  |  |  |  |  |  |  |
| Context Trace 2 | 142 | 33.1 | 76.0 | 66.0 | 92.9 | 74.0 | 65.7 | 81.8 |  | 128 | 131 | 86.6 | 96.7 |  |
| $\pm$14.2 | $\pm$19.0 | $\pm$15.3 | $\pm$18.2 | $\pm$10.3 | $\pm$12.7 | $\pm$17.5 | $\pm$14.0 |  |  |  |  |  |  |  |
| Context Trace 3 | 508 | 32.0 | 240 | 264 | 469 | 424 | 224 | 256 |  | 520 | 492 | 327 | 308 |  |
| $\pm$10.4 | $\pm$39.8 | $\pm$81.7 | $\pm$49.9 | $\pm$73.5 | $\pm$67.5 | $\pm$9.42 | $\pm$125 |  |  |  |  |  |  |  |
| Context Trace 4 | 298 | 25.3 | 127 | 205 | 277 | 201 | 157 | 223 |  | 319 | 297 | 189 | 189 |  |
| $\pm$11.0 | $\pm$18.2 | $\pm$27.3 | $\pm$37.5 | $\pm$2.47 | $\pm$74.7 | $\pm$17.9 | $\pm$66.9 |  |  |  |  |  |  |  |
| Reacher | Context Trace 1 | -26.8 | -15.8 | -169 | -1862 | -138 | -2394 | -7.86 | -224 |  | -8.00 | -7.59 | -7.59 | -14.3 |
| $\pm$0.80 | $\pm$1.63 | $\pm$122 | $\pm$2255 | $\pm$136 | $\pm$1266 | $\pm$0.11 | $\pm$412 |  |  |  |  |  |  |  |
| Context Trace 2 | -20.0 | -16.2 | -339 | -4308 | -108 | -5267 | -7.78 | -733 |  | -8.33 | -3866 | -7.41 | -15.3 |  |
| $\pm$0.32 | $\pm$0.79 | $\pm$493 | $\pm$3764 | $\pm$130 | $\pm$1190 | $\pm$0.18 | $\pm$1591 |  |  |  |  |  |  |  |
| Context Trace 3 | -27.5 | -15.4 | -596 | -2058 | -208 | -43.9 | -8.60 | -137 |  | -9.63 | -8.42 | -8.41 | -15.7 |  |
| $\pm$2.40 | $\pm$0.19 | $\pm$857 | $\pm$4173 | $\pm$400 | $\pm$51.6 | $\pm$0.20 | $\pm$212 |  |  |  |  |  |  |  |
| Context Trace 4 | -29.3 | -16.3 | -121 | -2354 | -149 | -1482 | -11.9 | -174 |  | -9.63 | -8.57 | -7.97 | -15.7 |  |
| $\pm$1.04 | $\pm$1.20 | $\pm$161 | $\pm$2988 | $\pm$227 | $\pm$639 | $\pm$7.37 | $\pm$353 |  |  |  |  |  |  |  |

**Table 3: Average and 95th percentile confidence ranges for episodic return in LCPO with multiple buffer sizes and other agents for different conditions in environments with external wind processes.**
|  |  | Online Learning |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | LCPO 20M | LCPO 200K | LCPO 500 | LCPO 25 | Online EWC | MBCD | MBPO | A2C |  | Best Oracle |
| Pendulum | Context Trace 1 | -190 | -190 | -191 | -193 | -197 | -207 | -325 | -203 |  | -187 |
| $\pm$2.39 | $\pm$1.29 | $\pm$4.95 | $\pm$2.67 | $\pm$7.63 | $\pm$23.4 | $\pm$47.2 | $\pm$10.2 |  | (SAC) |  |  |
| Context Trace 2 | -361 | -357 | -353 | -360 | -523 | -577 | -843 | -373 |  | -376 |  |
| $\pm$6.16 | $\pm$6.38 | $\pm$5.07 | $\pm$15.5 | $\pm$58.8 | $\pm$71.3 | $\pm$24.2 | $\pm$24.2 |  | (DDQN) |  |  |
| Context Trace 3 | -238 | -237 | -232 | -256 | -314 | -349 | -603 | -330 |  | -203 |  |
| $\pm$17.7 | $\pm$14.6 | $\pm$22.3 | $\pm$23.4 | $\pm$45.8 | $\pm$28.9 | $\pm$213 | $\pm$111 |  | (SAC) |  |  |
| Context Trace 4 | -378 | -383 | -380 | -404 | -415 | -430 | -553 | -395 |  | -357 |  |
| $\pm$8.09 | $\pm$7.73 | $\pm$16.3 | $\pm$7.74 | $\pm$22.0 | $\pm$15.4 | $\pm$90.9 | $\pm$14.0 |  | (SAC) |  |  |
| Inverse Pendulum | Context Trace 1 | 21.5 | 18.2 | 21.7 | 14.1 | 3.45 | 6.17 | 2.88 | 6.46 |  | 25.7 |
| $\pm$1.69 | $\pm$12.3 | $\pm$2.18 | $\pm$11.5 | $\pm$1.47 | $\pm$6.10 | $\pm$1.78 | $\pm$8.35 |  | (TRPO) |  |  |
| Context Trace 2 | 4.19 | 4.63 | 4.10 | 4.35 | 1.50 | 2.10 | 1.52 | 3.54 |  | 8.51 |  |
| $\pm$2.95 | $\pm$2.20 | $\pm$2.66 | $\pm$3.26 | $\pm$0.08 | $\pm$1.06 | $\pm$0.01 | $\pm$2.38 |  | (TRPO) |  |  |
| Context Trace 3 | 15.4 | 15.1 | 15.3 | 15.5 | 6.36 | 14.2 | 14.4 | 14.5 |  | 16.5 |  |
| $\pm$0.83 | $\pm$0.87 | $\pm$0.78 | $\pm$0.22 | $\pm$3.12 | $\pm$0.77 | $\pm$0.23 | $\pm$2.11 |  | (TRPO) |  |  |
| Context Trace 4 | 92.2 | 92.2 | 92.1 | 92.3 | 4.78 | 46.5 | 37.6 | 93.7 |  | 112 |  |
| $\pm$0.76 | $\pm$0.48 | $\pm$0.61 | $\pm$0.55 | $\pm$1.15 | $\pm$4.85 | $\pm$0.98 | $\pm$0.71 |  | (TRPO) |  |  |
| Inverse Double Pendulum | Context Trace 1 | 113 | 110 | 119 | 116 | 38.2 | 77.3 | 68.5 | 110 |  | 165 |
| $\pm$4.16 | $\pm$5.84 | $\pm$32.7 | $\pm$12.4 | $\pm$5.46 | $\pm$13.5 | $\pm$19.3 | $\pm$15.6 |  | (TRPO) |  |  |
| Context Trace 2 | 55.3 | 59.8 | 56.5 | 54.5 | 30.1 | 44.2 | 37.2 | 54.2 |  | 64.0 |  |
| $\pm$3.82 | $\pm$5.03 | $\pm$7.47 | $\pm$4.35 | $\pm$1.35 | $\pm$3.27 | $\pm$9.92 | $\pm$4.25 |  | (TRPO) |  |  |
| Context Trace 3 | 64.5 | 66.3 | 65.2 | 65.2 | 37.4 | 45.8 | 42.7 | 59.7 |  | 74.0 |  |
| $\pm$2.18 | $\pm$2.47 | $\pm$2.01 | $\pm$2.77 | $\pm$2.88 | $\pm$2.38 | $\pm$1.55 | $\pm$11.4 |  | (TRPO) |  |  |
| Context Trace 4 | 89.2 | 88.6 | 89.5 | 89.5 | 39.1 | 75.0 | 59.5 | 88.0 |  | 96.9 |  |
| $\pm$0.36 | $\pm$1.55 | $\pm$0.57 | $\pm$0.19 | $\pm$12.6 | $\pm$4.65 | $\pm$2.86 | $\pm$1.87 |  | (TRPO) |  |  |
| Hopper | Context Trace 1 | 242 | 245 | 244 | 231 | 31.4 | 153 | 130 | 199 |  | 240 |
| $\pm$19.4 | $\pm$24.0 | $\pm$12.0 | $\pm$7.57 | $\pm$21.6 | $\pm$33.2 | $\pm$12.8 | $\pm$10.3 |  | (A2C) |  |  |
| Context Trace 2 | 144 | 142 | 148 | 132 | 33.1 | 76.0 | 66.0 | 92.9 |  | 131 |  |
| $\pm$10.9 | $\pm$14.2 | $\pm$7.73 | $\pm$13.8 | $\pm$19.0 | $\pm$15.3 | $\pm$18.2 | $\pm$10.3 |  | (TRPO) |  |  |
| Context Trace 3 | 505 | 508 | 509 | 506 | 32.0 | 240 | 264 | 469 |  | 520 |  |
| $\pm$5.12 | $\pm$10.4 | $\pm$8.82 | $\pm$5.77 | $\pm$39.8 | $\pm$81.7 | $\pm$49.9 | $\pm$73.5 |  | (A2C) |  |  |
| Context Trace 4 | 295 | 298 | 290 | 279 | 25.3 | 127 | 205 | 277 |  | 319 |  |
| $\pm$13.4 | $\pm$11.0 | $\pm$6.15 | $\pm$11.0 | $\pm$18.2 | $\pm$27.3 | $\pm$37.5 | $\pm$2.47 |  | (A2C) |  |  |
| Reacher | Context Trace 1 | -26.6 | -26.8 | -26.8 | -26.3 | -15.8 | -169 | -1862 | -138 |  | -7.59 |
| $\pm$0.38 | $\pm$0.80 | $\pm$0.54 | $\pm$0.25 | $\pm$1.63 | $\pm$122 | $\pm$2255 | $\pm$136 |  | (DDQN) |  |  |
| Context Trace 2 | -20.0 | -20.0 | -20.0 | -19.9 | -16.2 | -339 | -4308 | -108 |  | -7.41 |  |
| $\pm$0.37 | $\pm$0.32 | $\pm$0.37 | $\pm$0.38 | $\pm$0.79 | $\pm$493 | $\pm$3764 | $\pm$130 |  | (DDQN) |  |  |
| Context Trace 3 | -28.4 | -27.5 | -28.2 | -28.3 | -15.4 | -596 | -2058 | -208 |  | -8.41 |  |
| $\pm$1.05 | $\pm$2.40 | $\pm$1.46 | $\pm$0.50 | $\pm$0.19 | $\pm$857 | $\pm$4173 | $\pm$400 |  | (DDQN) |  |  |
| Context Trace 4 | -29.5 | -29.3 | -29.5 | -29.4 | -16.3 | -121 | -2354 | -149 |  | -7.97 |  |
| $\pm$0.87 | $\pm$1.04 | $\pm$2.56 | $\pm$2.76 | $\pm$1.20 | $\pm$161 | $\pm$2988 | $\pm$227 |  | (DDQN) |  |  |

**Table 4: Average and 95th percentile confidence ranges for episodic returns for online EWC variants in the Pendulum-v1 environment with external wind processes.**
|  | Online EWC |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $\alpha=0.05$ |  | $\alpha=0.1$ |  | $\alpha=0.5$ |  | $\alpha=1.0$ |  |  |  |  |  |  |  |  |
| $\beta^{-1}\times$ rollout length: | $1M$ | $3M$ | $10M$ |  | $1M$ | $3M$ | $10M$ |  | $1M$ | $3M$ | $10M$ |  | $1M$ | $3M$ | $10M$ |
| Context Trace 1 | -209 | -207 | -206 |  | -199 | -203 | -204 |  | -201 | -204 | -207 |  | -208 | -212 | -209 |
| $\pm$13.0 | $\pm$9.59 | $\pm$16.6 |  | $\pm$5.47 | $\pm$7.28 | $\pm$8.06 |  | $\pm$7.57 | $\pm$11.5 | $\pm$3.67 |  | $\pm$9.69 | $\pm$3.87 | $\pm$4.49 |  |
| Context Trace 2 | -489 | -545 | -544 |  | -477 | -527 | -525 |  | -540 | -531 | -541 |  | -545 | -510 | -539 |
| $\pm$68.6 | $\pm$74.1 | $\pm$95.3 |  | $\pm$26.9 | $\pm$67.9 | $\pm$47.6 |  | $\pm$84.6 | $\pm$65.9 | $\pm$48.0 |  | $\pm$49.0 | $\pm$48.5 | $\pm$71.2 |  |
| Context Trace 3 | -380 | -356 | -372 |  | -422 | -349 | -393 |  | -335 | -340 | -330 |  | -320 | -320 | -316 |
| $\pm$82.4 | $\pm$108 | $\pm$70.0 |  | $\pm$102 | $\pm$72.4 | $\pm$111 |  | $\pm$114 | $\pm$66.1 | $\pm$68.5 |  | $\pm$29.1 | $\pm$46.1 | $\pm$45.7 |  |
| Context Trace 4 | -396 | -400 | -407 |  | -413 | -408 | -397 |  | -404 | -406 | -404 |  | -412 | -409 | -416 |
| $\pm$16.1 | $\pm$36.7 | $\pm$15.6 |  | $\pm$21.1 | $\pm$23.0 | $\pm$31.9 |  | $\pm$11.7 | $\pm$15.4 | $\pm$18.5 |  | $\pm$23.5 | $\pm$20.1 | $\pm$21.7 |  |

### A.2 MBPO with ground truth models

MBPO did not perform well in our online experiments. If the fault is the accuracy of the learned environment models, it could be improved with approaches such online meta-learning (Finn et al., ). We investigated the learned models for the Pendulum-v1 experiments manually. We found these models to be very accurate, since the environment dynamics are simple. As stated in §[6.1](#S6.SS1), MBPO’s fails to address CF due to the way it samples data for training its policies.

To concretely verify that the accuracy of MBPO learned models is not the reason it underperforms, we instantiated an MBPO agent with access to the ground truth environment dynamics and context traces, which we call Ideal MBPO. We compare the performance of Ideal MBPO vs. standard MBPO in [Table 5](#A1.T5). The performance of these two agents is widely similar. This confirms the fact that the learning algorithm itself is the problem, and not the learned models.

**Table 5: Average and 95th percentile confidence ranges for episodic returns for different algorithms and conditions in the Pendulum-v1 environment with external wind processes. An MBPO agent with access to the ground truth model performs similarly to the MBPO model.**
|  | Online Learning |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | LCPO | MBPO | Ideal MBPO | A2C | Best Oracle |
| Context Trace 1 | -190 | -325 | -320 | -203 | -187 (SAC) |
| $\pm$1.29 | $\pm$47.2 | $\pm$44.2 | $\pm$10.2 |  |  |
| Context Trace 2 | -357 | -843 | -870 | -373 | -376 (DDQN) |
| $\pm$6.38 | $\pm$24.2 | $\pm$56.4 | $\pm$24.2 |  |  |
| Context Trace 3 | -237 | -603 | -637 | -330 | -203 (SAC) |
| $\pm$14.6 | $\pm$213 | $\pm$90.0 | $\pm$111 |  |  |
| Context Trace 4 | -383 | -553 | -555 | -395 | -357 (SAC) |
| $\pm$7.73 | $\pm$90.9 | $\pm$84.1 | $\pm$14.0 |  |  |

### A.3 Action Space

Pendulum-v1 (and Mujoco environments) by default have continuous action spaces, ranging from $-2$ to $2$ for Pendulum-v1. We observed instability while learning policies with continuous policy classes even without external wind, and were concerned about how this can affect the validity of our experiments with wind, which are considerably more challenging. As the action space is tangent to our problem, we discretized the action space to $15$ atoms, spaced equally from the minimum to the maximum action in each dimension. This stabilized training greatly, and is not surprising, as past work (Tang & Agrawal, ) supports this observation. The reward metric, continuous state space and truncation and termination conditions remain unchanged.

We provide the achieved episodic return for all baselines in §[6](#S6) in [Table 6](#A1.T6), over 10 seeds for the Pendulum-v1 envioronment, which we can compare to SB3 (Raffin et al., ) and RL-Zoo (Raffin, ) reported figures. These experiments finished in approximately 46 minutes. A2C, DDQN and SAC were trained for 8000 epochs, and TRPO was trained for 500 epochs. Evaluations are on 1000 episodes. As these results show, the agents exhibit stable training with a discretized action space.

**Table 6: Average episodic returns and 95th percentile confidence ranges for different algorithms in the Pendulum-v1 environment with discretized and continuous action space.**
| Episodic Reward | A2C | TRPO | DDQN | SAC | MBPO |
| --- | --- | --- | --- | --- | --- |
| Discrete | -165+-5 | -166+-8 | -149+-2 | -146+-2 | -161+-3 |
| SB3 (Raffin et al., ) + RL-Zoo (Raffin, ) | -203 | -224 | — | -176 | — |

### A.4 Experiment Setup

We use Gymnasium (v0.29.1) and Mujoco (v3.1.1). Our implementations of A2C, TRPO, DDQN and SAC use the Pytorch (Paszke et al., ) library.
[Table 7](#A1.T7) is a comprehensive list of all hyperparameters used in training and the environment.

All baselines were tuned on Pendulum-v1. Several LCPO hyperparameters were copied from TRPO, SAC and A2C (namely, entropy target, entropy learning rate, damping coefficient, rollout length, $\lambda$, $\gamma$) and the rest ($c_{anchor}$, $c_{recent}$ and base entropy) were tuned with an informal search with a separate context trace (not in the evaluation set) in Pendulum-v1. The OOD threshold $\sigma$ was not tuned with a search.

MBCD’s sensitivity for detecting environment changes is a tunable hyperparameter; we tuned it by trying 6 values in a logarithmic space spanning $10^{1}$ to $10^{6}$ on the evaluation context traces woth Pendulum-v1, and chose the best performing hyperparameter on the test set. This tuning strategy is not realistic in practice, but even with this advantage MBCD fails to perform well over the diverse set of contexts. MBCD still endlessly spawned new policies for other environments, and therefore we limited the maximum number of models to 7.

**Table 7: Training setup and hyperparameters for gymnasium environments with external wind.**
| Group | Hyperparameter | Value |
| --- | --- | --- |
| Neural network | Hidden layers | (64, 64) |
| Hidden layer activation function | Relu |  |
| Output layer activation function | Actors: Softmax, Critics and DDQN: Identity mapping |  |
| Optimizer | Adam ($\beta_{1}=0.9$, $\beta_{2}=0.999$) (Kingma & Ba, ) |  |
| Learning rate | Actor: 0.0004, Critic and DDQN: 0.001 |  |
| Weight decay | $10^{-4}$ |  |
| RL training (general) | Random seeds | 5 |
| $\lambda$ (for GAE in A2C and TRPO) | 0.9 |  |
| $\gamma$ | 0.99 |  |
| A2C | Rollout per epoch | 200 |
| TRPO | Rollout per epoch | 3200 |
| Damping coefficient | 0.1 |  |
| Stepsize | 0.01 |  |
| DDQN | Rollout per epoch | 200 |
| Batch Size | 512 |  |
| Initial fully random period | 1000 epochs |  |
| $\epsilon$-greedy schedule | 1 to 0 in 5000 epochs |  |
| Polyak $\alpha$ | 0.01 |  |
| Buffer size $N$ | All samples ($N=20M$ or $N=8M$) |  |
| SAC | Rollout per epoch | 200 |
| Batch Size | 512 |  |
| Initial fully random period | 1000 epochs |  |
| Base Entropy | 0.1 |  |
| Entropy Target | $0.1\ln(15)$ |  |
| Log-Entropy Learning Rate | 1e-3 |  |
| Polyak $\alpha$ | 0.01 |  |
| Buffer size $N$ | All samples ($N=20M$ or $N=8M$) |  |
| LCPO | Rollout per epoch | 200 |
| Base Entropy | 0.03 |  |
| Entropy Target | $0.1\ln(15)$ |  |
| Log-Entropy Learning Rate | 1e-3 |  |
| Buffer Size $n_{b}$ | 1% of samples ($200K$ or $80K$) |  |
| Damping coefficient | 0.1 |  |
| $c_{anchor}$ | 0.0001 |  |
| $c_{recent}$ | 0.1 |  |
| $\sigma$ | 1 |  |
| MBCD | h | 1000 (default was 100/300) |
| max_std | 3 (default was 0.5) |  |
| N (ensemble size) | 5 |  |
| NN hidden layers | (64, 64, 64) |  |
| MBPO | M (model rollouts) | 512 |
| N (ensemble size) | 5 |  |
| k (rollout length) | 1 |  |
| G (gradient steps) | 1 |  |
| Online EWC | averaging weight $\beta$ | 0.00007 (equivalent to $\sim 3M$ samples at rollout=200) |
| scaling factor $\alpha$ | 0.1 |  |

## Appendix B Straggler mitigation

Figure: Figure 8: Illustration of a request proxy with hedging.
Refer to caption: /html/2302.02182/assets/x10.png

Figure: Figure 9: Request arrival rate and processing time per context.
Refer to caption: /html/2302.02182/assets/x11.png

### B.1 Full results

[Figure 10](#A2.F10) plots the tail latency across experiment time in the straggler mitigation environment for both contexts.

Figure: Figure 10: Tail latency with 95th percentile confidence intervals as training progresses (lower is better). We consider an initial learning period of 3.5 million samples. LCPO remains close to the oracle throughout contexts, while baselines suffer from non-stationarity.
Refer to caption: /html/2302.02182/assets/x12.png

### B.2 Experiment Setup

We use the straggler mitigation environment from prior work (Hamadanian et al., ), with a similar configuration except with 9 actions (timeouts of $600^{ms}$ and $1000^{ms}$ added). Similar to §[A.4](#A1.SS4), our implementations of A2C, TRPO, DDQN and SAC use the Pytorch (Paszke et al., ) library.
[Table 8](#A2.T8) is a comprehensive list of all hyperparameters used in training and the environment.

All baselines were tuned on a separate workload. LCPO hyperparameters were copied from the gymnasium experiments, except for base entropy which was tuned with an informal search with a separate workload (not in the evaluation set).

**Table 8: Training setup and hyperparameters for straggler mitigation experiments.**
| Group | Hyperparameter | Value |
| --- | --- | --- |
| Neural network | Hidden layers | $\phi$ network: (32, 16) |
| $\rho$ network: (32, 32) |  |  |
| Hidden layer activation function | Relu |  |
| Output layer activation function | Actors: Softmax, Critics and DDQN: Identity mapping |  |
| Optimizer | Adam ($\beta_{1}=0.9$, $\beta_{2}=0.999$) (Kingma & Ba, ) |  |
| Learning rate | 0.001 |  |
| Weight decay | $10^{-4}$ |  |
| RL training (general) | Random seeds | 10 |
| $\lambda$ (for GAE in A2C and TRPO) | 0.95 |  |
| $\gamma$ | 0.9 |  |
| A2C | Rollout per epoch | 4608 |
| TRPO | Rollout per epoch | 10240 |
| Damping coefficient | 0.1 |  |
| Stepsize | 0.01 |  |
| DDQN | Rollout per epoch | 128 |
| Batch Size | 512 |  |
| Initial fully random period | 1000 epochs |  |
| $\epsilon$-greedy schedule | 1 to 0 in 5000 epochs |  |
| Polyak $\alpha$ | 0.01 |  |
| Buffer size $N$ | All samples ($N=21M$) |  |
| SAC | Rollout per epoch | 128 |
| Batch Size | 512 |  |
| Initial fully random period | 1000 epochs |  |
| Base Entropy | 0.01 |  |
| Entropy Target | $0.1\ln(9)$ |  |
| Log-Entropy Learning Rate | 1e-3 |  |
| Polyak $\alpha$ | 0.005 |  |
| Buffer size $N$ | All samples ($N=21M$) |  |
| LCPO | Rollout per epoch | 128 |
| Base Entropy | 0.01 |  |
| Entropy Target | $0.1\ln(9)$ |  |
| Log-Entropy Learning Rate | 1e-3 |  |
| Buffer Size $n_{b}$ | $210K$ |  |
| Damping coefficient | 0.1 |  |
| $c_{anchor}$ | 0.0001 |  |
| $c_{recent}$ | 0.1 |  |
| MBCD | h | 300000 (default was 100/300) |
| max_std | 3 (default was 0.5) |  |
| N (ensemble size) | 5 |  |
| NN hidden layers | (64, 64, 64) |  |
| MBPO | M (model rollouts) | 512 |
| N (ensemble size) | 5 |  |
| k (rollout length) | 1 |  |
| G (gradient steps) | 1 |  |
| Online EWC | averaging weight $\beta$ | 0.00007 (equivalent to $\sim 2M$ samples at rollout=128) |
| scaling factor $\alpha$ | 0.1 |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: **Decision:**

Although the paper has some weaknesses, the positives outweigh them. The paper is well-executed and it is a good contribution to continual reinforcement learning. Therefore, I recommend an **acceptance.**

**Areas of improvement:**

* The paper should discuss how [1] and [2] are related to the proposed approach. [1] uses a KL constraint on the current policy to be closer to the global policy, and [2] learns a global value function as a baseline estimate for all contexts on top of which the learning happens. It seems like the global policy and the global value functions induce the regularization effect on the past that the paper proposes.
* In Sec 4, the episodic returns are -4 and -6 and not -3 and -5 as mentioned in the paper.
The ablation experiments and the results presented in the appendix can be summarized as one-line bullet points in the main paper.

**Questions:**

* Deep neural networks are known to overfit to the early samples [3], wouldn’t regularizing with respect to the past data exacerbate this process?
* How is the context information different from a task ID?
* In Sec 4, does the A2C use both context and state information to learn the policy?
* Is it possible to approximate the proposed objective in the PPO style as opposed to the TRPO style to avoid using second-order information?
* What happens if the reverse KL is used in the constraint proposed in Eq. 1?
* What is a warm-up period and why is it needed in the experiments?

**References:**

[1] Teh, Yee, et al. "Distral: Robust multitask reinforcement learning." Advances in neural information processing systems 30 (2017).

[2] Anand, Nishanth, and Doina Precup. "Prediction and control in continual reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

[3] Nikishin, Evgenii, et al. "The primacy bias in deep reinforcement learning." International conference on machine learning. PMLR, 2022.
soundness: 3
strengths: * The paper is well-written and easy to read: the introduction is thorough, the related works section is exhaustive, preliminary and the LCPO sections provide adequate details, and the methodology section is well-presented.
* The specific kind of non-stationary problem that the paper considers is well-described. And, the proposed solution fits like a glove to this problem. The approach is also intuitive and parallels the TRPO algorithm.
* The authors evaluate their approach against numerous baselines on several tasks, which provides substantial confidence in the presented results and conclusions. The ablation studies are also welcome, as they offer more insights into the proposed approach.
* The illustrative examples discussed throughout the paper offer an intuitive understanding of various complex ideas – the authors have done a great job of including them.
summary: The paper proposes Locally Constrained Policy Optimization (LCPO), a policy gradient algorithm for non-stationary reinforcement learning where the context changes are observed and exogenous. LCPO’s updates are constrained to preserve the policy trained on the past contexts while it maximizes returns on newer samples. Two replay buffers are maintained in order to store the samples belonging to the past contexts and the current context. Whenever the distribution of samples is different in these two buffers, which is detected using an OOD threshold-based detector, then a “regularization” update is performed to preserve past learnings. The proposed approach is tested on a range of environments with a whole suite of baselines to demonstrate its benefits.
weaknesses: * The biggest weakness is that the paper considers a specific kind of non-stationarity in the study. Although exogenous processes could be common in real-world scenarios, many other kinds of non-stationarities exist and it is unclear whether the proposed approach works well in those cases. Besides, the OOD detector is an important piece in the algorithm and its effectiveness depends on the threshold value – a hyperparameter that is environment-dependent and unknown beforehand.
* The approach also assumes that the context is observed, although it is not used as a task detector or task boundary detector. In many practical cases, the context information is unavailable and it is unclear how one could use the proposed approach in that case.
* The proposed constrained optimization problem is approximately solved, which in itself is fine, but when paired with the fact that the approach doesn’t use TRPO-style local constraint, the training procedure could be unstable.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: The proposed algorithm places a constraint that aims to keep the new policy relevant for old samples. This constraint replaces the TRPO policy update constraint. The original TRPO formulation results in an approximate natural gradient due to which in TRPO the gradient update does not in principle depend on the parameterization of the (neural network) policy. This property is lost here? Do we get other properties?
soundness: 4
strengths: Well written paper. Clear definitions and derivations. The paper explains well all the design decisions in the proposed approach.

The main algorithmic idea of sampling states from the history buffer that differ from the samples currently collected from the environment, and, then using these samples to constrain the policy update is simple and based on the experimental results appears very effective.

I like that the resulting algorithm is an almost straight forward modification of TRPO but for a completely different problem. Note that while being simple the main idea is only obvious in hindsight.
summary: The paper addresses "catastrophic forgetting" in continual reinforcement learning with non-stationary dynamics. The dynamics depends on the current context which is observable but can change arbitrarily. A problem in these kind of settings is that the agent over time forgets how to behave in prior contexts. To address this problem the paper proposes a new on-policy algorithm that aims to keep policy updates such that the policy for prior contexts does not change. The proposed approach does not need task labels but only an out-of-distribution detector.
weaknesses: I could not find major weaknesses in the paper.

This is not a weakness but just a comment:
Choosing of the out of distribution (OOD) detector is one obvious design decision when running the method in a specific environment. Section 6.2 discusses different thresholds for the OOD detector. Although OOD detection has been discussed in the supervised learning literature, the paper could provide a more thorough discussion on which kind of OOD detectors could be beneficial in which kind of RL environments.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Have you tried learning the OOD proxy? e.g., using weighted norm with tunable parameters?

2. You do discuss the differences between your proposed OOD detector and CPD, but have you tried comparing the "whole package"? i.e. using LCPO with CPD.
soundness: 3
strengths: I think the paper is well-written, easy to understand and comprehensive. Notably, the empirical sections present the advantages of the proposed algorithm. For my opinion, with a few minor changes, it passes the acceptance threshold.

### Experiments
The experiments are extensive in terms of both benchmarks and baselines, and show good performance for the proposed method.

### Illustrative Example:
I think this empirical analysis helps to understand what's happening, and shows the value of the method.
summary: The paper introduces an online RL method for environments with non-stationary (but observable) context, which affects the dynamics/reward. The main difficulty in such settings is catastrophic forgetting; while training the agent on an unobserved context, it forgets the optimal behavior in past contexts. The authors aim to solve this issue by constraining the agent to maintain current behavior for previously-observed contexts, which is possible due to an OOD detector -- basically a distance measure over contexts. To test the proposed method, the authors use a single toy problem and a large amount of benchmarks, the results show that the method indeed surpasses other baselines.
weaknesses: **Paper exceeds the 10-page limit**

### Presentation
1. The references within the paper seem to have improper notation (using § instead of Sec.) 

2. I am not sure about the contribution of the opening sentence in the introduction, as this is not the first paper introducing CF, and I do not see any specific relation to the proposed method.

3. Since the method uses a few components, adding a block diagram could help, but is not mandatory in the main body. Such block diagram should show the interconnections at each time step between the buffer, new sampled data, OOD detector, and the agent (policy).

### Related Work:
While not exactly the same, maybe contextual MDPs could be related.

### Minor comments:

1. line 66: "These assumptions are rarely met and lead to poor performance in practice." -- is there any reference to support this claim? If it cannot be backed, you can alter the sentence to be more soft: "...and would likely lead to...".

2. line 268: "proof in §B in the Appendix." -- rephrase.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - How can you improve the clarity of the writing?
- How am I supposed to read Figure 3. What does the y-axis represent?
- Why have you used textual citations everywhere?
- What does the bolding in tables represent?
soundness: 2
strengths: Whilst this paper is slightly out of my expertise, I think the authors are addressing an important problem as
handling non-stationarity in online RL is important.
To the best of my knowledge, the method presented in this paper seems original.

I particularly liked Figure 1. I would suggest the authors move it earlier in the paper.
summary: This paper investigates online reinforcement learning (RL) in non-stationary environments, 
where the non-stationarity arises from a time-varying exogenous context process affecting the dynamics.
In particular, they study the catastrophic forgetting (CF) problem.
They propose a new method named locally constrained policy optimization (LCPO),
which combats CF by constraining policy updates for a particular context on prior data from other contexts.
They evaluate their method in the classic control tasks from MuJoCo and computer systems environments.
weaknesses: Overall, I found the paper quite hard to follow. 
The method seems fairly straightforward but the paper lacks a clear and easy-to-follow structure I would expect from an ICLR paper.
I am confused as to why Section 4 "Locally-Constrained Policy Optimization" and Section 4 "Methodology" are different sections.
I would recommend the authors combine these sections as they both explain the method. 
If needed, use sub-sections to structure the content.

Further to this, the figures are not explained thoroughly enough in the text. What does the y-axis in Figures 3 and 4 represent? I assume it is the cumulative distribution function but this should be stated in the paper. 
Further to this, how should the reader interpret the results?
The authors should explain how to interpret the plots, including what the steepness of curves represents, what does it mean
when curves are intersecting, etc?
I suggest the authors check out [rliable](https://github.com/google-research/rliable/tree/master) and use this to create figures.
I am also not sure at what environment step these figures were created for.
Again, this should be clear from the text.

Why are there confidence intervals for Figure 3a but not for Figures 3b or 4?
In Section 6.3, the authors claim that buffer sizes less than $n_b=500$ results in a drop in performance.
I am not sure you can make this claim without showing confidence intervals in Figure 4.

The authors have incorrectly cited throughout the paper. 
The authors should take time to become familiar with when to use textual citations and when to use parenthetical citations.
If using the `natbib` package, the authors can use `\citet` for textual citations and `\citep` for parenthetical citations.
The authors almost always cite such that they should be using parenthetical citations but they use textual citations throughout.

What does the bolding in the table represent?
Does it represent statistical significance under a t-test?
This should be clear from the caption.

# Minor comments
- "Mujoco" should be "MuJoCo"
- Line 191 - the policy is stochastic but on line 112 it's a deterministic mapping.

Meta-review: The paper proposes Locally Constrained Policy Optimization (LCPO), which is a policy gradient method for changing environments. The reviewers highlight that the problem setting is well specified and reasonable, the proposed algorithm is a straightforward extension of TRPO, and that the illustrative examples are valuable. The primary highlighted weaknesses was around the context availability/information patterns and the OOD threshold method. 

Three of the four reviewers recommend acceptance with the last reviewer leaning reject. The last reviewer has primarily raised issues around paper formatting. I recommend acceptance of the paper, but I would ask the authors to make sure they have addressed all of these formatting issues.

Decision: Accept (Spotlight)