Title: A Bayesian Fast-Slow Framework to Mitigate Interference in Non-Stationary Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Non-stationary MDPs

Continual RL
Abstract: Given the ever-changing nature of the world and its inhabitants, agents must possess the ability to adapt and evolve over time. Recent research in Given the ever-changing nature of the world and its inhabitants, agents must possess the ability to adapt and evolve over time. Recent research in non-stationary MDPs has focused on addressing this challenge, providing algorithms inspired by task inference techniques. However, these methods ignore the detrimental effects of interference, which particularly harm performance in contradictory tasks, leading to low efficiency in some environments. To address this issue, we propose a Bayesian Fast-Slow Framework (BFSF) that tackles both cross-task generalization and resistance to cross-task interference. Our framework consists of two components: a 'fast' policy, learned from recent data, and a 'slow' policy, learned through meta-reinforcement learning (meta-RL) using data from all previous tasks. A Bayesian estimation mechanism determines the current choice of 'fast' or 'slow' policy, balancing exploration and exploitation. Additionally, in the 'fast' policy, we introduce a dual-reset mechanism and a data relabeling technique to further accelerate convergence when encountering new tasks. Experiments demonstrate that our algorithm effectively mitigates interference and outperforms baseline approaches.

A Bayesian Fast-Slow Framework to Mitigate Interference in Non-Stationary Reinforcement Learning

Abstract

Given the ever-changing nature of the world and its inhabitants, agents must possess the ability to adapt and evolve over time. Recent research in Given the ever-changing nature of the world and its inhabitants, agents must possess the ability to adapt and evolve over time. Recent research in non-stationary MDPs has focused on addressing this challenge, providing algorithms inspired by task inference techniques. However, these methods ignore the detrimental effects of interference, which particularly harm performance in contradictory tasks, leading to low efficiency in some environments. To address this issue, we propose a Bayesian Fast-Slow Framework (BFSF) that tackles both cross-task generalization and resistance to cross-task interference. Our framework consists of two components: a 'fast' policy, learned from recent data, and a 'slow' policy, learned through meta-reinforcement learning (meta-RL) using data from all previous tasks. A Bayesian estimation mechanism determines the current choice of 'fast' or 'slow' policy, balancing exploration and exploitation. Additionally, in the 'fast' policy, we introduce a dual-reset mechanism and a data relabeling technique to further accelerate convergence when encountering new tasks. Experiments demonstrate that our algorithm effectively mitigates interference and outperforms baseline approaches. Code is available at https://github.com/cedesu/BFSF .

Reinforcement Learning (RL) in non-stationary environments has long attracted significant attention, leading to the emergence of research areas such as continual RL [30, 19] and non-stationary MDPs [7, 24, 31]. These areas approach challenges from different perspectives. For example, catastrophic forgetting, a well-known issue in continual RL, arises due to limited memory storage. In contrast, research on non-stationary MDPs focuses on understanding the underlying dynamics of the environment to facilitate better adaptation across varying contexts.

One critical challenge in non-stationary environments is interference, where the learning process is negatively impacted by experiences from previous tasks, as illustrated in Figure 1. This interference arises primarily because task boundaries are either unknown or absent in the streaming task setting. Many real-world problems exhibit such non-stationarity and suffer from interference. For instance, a UAV must adapt its behavior under varying weather conditions [27, 26]. Similarly, the evolving regime of the stock market can be viewed as a time-varying environment, where an effective strategy

Interferenceoftask1and task2

Learningprocess

00

000

000

Task 1

Task 2

Task 3

Time

Unknownchangepoint

(a) The diagram of interference in non-stationary MDPs.

Cheetah-Dir

2000

1500

1000

500

0

-500

Fast+ slow(BFSF)

-1000

Upper bound (each task is trained individually)

Slow

0

5000

10000

15000

20000

25000

(b) The learning curve of how BFSF alleviates the problem of interference.

must stay adaptive while leveraging past experience [13, 3]. These real-world scenarios underscore the urgent need for a framework that can operate efficiently in non-stationary MDPs with interference.

Despite its significant negative impact on performance, the issue of interference has been largely overlooked in the literature. Some existing works [18] address interference from the perspective of representation, while others [20] discuss the inverse interference of current tasks on previously learned tasks, a phenomenon referred to as catastrophic forgetting in continual RL. This lack of attention to cross-task interference is concerning, as it can severely degrade performance in successive tasks. In this work, we specifically analyze the effects of interference and propose effective strategies to mitigate its detrimental impacts.

To address the interference problem, we introduce the Bayesian Fast-Slow Framework (BFSF). This framework dynamically selects between two learning strategies: a 'fast' policy, which quickly adapts to new tasks using recent data, and a 'slow' policy, which is learned through meta-RL and captures knowledge from historical data. Unlike previous approaches that focus solely on latter, often leading to severe interference in the face of sudden task changes, our framework not only mitigates interference but also preserves the advantages of meta-RL. A Bayesian estimation mechanism is employed in each epoch to decide which policy, fast or slow, is more promising based on recent history. Only the most recent returns are used to update the Bayesian estimates, ensuring that outdated data does not influence the decision.

We also identify that the 'fast' policy can sometimes underperform. One reason is that neural networks often experience performance degradation when trained on data from different distributions, a common issue in non-stationary environments. To address this, we introduce a dual-reset mechanism that periodically reinitializes one of the dual networks to prevent degradation, while alternating between the networks to ensure stable performance. Another challenge is that learning from scratch typically requires extensive online interaction. To mitigate this, we propose data relabeling, utilizing historical data from previous tasks to enhance learning efficiency and improve performance in few-shot settings when facing new tasks.

In summary, our contributions are twofold: i) We analyze the interference problem in non-stationary MDPs. ii) We propose the Bayesian Fast-Slow Framework (BFSF), which combines a fast policy, enhanced by a dual-reset mechanism and data relabeling, to efficiently handle recent tasks, and a slow policy for cross-task generalization. Through Bayesian estimation, we effectively address interference and improve overall performance. Experimental results demonstrate BFSF's superiority in resisting interference and outperforming baseline methods across various non-stationary environments.

1 Preliminaries

Notations and problem definition A Markov Decision Process (MDP) is defined as M = ⟨ S, A, P, R ⟩ , where S and A represent the state and action spaces, respectively. The transition function of the environment is denoted as P , and R represents the reward function. The expected return of a policy is given by E [ ∑ ∞ t =0 R t ] . In non-stationary MDPs, the underlying MDP evolves over time. These changes can occur sequentially, such as M 1 , M 2 , · · · , or gradually over time. The objective is to maximize the expected return, E [ ∑ ∞ t =0 R t ] with the evolving dynamics of the MDP.

Context-based policy In a standard MDP, the policy function is defined as π ( a | s ) , which determines the probability of selecting action a given state s . In non-stationary settings, a context-based policy is introduced to adapt to varying environments. This policy is denoted as π ( a | s, c ) , where c is the context, a set of trajectories related to the current environment. A trajectory consists of the sequence of states, actions, and rewards at each timestep, expressed as s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , s 2 , a 2 , r 2 , . . . . In implementation, the context is collected from recent interactions, reflecting the underlying MDP.

The technique of learning the context-based policy has been extensively studied in the field of metareinforcement learning (meta-RL) [24, 37]. However, meta-RL differs from the non-stationary MDPs setting in this work. In meta-RL, task information is explicitly available, and there is no continuous adaptation process. Context-based meta-RL methods typically map the contextual information, often represented as transition data, into a latent space Z . By assigning a latent variable z ∈ Z to represent the task, this approach effectively frames the problem as a partially observable MDP (POMDP) [16], where z constitutes the unobserved portion of the state. In meta RL, PEARL learns the posterior distribution q ( z | c ) , which means the posterior latent variable distribution given the context c , and uses posterior sampling to sample z to integrate these latent variables with off-policy RL algorithms.

Algorithm 1 Bayesian fast-slow framework (BFSF)

2 Bayesian Fast-Slow Framework (BFSF)

The Bayesian Fast-Slow Framework (BFSF) is designed to mitigate interference by dynamically deploying either a 'fast' policy, which learns from recent data, or a 'slow' policy, trained using meta-RL principles. The term 'fast' arises from its 'fast-adaptation' ability to learn directly and efficiently from recent data. In contrast, the 'slow' policy enables cross-task understanding and generalization, which may hinder training speed, especially when the number of observed tasks is limited in early phase. The decision is made based on Bayesian estimation of current expected return.

As illustrated in Algorithm 1, during each epoch, Bayesian inference is applied to estimate the posterior expected return using the recent return history, for both the fast and slow policies. Let R i denote the return obtained in the i -th epoch, and choice i indicate whether the 'fast' or 'slow' policy was selected during that epoch. During the online interaction phase, the policy with the higher estimated posterior value is selected, aiming to generate higher-quality experience. At the end of each epoch, both the fast and slow policies are updated using their respective replay buffers. The detailed computation of the Bayesian posterior, described in lines 5 and 6 of Algorithm 1, is elaborated in Section 2.1. In addition to the online interaction and Bayesian estimation, the training process for the 'fast' policy is detailed in Algorithm 2, while the 'slow' policy is trained according to the context-based meta-RL algorithm PEARL [24], which is one of the first context-based methods and serves as the baseline for numerous subsequent works. The visualization of choosing the 'fast' or 'slow' policy is provided in Appendix E.

2.1 Bayesian Inference

The detailed update rule for Bayesian posterior estimation is outlined below. For simplicity, assume that the recent returns of a given policy, R i 1 , R i 2 , . . . , are approximately drawn from a normal distribution N ( µ, 1 /ϕ ) , where ϕ is a constant. While this assumption is commonly used, other distributional forms could also be considered depending on the context. The prior distribution for the parameter µ is assumed to follow µ ∼ N ( µ 0 , 1 /ϕ 0 ) . The posterior estimation of µ then follows the standard derivation below, as detailed in Appendix D.

To better interpret the result of Bayesian inference, note that the posterior mean µ 1 can be decomposed:

It shows that the posterior mean µ 1 is a weighted average of the prior mean µ 0 and the sample average R . As more samples are collected, the weight shifts toward trusting the sample average R . Conversely, when only a few samples are available, the posterior relies more heavily on the prior µ 0 .

2.2 'Fast' Policy Learning

The 'fast' policy, learned from recent data, is critical for mitigating interference in non-stationary MDPs. However, the standard learning paradigm often encounters challenges under these conditions. To address these issues, we propose specific structural designs that significantly enhance the efficiency and adaptability of the 'fast' policy.

Algorithm 2 Training process of the 'fast' policy.

1: Input: The 'fast' policy π fast (including the dual policies π (1) fast , π (2) fast ), a contextual dynamics model M , current epoch e , reset frequency ν

2: Output: The updated π fast

3: # Dual-reset mechanism

4: if e mod ν = 0 then

5: π (1) fast , π (2) fast = π (2) fast , Init( π (1) fast )

6: end if

7: # Data relabeling

8: Relabel the recent data by M using the recent trajectories as the context.

9: # Training process

10: Train π (1) fast , π (2) fast using the relabeled data

Dual-reset Mechanism A key challenge in continual learning is the performance degradation of neural networks when trained on successive tasks. One common observation is that the learning curve for the second task often struggles to converge to an optimal point, even when task difficulty is comparable (as detailed in Appendix C.2). This phenomenon is also noted and studied in ITER [14].

To address this, we propose the dual-reset mechanism, as outlined in Algorithm 2, which mitigates performance degradation by periodically reinitializing the model. However, to avoid the inferior performance typically observed immediately after reinitialization, we introduce a dual-model system. This ensures that during any interaction phase, even directly after initialization, a fully trained model is always available for deployment.

Data Relabeling Another challenge lies in the limited recent data available for the 'fast' policy, which is necessary for rapid adaptation in non-stationary environments. This small dataset size may not support learning a robust policy, especially over prolonged training periods, in contrast to the 'slow' policy that can utilize the entire historical dataset. As a result, the 'fast' policy tends to perform significantly worse, as shown in Figure 5. To address this limitation, we incorporate data relabeling, which significantly enhances the amount of usable data, enabling the learning of a stronger policy.

Specifically, the data relabeling process relies on maintaining a context-based dynamics model. This model takes the context, state, and action as input, and outputs the relabeled next state and reward. Leveraging the context-based property, it becomes possible to relabel historical data from other tasks into the context of the current task. By combining relabeled data with the original dataset, the learning efficiency of the 'fast' policy is substantially improved. Further details can be found in Appendix C.1.

2.3 Theoretical Analysis

In this section, we provide a theoretical analysis of the sub-optimality bound of the Bayesian Fast-Slow Framework (BFSF) and present Theorem 2.1.

Theorem 2.1.

where M,M ′ denote the original and relabeled MDPs, and p M , p M ′ are their transition functions. H is the horizon, r max , V max are the maximum reward and value, r diff represents the maximum reward gap between M,M ′ , U r,r ′ ( π ) is defined as E ( s,a ) ∼ ρ π M [ r ′ ( s, a ) -r ( s, a )] , and D ℓ 1 is the L1 distance.

The following provides a proof sketch and interpretation of Theorem 2.1. First, the suboptimality is defined as the minimum between the fast and slow policies, with the Bayesian estimation serving as an unbiased estimate. We focus on the suboptimality of the 'fast' policy, | η M ( π ∗ M ) -η M ( π ∗ M ′ ) | . This suboptimality can then be decomposed into several components as detailed in Appendix A.1. For ease of comprehension, the first term represents the gap in optimal expected return between the relabeled and original MDPs. The second and third terms arise from the performance difference of the same policy under different dynamics. A further analysis of the bound that incorporates optimization error is provided in Appendix A.2.

3 Experiments

The Bayesian Fast-Slow Framework (BFSF) is designed to address the complexities of non-stationary MDPs, specifically tackling the interference issue while ensuring generalization across experiences from different tasks. In this section, we focus on two main questions: i) How does BFSF overcome interference in non-stationary environments? ii) How does BFSF perform in real-world scenarios? iii) How do the individual components of BFSF contribute to improving performance?

To answer the first question, we provide experimental comparisons with baselines in Section 3.1, demonstrating the superiority of BFSF in mitigating interference. For the second question, we design an infinite-world simulation to better approximate real-world conditions and evaluate the performance

Cheetah-Dir

Cheetah-Vel

Ant-Dir

2000

1500

-100

1250

1000

BFSF

LILAC (only context)

ITER

1000

-200

750

500

-300

500

0

400

250

-1000

-500

LILAC (only context)

BFSF

-500

LILAC (only context)

BFSF

-250

0

-1500

ITER

I

600

1

！

ITER

500

0

5000

10000

15000

20000

25000

0

2000

4000

0009

8000

10000

12000

0

2000

4000

0009

8000

10000

12000

Walker-Rand-Params

Metaworld Reach

1000

BFSF

2000

200

LILAC (only context)

800

ITER

400

1500

600

-600

1000

400

800

500

BFSF

200

BFSF

-1000

LILAC (only context)

ITER

T

T

！

0

0

I

LILAC (only context)

ITER

0

2000

4000

6000

8000

10000

12000

0

2000

4000

6000

8000

10000

12000

0

2000

4000

0009

8000

10000

12000

of both BFSF and the baselines. Finally, for the third question, we conduct detailed ablation studies on the modules within BFSF, offering evidence of their effectiveness.

3.1 Mains Results

We evaluate the Bayesian Fast-Slow Framework (BFSF) on five MuJoCo environments and one Meta-World environment. The MuJoCo environments [28] focus on robotic locomotion and are based on the MuJoCo simulator, while Meta-World [35] is a benchmark designed for Multi-Task and meta-RL, specifically with robot manipulation tasks. These environments require adaptation across different reward functions (e.g., walking direction for Cheetah-Dir and Ant-Dir, target velocity for Cheetah-Vel, and goal location for Ant-Goal and Meta-World Reach), or across different dynamics (e.g., environment parameters for Walker-Rand-Params). These meta-RL environments are widely used in the meta-RL literature and are well-suited for non-stationary MDPs as well. We set the switching frequency of the underlying task to 2000 episodes.

We compare BFSF with four reproduced baselines. LILAC [31] is an algorithm for non-stationary MDPs that uses meta-RL techniques. It learns a latent variable to discriminate between tasks based on experiences. ITER [14] proposes an iterative approach to relearn the neural network, aiming to overcome non-stationarity. CEMRL [5] learns a task encoder from the gradients of a decoder

and provides the task encoding to downstream RL. CoMPS [4] continuously alternates between two subroutines: learning a new task using RL and performing completely offline meta-learning to prepare for subsequent task learning.

As shown in Figure 2, BFSF outperforms the baselines in non-stationary environments. The task switching is indicated by the gray dashed lines for clarity. While we conduct experiments with a fixed switching interval, it is important to note that our algorithms are designed for the general setting where the task distribution and switching timing are completely unknown to the agent. For comprehensiveness, we also experiment with an unfixed switching interval in Section 3.3.

In general, all algorithms show gradual performance improvements within a single task phase but experience a sudden performance drop immediately after task switches. This decay is expected, as the new task is unfamiliar to the agent. However, we observe that the learning curve in the second phase does not increase as quickly as in the first, which we refer to as the interference phenomenon.

BFSF effectively mitigates interference, leading to better overall performance across continual phases. LILAC, based on meta-RL methods, provides good cross-task generalization. However, it fails to address interference, resulting in slower learning during the second task. ITER's iterative relearning approach is a solid defense against interference, but relying solely on this approach leads to the learning of elementary policies, which hinders further generalization and improvement. For clarity, we only compare the curves of two baselines in the given figure, while a full comparison of average return is provided in Table 1. Full experiment results on the learning curve is provided in Appendix B.2.

3.2 Infinite-World Simulation

While the main results in Section 3.1 demonstrate the superiority of BFSF in mitigating interference and achieving cross-task generalization, it remains unclear how such methods perform in more realistic scenarios. In the real world, there are typically no explicit task boundaries, no fixed task initializations, and no finite set of predefined tasks. To better approximate these characteristics, we introduce an Infinite-World Simulation in MuJoCo, an environment where the agent operates in a non-episodic, continuous manner without resets.

Unlike traditional MuJoCo benchmarks, where each episode ends and resets after a fixed number of steps (e.g., every 1000 steps), our infinite-world environment allows the agent to move seamlessly through a boundless plane without ever being reinitialized. This design leads to a non-episodic interaction flow, closely mimicking the persistent nature of real-world settings. To support this, we implement a dynamic terrain loading module that handles environment generation on the fly, avoiding memory overload while preserving the illusion of an endless space.

We design two signature environments to evaluate BFSF and baselines under this setup. Ant-Dir-Inf is a non-episodic, infinite-world variant of the standard Ant-Direction environment, where the agent is required to walk alternately left and right across the plane; each time a directional goal is reached, the target direction flips. Ant-Goal-Inf is derived from Ant-Goal, where the target moves along a circular trajectory of infinite radius, requiring the agent to constantly adjust and track it over time.

The corresponding results are reported in Table 1. In addition, Figure 3 visualizes the trajectories of different methods. BFSF consistently follow the evolving goal direction, while baseline methods react more slowly. This highlights BFSF's effectiveness in non-episodic and task-free environments.

3.3 Ablation Studies

The ablation studies on each module of BFSF are conducted to answer the question: How do the individual components of BFSF contribute to improving performance? The results show that the 'fast' policy, as a whole, alleviates interference and enhances overall performance in non-stationary MDPs. Additionally, the dual-reset mechanism and relabeling further support the 'fast' policy by enabling more effective learning.

Unfixed Switching Interval We conducted experiments with an unfixed switching interval, as shown in Figure 4(a). The overall performance exhibits a pattern similar to that observed in the main experiments with a fixed switching interval: the interference issue is mitigated by BFSF, and the agent's performance continues to improve as training progresses.

the trajectory of x position in Ant-Dir-lnf

the trajectory of x, y position in Ant-Cir-lnf

BFSF

60000

BFSF

30000

LILAC

LILAC

ITER

ITER

CEMRL

50000

CEMRL

CoMPS

CoMPS

20000

targetxposition

40000

xposition

position

10000

30000

>

0

20000

10000

-10000

0

0

100

200

300

400

500

-30000

-20000

-10000

0

10000

20000

epoch xposition

Cheetah-Dir

(a) BFSF with an unfixed switching interval.

(b) The ablation study of BFSF.

2000

BFSF

2000

LILAC

1500

1500

1000

1000

500

500

0

0

BFSF

-500

-500

withoutfastpolicy

withoutdual-reset

-1000

-1000

without relabeling

withoutBayesianinference

0

5000

10000

15000

20000

25000

0

5000

10000

15000

20000

25000

Cheetah-Dir

'Fast' Policy Ablation The presence of the 'fast' policy, which learns from recent data, enables the agent to better adapt to changes in non-stationary MDPs, as shown in Figure 4(b). The curve labeled 'without fast policy' is identical to the baseline LILAC, as introduced in Section 3.1. While LILAC can generalize across tasks, it struggles to efficiently learn the optimal policy in the second task due to the interference problem. In contrast, BFSF addresses this issue, learning the second task at nearly the same speed as the first task, effectively overcoming interference. This trend is also observed in ongoing tasks, where interference does not negatively impact the performance of BFSF.

Dual-Reset Ablation The dual-reset mechanism ensures the effectiveness of the 'fast' policy. As shown in Figure 4(b), without the dual-reset, the learning curve exhibits lower performance due to a suboptimal 'fast' policy.

Relabeling Ablation As seen in Figure 4(b), relabeling significantly improves the learning efficiency of the 'fast' policy. With relabeling, the 'fast' policy can continuously improve its performance, even on later tasks. In contrast, BFSF without relabeling, as shown in Figure 5, struggles to achieve similar improvement in later tasks.

Bayesian Inference Ablation As introduced in Section 2.1, Bayesian inference provides a suitable estimate of the expected return for both the 'fast' and 'slow' policies. Without it, the estimation becomes less effective, and proper hyperparameter tuning may be required. As illustrated in Figure 4(b), the performance and resistance to interference deteriorate in the absence of Bayesian inference.

4 Related works

Non-Stationary MDPs Research on non-stationary MDPs primarily focuses on the challenge of recognizing potential tasks, as understanding the task transforms the non-stationary MDP into a fixed MDP. LILAC [31] first employs latent variable models to learn environment representations based on

Cheetah-Dir

Cheetah-Dir

2000

1500

1500

1000

1000

500

500

BFSF

0

fast policy

slowpolicy

0

-500

-500

-1000

withoutrelabeling

-1500

-1000

slow policy

fast policy

-2000

-1500

0

500

1000

1500

2000

2500

0

500

1000

1500

2000

2500

current and past experiences, drawing inspiration from online learning and probabilistic inference. Subsequent works identified shortcomings in LILAC and proposed solutions. For instance, FANS-RL [10] models non-stationarity in terms of individual latent change factors and causal graphs. ITER [14] highlights the impact of non-stationarity on latent representations, a form of interference similar to the one discussed in our work, leading to the proposal of Iterated Relearning (ITER). Additionally, several theoretical works have also focused on non-stationary MDPs [1, 11, 2, 7]. However, none of these prior works simultaneously address both cross-task generalization and the interference problem.

Besides, continual RL shares similarities with non-stationary MDPs but focuses on different challenges, particularly catastrophic forgetting [25]. Although non-stationary MDPs and continual RL are often treated as distinct problems, their focus differs, primarily due to the assumption that task boundaries are known to agents in continual RL. As a result, research in continual RL focuses on designing submodules within the overall algorithm [30], such as replay buffers [6], network architecture [23], representation [22], or optimization strategies [21], rather than addressing the non-stationarity itself.

Meta-RL Meta-RL aims to enable agents to adapt more quickly to new tasks by leveraging prior experience from multiple tasks. It bears strong resemblance to non-stationary MDPs, but assumes that agents have full access to all tasks, thus ignoring interference effects. RL 2 [9] learns an agent's learning algorithm, enabling it to adapt quickly to new tasks by adjusting its internal update rule based on prior experience. MAML [12], a gradient-based meta-RL algorithm, seeks a set of model parameters that can be quickly adapted to new tasks with minimal gradient updates. In contrast, context-based meta-RL methods, such as PEARL [24] and VariBAD [37], leverage contextual information to enable more efficient adaptation. While traditional meta-RL assumes access to all tasks during training, recent research has explored meta-learning in the continual task setting [4, 5], which is closely related to our work on non-stationary MDPs.

Inference-related Works The issue of interference has been widely explored in related areas. In multi-task RL, task interference has been observed, and specialized network architectures have been proposed to mitigate this challenge [17, 8]. However, the source of interference in these works differs from that in non-stationary MDPs, where interference arises from unknown, streaming tasks. In continual RL, which focuses on maintaining high performance across incremental tasks, interference is also recognized and investigated at the representation level [18].

5 Conclusion

Non-stationarity poses a significant challenge when deploying RL agents in real-world environments. In addition to cross-task generalization through context-based algorithms, a challenge that has been thoroughly explored in previous works, we have identified that interference can severely hinder performance, especially when tasks conflict with each other. To address this issue, we introduce the Bayesian Fast-Slow Framework (BFSF), which incorporates a 'fast' policy that learns from recent history to prevent interference from previous tasks, and a 'slow' policy that maintains strong cross-task generalization. The use of Bayesian estimation ensures an effective and unbiased selection between the fast and slow policies, enhancing the framework's adaptability and robustness. We also introduce a dual-reset mechanism and data relabeling to further enhance efficiency. Experimental

results demonstrate BFSF's effectiveness in resisting interference and show that it outperforms baseline methods in various non-stationary environments.

Although BFSF improves adaptability and efficiency in non-stationary MDPs, the current experiments are limited to a small set of environments. Other types of non-stationarity, such as blurred boundaries or stochastic non-stationary MDPs, have not yet been tested. Additionally, more realistic scenarios are needed to assess its applicability in real-world situations. In future work, we aim to develop more realistic benchmarks that align closely with real-world applications of non-stationary MDPs, while further testing BFSF and exploring new challenges.

6 Acknowledgment

This work was supported by AF Office of Scientific Research (AFOSR) under grant FA9550-25-10318 and with the generous support of the Amazon Research Award program.

References

Alekh Agarwal, Sham Kakade, Mikael Henaff, and Wen Sun. Pc-pg: policy cover directed exploration for provable policy gradient learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Reda Alami, Mohammed Mahfoud, and Eric Moulines. Restarted bayesian online change-point detection for non-stationary markov decision processes. In Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup, editors, Proceedings of The 2nd Conference on Lifelong Learning Agents , volume 232 of Proceedings of Machine Learning Research , pages 715-744. PMLR, 22-25 Aug 2023.

Andrew Ang and Allan Timmermann. Regime changes and financial markets. Annual Review of Financial Economics , 4(Volume 4, 2012):313-337, 2012.

Glen Berseth, Zhiwei Zhang, Grace Zhang, Chelsea Finn, and Sergey Levine. CoMPS: Continual meta policy search. In Deep RL Workshop NeurIPS 2021 , 2021.

Zhenshan Bing, David Lerch, Kai Huang, and Alois Knoll. Meta-reinforcement learning in non-stationary and dynamic environments. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3476-3491, 2023.

Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. ArXiv , abs/2203.03798, 2021.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for nonstationary Markov decision processes: The blessing of (More) optimism. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1843-1854. PMLR, 13-18 Jul 2020.

Chuntao Ding, Zhichao Lu, Shangguang Wang, Ran Cheng, and Vishnu Naresh Boddeti. Mitigating task interference in multi-task learning via explicit task routing with non-learnable primitives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7756-7765, June 2023.

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and P. Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learning. ArXiv , abs/1611.02779, 2016.

Fan Feng, Biwei Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for nonstationary reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 31957-31971. Curran Associates, Inc., 2022.

Songtao Feng, Ming Yin, Ruiquan Huang, Yu-Xiang Wang, Jing Yang, and Yingbin Liang. Non-stationary reinforcement learning under general function approximation. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, pages 1126-1135. JMLR.org, 2017.

Weiyu Guo and Mark E. Wohar. Identifying regime changes in market volatility. Journal of Financial Research , 29(1):79-93, 2006.

Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations , 2021.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: model-based policy optimization . Curran Associates Inc., Red Hook, NY, USA, 2019.

Leslie P Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Technical report, USA, 1996.

Menelaos Kanakis, David Bruggemann, Suman Saha, Stamatios Georgoulis, Anton Obukhov, and Luc Van Gool. Reparameterizing convolutions for incremental multi-task learning without task interference. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16 , pages 689-707. Springer, 2020.

Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J. Roberts. Same state, different task: Continual reinforcement learning without interference. 36:7143-7151, Jun. 2022.

Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. J. Artif. Intell. Res. , 75:1401-1476, 2020.

Vincent Liu, Han Wang, Ruo Yu Tao, Khurram Javed, Adam White, and Martha White. Measuring and mitigating interference in reinforcement learning. In Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup, editors, Proceedings of The 2nd Conference on Lifelong Learning Agents , volume 232 of Proceedings of Machine Learning Research , pages 781-795. PMLR, 22-25 Aug 2023.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, pages 6470-6479, Red Hook, NY, USA, 2017. Curran Associates Inc.

Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In International Conference on Learning Representations , 2022.

Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 , pages 72-88, Cham, 2018. Springer International Publishing.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient offpolicy meta-reinforcement learning via probabilistic context variables. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 5331-5340. PMLR, 09-15 Jun 2019.

ANTHONY ROBINS. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science , 7(2):123-146, 1995.

Amila Thibbotuwawa, Grzegorz Bocewicz, Grzegorz Radzki, Peter Nielsen, and Zbigniew Banaszak. Uav mission planning resistant to weather uncertainty. Sensors , 20(2), 2020.

Amila Thibbotuwawa, Grzegorz Bocewicz, Banaszak Zbigniew, and Peter Nielsen. A solution approach for uav fleet mission planning in changing weather conditions. Applied Sciences , 9(19), 2019.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, 2012.

Michael Wan, Jian Peng, and Tanmay Gangwani. Hindsight foresight relabeling for metareinforcement learning. In International Conference on Learning Representations , 2022.

Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(8):5362-5383, 2024.

Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst continual structured non-stationarity. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11393-11403. PMLR, 18-24 Jul 2021.

Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based reinforcement learning with theoretical guarantees. CoRR , abs/1807.03858, 2018.

Tengye Xu, Zihao Li, and Qinyuan Ren. Meta-reinforcement learning robust to distributional shift via performing lifelong in-context learning. In Proceedings of the 41st International Conference on Machine Learning , ICML'24. JMLR.org, 2024.

Zhuoran Yang, Yuchen Xie, and Zhaoran Wang. A theoretical analysis of deep q-learning, 2020.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL) , 2019.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization, 2020.

Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: variational bayes-adaptive deep rl via meta-learning. J. Mach. Learn. Res. , 22(1), January 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The introduction mentions all the parts in the text.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are included in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions and proof are in the main text and Appendix A.1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The implementation details can be found in the code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is in the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The implementation details can be found in the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The error bars are provided in the Table 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The implementation details can be found in the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research satisfies the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creator of the code is mentioned in the code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Technical Appendices and Supplementary Material

A About Theorem 2.1

A.1 Proof of Theorem 2.1

Proof. First, since Bayesian estimation provides an unbiased estimate, the suboptimality of BFSF is bounded by the better performance between the fast and slow policies.

We analyze the suboptimalimality of the 'fast' policy | η M ( π ∗ M ) -η M ( π ∗ M ′ ) | , in the following proof. Let MDP M have the dynamic function p and reward function r . Similarly, let MDP M ′ have the dynamic function p ′ and reward function r ′ . We denote M p,r as the MDP with dynamics function p and reward function r . In Equation 5, we decompose | η M ( π ∗ M ) -η M ( π ∗ M ′ ) | and use Theorem A.1, Lemma A.2, A.3 to complete the proof.

Theorem A.1. (Relabeling gap 1)Let M 1 , M 2 be two finite-horizon MDPs with the same reward function r . Then the distance of η M 1 ( π ∗ M 1 ) and η M 2 ( π ∗ M 2 ) is bounded by

where ϵ h = [ D ℓ 1 ( p M 1 , p M 2 )( r max + V max ) + r diff ]( H -h ) , ∀ h ∈ [ H ] , s ∈ S .

Proof. Since π ∗ M 1 is the optimal policy of MDP M 1 and π ∗ M 2 is the optimal policy of MDP M 2 , η M 1 ( π ∗ M 1 ) = V ∗ M 1 , η M 2 ( π ∗ M 2 ) = V ∗ M 2 .

We begin our proof from the final horizon, h = H , and use the closeness at horizon h to establish the closeness at horizon h -1 .

For the final horizon h = H , we have V ∗ M,H ( s ) = 0 ∀ M since it is the terminal state. Therefore, || V ∗ M 1 ,H ( s ) -V ∗ M 2 ,H || ∞ ≤ 0 = ϵ H . Suppose

We need to prove

It is equivalent to prove

For simplicity, but without loss of generality, we will prove the inequality on the right-hand side of the above equation.

Lemma A.2. Let M 1 , M 2 be two MDPs with the same dynamics function, but different reward functions r 1 , r 2 . Define U r 1 ,r 2 ( π ) = E ( s,a ) ∼ ρ π M 1 [ r 2 ( s, a ) -r 1 ( s, a )] , which characterizes how erroneous the model is along trajectories induced by π . Then

Proof. We know that ˜ M and ˆ M shares the same transition dynamics p , but different reward functions ˜ r ( s, a ) = ˆ r ( s, a ) -λu ( s, a ) . Therefore,

}

Lemma A.3. (Telescoping lemma) [36, 32]. Let M 1 and M 2 be two MDPs with the same reward r ( s, a ) , but different dynamics p M 1 and p M 2 respectively. Let

Then,

For each s ∈ S , a ∈ A , a ℓ 1 -based bound of | G π M 2 ( s, a ) | is

A.2 Bound Considering the Optimization Error

Theorem 2.1 accounts for the error introduced by the relabeling process. To maintain consistency with prior work, we explicitly incorporate optimization suboptimality, following the approach in [34], by considering the policy obtained after K policy-update iterations π K M ′ , rather than the idealized optimal policy π ∗ M ′ .

Total Suboptimality

where each term explicitly captures different sources of error:

The first 3 terms (matching our original Theorem 3.1) represent the suboptimality caused by the relabeling process , quantifying the error introduced due to differences in dynamics and reward functions between the original MDP M and the relabeled MDP M ′ .

The newly introduced 4th and 5th terms explicitly quantify the optimization suboptimality , representing errors arising from finite-sample approximations and iterative optimization.

B Experiments

B.1 Implementation Details

The experiments are repeated three times, with the mean and standard deviation shown in the curves and table.

The common hyperparameters are consistent with the original PEARL implementation. Additionally, the context consists of 200 episodes, and the relabeling percentage is set to 50% (i.e., half of the used batch is relabeled). The window of recent data w = 100 episodes. The reset frequency ν = 50 episodes. The discount factor γ = 0 . 99 .

B.2 Full Learning Curves

A complete comparison of the learning curves for all four baselines is provided in Figure 6.

B.3 Sensitive Studies and Other Experiments

Sensitive study on the relabeling percentage We performed an sensitive study on the relabeling percentage in the Table 2.

Cheetah-Dir

Cheetah-Vel

Ant-Dir

2000

1250

BFSF

1500

-100

1000

LILAC (only context)

ITER

1000

200

750

CEMRL

500

-300

500

CoMPs

0

BFSF

400

BFSF

250

500

LILAC (only context)

LILAC (only context)

0

-1000

CEMRL

ITER

500

ITER

CEMRL

250

1500

CoMPs

-600

CoMPs

500

I

0

5000

10000

15000

20000

25000

0

2000

4000

6000

8000

10000

12000

0

2000

4000

6000

8000

10000

12000

Walker-Rand-Params

Metaworld Reach

1000

BFSF

2000

200

LILAC (only context)

800

ITER

400

600

CEMRL

CoMPs

1500

600

1000

BFSF

400

BFSF

800

LILAC (only context)

500

LILAC(only context)

ITER

200

ITER

-1000

CEMRL

CoMPs

0

i

CEMRL

CoMPs

0

2000

4000

6000

8000

10000

12000

0

2000

4000

6000

8000

10000

12000

0

2000

4000

6000

8000

10000

12000

Sensitive study on the window of recent data w and the reset frequency ν is shown in Table 3. For reference, the average return of the baseline (LILAC) is 757 . 8 . The results indicate that performance of our method is relatively insensitive to the choice of window size w and reset frequency ν .

The experiment in gradually changing environments To validate our method in dynamically changing environments, we conducted supplementary experiments in Table 4 in a 180-task gradually changing Ant-Goal environment, with goals evenly distributed around a circle.

The experiment in stochastically changing environments As to settings with highly stochastic task boundaries, we conduct experiments on a Cheetah-Dir environment in Table 5. This environment consists of two tasks: moving forward and moving backward. A boundary period (one-third of the task length) exists during task switching, with each task having a probability of 50% .

C Algorithm Details

C.1 Data Relabeling

To elaborate, relabeling is accomplished by a learned dynamics and reward model s ′ , r = f ( s, a ) [15], which estimates the next state s ′ and reward r after taking action a in the state s . We can

substitute the original next state and reward in the experience replay, even from different tasks, with those predicted by the model, represented as s ′ relabel , r relabel = f ( s, a ) .

Relabeling is widely used in the RL community [29, 33]. The underlying principle is to maximize data reuse for sample efficiency. In our work, given that the environment evolves over time, we leverage a context-based dynamics model f ( s, a, c ) , which provides different dynamics depending on context c .

As discussed in [29, 33] and confirmed by our ablation studies, data relabeling substantially enhances sample efficiency. In our work, we adopt data relabeling to mitigate the issue of performance degradation in a non-stationary environment, because the increased amount of relabeled data allow the 'fast' policy to better and faster adapt to these tasks, which is verified by our ablation results.

C.2 Motivation for the Dual-Reset Mechanism

The dual-reset mechanism was introduced upon observing that RL algorithms tend to encounter performance degradation when alternating between tasks, a trend corroborated by prior research [4]. This mechanism effectively addresses the issue. We present the phase performance for SAC in the alternating Cheetah-Dir environment (where the two tasks are moving forward and backward) in Table 6.

C.3 Bayesian Fast-Slow Framework

We only utilize data within a recent window, which keeps the effective sample size small, and we set the prior value to a dynamically updated upper bound to encourage exploration. As a result, if a policy has not been sufficiently selected, the prior strongly influences the Bayesian estimate, leading to a large posterior value that naturally encourages exploration of that policy.

D Posterior Calculation of Normal Distributions

Assume ϕ is known.

E Visualization of Choosing Slow/Fast Policies

We presented visualization in left sub-figure of Figure 5 in our paper (attached here in Figure 7). In the left sub-figure, the 'fast' policy predominates in the selection during the initial phases, while the 'slow' policy shows competitive performance as the amount of accumulated data from different tasks increases. Additionally, in the latter part of a single phase, the 'fast' policy surpasses the 'slow' policy after learning from relabeled recent data.

Cheetah-Dir

Cheetah-Dir

2000

1500

1500

1000

1000

500

500

BFSF

0

fastpolicy

slowpolicy

0

-500

500

-1000

withoutrelabeling

-1500

-1000

slowpolicy

fast policy

-2000

-1500

0

500

1000

1500

2000

2500

500

1000

1500

2000

2500

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an RL framework (BFSF) for continual non-stationary reinforcement learning. The agent maintains two policies: a “slow” meta-RL policy (implemented with PEARL) that exploits long-term experience for cross-task generalisation, and a “fast” policy trained only on recent data to react quickly to distribution shifts and reduce negative transfer (termed “interference”). The current policy is chosen each epoch via Bayesian model selection on a sliding window of recent returns. The fast policy further uses (i) a dual-reset mechanism that periodically re-initialises one of two identical networks to avoid performance collapse, and (ii) context-conditioned data relabeling to augment its recent replay buffer. Theoretical analysis gives a sub-optimality bound that decomposes errors from relabelling and optimisation. Experiments on six tasks (from MuJoCo and Meta-World benchmarks) plus two “infinite-world” variants show improved average returns over baselines such as LILAC, ITER, CEMRL and CoMPs.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n### Quality\n- Really solid engineering effort. BFSF combines a principled fast/slow decomposition, Bayesian model selection, dual-reset, and experience relabeling using learned world model. Each component is at least intuitively motivated and evaluated in ablations.\n- It outperforms strong baselines across eight non-stationary tasks and maintains higher returns through task switches.\n- The empirical investigation of “interference” is interesting, especially with the visualisations in Figure 3.\n- The hyperparameter sensitivity ablations are also very commendable.\n\n### Clarity\n- The High-level intuition of BFSF is easy to grasp and the algorithms are summarised in pseudo-code.\n- Decent related-work section clearly positions BFSF with respect to meta-RL and continual RL literature.\n\n### Significance\n- Addresses an important pain-point in deploying meta-RL agents in streaming settings where sharp task boundaries are unknown.\n- The fast/slow idea could inspire follow-up work that plugs in different meta-learners or fast learners, and the infinite-world experiments push beyond the usual episodic benchmarks.\n\n\n### Originality\nIt provides a novel combination of known components. While fast-adaptation, Bayesian policy selection, and relabeling are known ingredients, their integration as a coherent framework for mitigating cross-task interference in non-stationary MDPs is new. The dual-reset mechanism is also a simple but clever trick that may interest practitioners suffering from representation collapse (although it is unclear how sensitive it is to its hyperparameter).\n\n\n# Weaknesses\n\n### Quality\n- Theoretical section is unfortunately weak. Theorem 2.1 appears vacuous if I understand it correctly. The bound is ≥ V_max when M≠M′, with undefined symbols p_M, η_M, etc. and the promised proof sketch (line 145) is never given.\n- There is no discounting in the MDP problem definition. Is there no discounting in the experiments? In general, critical hyper-parameters like discount factor, window size w, reset frequency ν, RL algorithm for line 10 in Alg. 2) and number of seeds (only 3) are either buried in the appendix or absent, undermining statistical reliability.\n- PEARL is used as the slow policy but is not reported as a standalone baseline. Likewise there is no “fast learner + relabeling only” baseline (since that's not new to this work), so improvements may conflate benefits from the proposed approach.\n- It looks like the authors did not also do data relabeling for all the baselines with fast policies (e.g. ITER). This is a major concern if that is indeed the case.\n- Unlike the authors claim on line 471, Table 3 shows (I assume average returns) that the approach is highly sensitive to the window size w and reset frequency v with a very clear trend (higher w and v lead to poorer performance).\n- For Figure 4.b, the authors say \"without fast policy\" is identical to LILAC. Shouldn't it be PEARL since that's what the authors claim to be using for the slow policy? There's also too much noise in the figure. E.g. It is hard to justify the \"Dual-Reset Ablation\" interpretation given by the authors given the noise. An easier domain like Cheetah-vel or Ant-Goal (from looking at the results in Fig 2) would have been better for this ablation.\n- The Bayesian inference ablation is inconclusive. Figure 4(b) shows almost identical returns on average with and without Bayesian selection, casting doubt on the necessity of the Bayesian component.\n\n\n### Clarity\n- Key quantities like Posterior(...), p_M, η_M, are undefined in the main text. \n- Several typos hinder readability (lines 114, 122, 228).\n- Figures use raster graphics and colour palettes that are hard to distinguish for colour-blind readers; \n- Regular drops in BFSF curves in-between task switches are unexplained (e.g. in Fig 2). Also why does BFSF have no performance drop at 6000 and 10000 in Metaworld Reach (Fig 2)?\n- Algorithmic flow between Alg. 1, Alg. 2, and environment interaction is still ambiguous: when exactly is the environment reset, and how are online interactions interleaved with dual-reset?\n\n### Significance and Originality\n- With only three seeds it is unclear if the results are significant.\n- The main novelty in this work seems to be from the specific Bayesian inference and dual-reset ideas. Unfortunately, Figure 4 does not support the claim that these are useful significant components of the approach.\n\nOverall, the paper tackles an important practical issue and shows promising empirical gains. However, clarity, baseline coverage, and theoretical rigor need strengthening before the contribution can be considered fully conclusive.\n\nQUESTIONS:\nPlease refer to the weaknesses above. Mainly,\n- Is data relabeling also used for all the baselines with fast policies? If not, please do so. \n- Report PEARL alone (the \"slow\" learner from prior work). Also report the \"fast\" learner used in Algorithm 2 step 10 with experience relabeling. The absence of these two baselines are of major concern to me.\n- Increase seeds to at least 10 and include confidence intervals. Otherwise the claimed gains may not be significant.\n- Please define all symbols in Theorem 2.1 and provide a proof sketch. Under what realistic conditions can the bound be < V_max? If the bound is always loose, consider reframing the theory section.\n- Explain algorithmic timing. Describe precisely how training, dual-reset, and environment interaction interleave.\n- In Fig. 2, what causes the periodic drops in BFSF between switches, and why are the drops absent at 6 k / 10 k steps on Meta-World Reach?\n-  Line 497 says \"As we cannot include images in this rebuttal\" in the appendix of the submission. Please include said images.\n\nLIMITATIONS:\nThere is no broader impacts statement.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Bayesian Fast-Slow Framework (BFSF), a novel approach to address the challenge of interference in non-stationary reinforcement learning environments. Unlike previous works that focus solely on cross-task generalization, BFSF dynamically switches between a fast policy trained on recent data for rapid adaptation and a slow policy trained via meta-RL to leverage past experience. A Bayesian inference mechanism determines the optimal policy to use at each point in time.\n\nSTRENGTHS AND WEAKNESSES:\n1. Strengths\n- The paper introduces a well-motivated and novel framework (BFSF) specifically designed to tackle interference in non-stationary RL environments.\n- The use of Bayesian inference for dynamically selecting between fast and slow policies is principled and adaptive, allowing the agent to respond to changing task distributions efficiently.\n- The incorporation of the dual-reset mechanism and context-aware data relabeling significantly enhances the fast policy’s adaptability and sample efficiency.\n\n2. Weaknesses\n- The framework deterministically selects the policy (fast or slow) with the higher expected return, without incorporating stochastic exploration, which may lead to overcommitting to suboptimal policies in early stages.\n- If one policy (e.g., fast) dominates selection early on, the other (e.g., slow) may suffer from under-training due to limited data collection, which is not fully analyzed in the paper.\n- The performance of the fast policy partially depends on the accuracy of the learned context-based dynamics model, but the paper does not deeply analyze its limitations or robustness.\n- The framework introduces several new hyperparameters (e.g., reset frequency, window size for Bayesian updates), but sensitivity to these is not extensively studied or theoretically justified.\n\nQUESTIONS:\n- Could the Bayesian selection mechanism be enhanced with exploration to avoid early commitment bias? The current policy selection is fully deterministic based on posterior mean estimates. This may lead to under-utilization of one policy (e.g., the slow policy) if the other dominates early on. Have you considered incorporating stochastic policy selection mechanisms to encourage balanced exploration of both fast and slow policies, especially in the early phases?\n- How robust is the fast policy’s performance to inaccuracies in the context-based dynamics model used for relabeling? Since relabeling heavily depends on the quality of the learned model, it would be helpful to understand how model errors affect fast policy training. Have you evaluated the impact of model approximation errors on policy performance? \n- What happens when fast policy dominates data collection for extended periods? Although slow policy is updated in each epoch, it depends on data collected through interaction through the past policy. It would be helpful to clarify how such imbalance is mitigated in practice, and whether you considered mechanisms for guaranteeing minimum data exposure for each policy.\n- In Figure 4(b), the ablation results suggest that the fast policy and relabeling have the most significant impact on overall performance, whereas the removal of Bayesian inference seems to cause only a minor degradation. Could you clarify the relative importance of the Bayesian selection mechanism in the overall framework? Specifically, how critical is Bayesian inference to the stability and adaptability of BFSF? If the system performs almost as well without Bayesian inference and slow policy, what unique benefits does it provide in more challenging or uncertain settings?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Bayesian Fast-Slow Framework (BFSF) to address the problem of interference in non-stationary reinforcement learning environments. The key issue is that when RL agents encounter changing tasks over time, learning from previous tasks can negatively interfere with performance on new tasks, especially when tasks are contradictory.\n\nBFSF combines two complementary policies: a \"fast\" policy that learns quickly from recent data to mitigate interference, and a \"slow\" policy that uses meta-reinforcement learning to capture knowledge from all previous tasks for cross-task generalization. A Bayesian estimation mechanism dynamically selects between these policies based on their expected performance using recent return history.\n\nThe framework also incorporates a dual-reset mechanism to prevent performance degradation in neural networks and a data relabeling technique to enhance learning efficiency by augmenting recent data with relabeled historical experiences. Experimental results on MuJoCo and Meta-World environments demonstrate that BFSF effectively mitigates interference while maintaining good cross-task generalization, outperforming existing baseline methods in non-stationary settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) Addresses the understudied but critical issue of cross-task interference in non-stationary MDPs\n2) The dual-policy approach elegantly balances fast adaptation (recent data) with cross-task generalization (historical data), with principled Bayesian selection between them.\n3) Thorough experimental validation across multiple environments\n\nWeaknesses:\n1) Experiments are restricted to a small set of MuJoCo and Meta-World environments with fixed task switching intervals\n2) The framework requires maintaining dual networks, meta-RL training, Bayesian inference, and data relabeling, which likely increases computational complexity compared to simpler baselines.\n3) Relies on assumptions like normal distribution for Bayesian inference and requires a learned dynamics model for data relabeling\n\nQUESTIONS:\n1) How does the computational overhead of maintaining dual policies, meta-RL training, and Bayesian inference scale with the number of tasks and complexity of environments?\n\n2) How sensitive is the framework to key hyperparameters like the window size (w=100), reset frequency (ν=50), and relabeling percentage (50%)?\n\n3) Since the method assumes unknown task boundaries, how would it perform in scenarios with gradual task transitions or when task changes are very subtle?\n\n4) Would BFSF be effective in other domains like discrete action spaces, vision-based tasks, or environments with fundamentally different dynamics than the tested locomotion tasks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}