A Bayesian Fast-Slow Framework to Mitigate Interference in Non-Stationary Reinforcement Learning
Authors: Yihuan Mao
Chongjie Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Non-stationary MDPs
Continual RL
Primary area: reinforcement_learning

Abstract: Given the ever-changing nature of the world and its inhabitants, agents must possess the ability to adapt and evolve over time. Recent research in Given the ever-changing nature of the world and its inhabitants, agents must possess the ability to adapt and evolve over time. Recent research in non-stationary MDPs has focused on addressing this challenge, providing algorithms inspired by task inference techniques. However, these methods ignore the detrimental effects of interference, which particularly harm performance in contradictory tasks, leading to low efficiency in some environments. To address this issue, we propose a Bayesian Fast-Slow Framework (BFSF) that tackles both cross-task generalization and resistance to cross-task interference. Our framework consists of two components: a 'fast' policy, learned from recent data, and a 'slow' policy, learned through meta-reinforcement learning (meta-RL) using data from all previous tasks. A Bayesian estimation mechanism determines the current choice of 'fast' or 'slow' policy, balancing exploration and exploitation. Additionally, in the 'fast' policy, we introduce a dual-reset mechanism and a data relabeling technique to further accelerate convergence when encountering new tasks. Experiments demonstrate that our algorithm effectively mitigates interference and outperforms baseline approaches.

A Bayesian Fast-Slow Framework to Mitigate Interference in Non-Stationary Reinforcement
Learning
Yihuan Mao
Institute for Interdisciplinary Information Sciences Tsinghua University
maoyh1024@gmail.coo com
Chongjie Zhang
Department of Computer Science & Engineering Washington University in St. Louis
chongjie@rustl.eed
Abstract
Given the ever-changing nature of the world and its inhabitants, agents must possess the ability adapt and evolve over time. Recent research in Given the ever-changing nature of the world and its inhabitants, agents must possess the ability adapt and evolve over time. Recent research in non-stationary MDPs has focused on addressing this challenge. providing algorithms inspired by task inference techniques. However, these methods ignore the detrimental effects interference, which particularly harm performance in contradictory tasks, lead- ing to low efficiency in some environments. To address this issue, we propose a Bayesian Fast-Slow Framework (BFSF) that tackles both cross-task generalization and resistance to cross-task interference. Our framework consists of two com- ponents: 'fast' policy, learned from recent data, and a 'slow' policy, learned through meta-reiinforcement learning (meta-RL) using data from all previous tasks. A Bayesian estimation mechanism determines the current choice of 'fast' or 'slow' policy, balancing exploration and exploitation. Additionally, in the 'fast' policy, we introduce dual-reset mechanism and data relabeling technique to further accel- crate convergence when encountering new tasks. Experiments demonstrate that our algorithm effectively mitigates interference and outperforms baseline approaches. Code is available at ltthhttt///// h/tp:h/.///ttub.... com cwtese/deeeeeeeeeee BFSF
Reinforcement Learning (RL) in non-stationary environments has long attracted significant attention, leading to the emergence of research areas such as continual RL 30 19 and non-stationary MDPs 7224 31 These areas approach challenges from different perspectives. For example, catastrophic forgetting, well-known issue in continual RL, arises due to limited memory storage. contrast, re- search on non-stationary MDPs focuses on understanding the underlying dynamics of the environment facilitate better adaptation across varying contexts.
One critical challenge in non-stationary environments is interference, where the learning process is negatively impacted by experiences from previous tasks, as illustrated in Figure [ This interference arises primarily because task boundaries are either unknown or absent in the streaming task setting. Many real-world problems exhibit such non-stationarity and suffer from interference. For instance, a UAV must adapt its behavior under varying weather conditions 27 26). Similarly, the evolving regime of the stock market can be viewed time-varying environment, where an effective strategy
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Interference of lask I and task
Learning process
Task
Task 2
Task
Time
-000
Unknown changepoint
The diagram of interference stationary MDPs.
(b) The learning curve how BFSF al- leviates the problem interference.
Figure I: Figure ((a) illustrates the interference problem non-stationary MDPs. The agent learns to perform well the current task, but the changepoint is unknown. As result, when the agent begins learning new task (e.g., task 2), experience from previous tasks (e-g., task 1) can hinder performance. This interference phenomenon also occurs across consecutive tasks. To address this, propose BFSF, which incorporates 'fast' policy that learns from recent data to mitigate interference, alongside 'slow' policy using meta-RL to Iearn context-based policy from all previous tasks. Figure [b
BFSF's ability to resist interference in the Cheetah-Dir task, which involves two contradictory tasks: moving forward and backward. In the second phase, the learning shows less disruption compared to the "slow' policy only. The highest performance throughout the non-stationary MDP process is close to the upper bound, which represents the scenario where the two tasks are trained separately.
must stay adaptive while leveraging past experience [3 the urgent need for framework that can operate efficiently in non-stationary MDPs with interference.
These real-world scenarios underscore
Despite its significant negative impact on performance, the issue of interference has been largely overlooked in the literature. Some existing works (18) address interference from the perspective of representation, while others (20) discuss the inverse interference of current tasks on previously learned tasks, phenomenon referred to catastrophic forgetting in continual RL. This lack of attention to cross-task interference concerning, as it can severely degrade performance in successive tasks. In this work, specifically analyze the effects of interference and propose effective strategies to mitigate its detrimental impacts.
To address the interference problem, we introduce the Bayesian Fast-Slow Framework (BFSF). This framework dynamically selects between two learning strategies: 'fast' policy, which quickly adapts to new tasks using recent data, and a 'slow' policy, which is learned through meta-RL and captures knowledge from historical data. Unlike previous approaches that focus solely on latter, often leading to severe interference in the face of sudden task changes, framework not only mitigates interference but also preserves the advantages of meta-RL. Bayesian estimation mechanism is employed in each epoch to decide which policy, fast slow, is more promising based recent history. Only the most recent returns are used to update the Bayesian estimates, ensuring that outdated data does not influence the decision.
We also identify that the 'fast' policy can sometimes underperform. One reason is that neural networks often experience performance degradation when trained data from different distributions, a common issue non-stationar environments. To address this, we introduce dual-reset mechanism that periodically reinitializes one of the dual networks to prevent degradation, while alternating between the networks to ensure stable performance. Another challenge is that learning from scratch typically requires extensive online interaction. To mitigate this, propose data relabeling, utilizing historical data from previous tasks to enhance learning efficiency and improve performance in few-shot settings when facing new tasks.
In summary, our contributions are twofold: i) We analyze the interference problem in non-stationary MDPs. ii) We propose the Bayesian Fast-Slow Framework (BFSF), which combines a fast policy, enhanced by a dual-reset mechanism and data relabeling, to efficiently handle recent tasks, and a slow policy for cross-task generalization. Through Bayesian estimation, we effectively address interference and improve overall performance. Experimental results demonstrate BFSF's superiority resisting interference and outperforming baseline methods across various non-stationary environments.

---

Preliminaries
Notations and problem definition A Markov Decision Process (MDP) is defined as M (S, A, P. R), where and A represent the state and action spaces, respectively. The transition function of the environment is denoted P, and R represents the reward function. The expected return of policy is given by E+= Re]. In non-stationary MDPs, the underlying MDP evolves time. These changes occur sequentially, such as M1, M2,-. ".., gradually over time. The objective is to maximize the expected return, E E
z= Ra] with the evolving dynamics of the MDP
Context-based policy In standard MDP, the policy function defined as (aas), which determines the probability of selecting action given state s. In non-stationary settings, context-based policy introduced to adapt to varying environments. This policy denoted as T(a 8, c), where c the context, a set of trajectories related to the current environment. A trajectory consists of the sequence of states, actions, and rewards at each timestep, expressed as se; dgy To, 61, d1, "1, 82, d22 f2,,.... In implementation, the context is collected from recent interactions, reflecting the underlying MDP.
The technique of learning the context-basee policy has been extensively studied in the field of meta- reinforcement learning (meta-R)) RL) 24 37 However, meta-RL. differs from the non-stationary MDPs setting in this work. In meta-RL., task information explicitly available, and there is no continuous adaptation process. Context-basse meta-RL. methods typically map the contextual information, often represented as transition data, into a latent space Z. By assigning latent variable E to represent the task, this approach effectively frames the problem as partially observable MDP (POMDP) [16], where z constitutes the unobserved portion of the state. In meta RL, PEARL learns the posterior distribution (==(c), which means the posterior latent variable distribution given the context and uses posterior sampling to sample a to integrate these latent variables with oof-po RL algorithms.
Algorithm Bayesian fast-slow framework (BFSF)
(1) 1: Input: 'fast' policy л fast (including the dual policies "H fastt क fast). 'slow' policy Nstows the number of epochs the window of recent data
2: Initialize return list ] Jie...E and choice list (choice, )iE1...
3: for epoch e 1... do
4: # Bayesian inference of the expected return
5: Reast = Posterior( (Rsele-w.e]] choice, = ffst()
6: Rutow = Posterior( (Re lchoice; = slow))
7:
# Online interaction
8:
if Rost > Retow then
9:
choice, := fast
10:
Collect data using fast else
11:
12:
choicc, := slow
13:
Collect data using Talow
14:
end if
# Training
16:
Update fast by Algorithm 2
Update Tslow by the meta RL algorithm
18: end for
2 Bayesian Fast-Slow Framework (BFSF)
The Bayesian Fast-Slow Framework (BFSF) designed mitigate interference by dynamically deploying either a 'fast' policy, which learns from recent data, or a "slow' policy, trained using meta-RL principles. The term 'fast' arises from its 'fast-adaptation ability to learn directly and efficiently from recent data. In contrast, the 'slow" policy enables cross-task understanding and generalization, which may hinder training speed, especially when the number of observed tasks limited in early phase. The decision is made based on Bayesian estimation of current expected return.

---

As illustrated in Algorithm [ during each epoch, Bayesian inference is applied to estimate the posterior expected return using the recent return history, for both the fast and slow policies. Let R; denote the return obtained in the i-th epoch, and choice; indicate whether the 'fast" or 'slow' policy was selected during that epoch. During the online interaction phase, the policy with the higher estimated posterior value is selected, aiming to generate higher-quality experience. At the end of each epoch, both the fast and slow policies are updated using their respective replay buffers. The detailed computation of the Bayesian posterior, described in lines 5 and 6 of Algorithm | is elaborated in Section 2.T] In addition to the online interaction and Bayesian estimation, the training process for the 'fast" policy is detailed in Algorithm 2 while the 'slow' policy is trained according to the context- based meta-RL algorithm PEARL [24], which is one of the first context- based methods and serves as the baseline for numerous subsequent works. The visualization of choosing the 'fast' or "slow" policy is provided in Appendix
2.1 Bayesian Inference
The detailed update rule for Bayesian posterior estimation is outlined below. For simplicity, assume that the recent returns of a given policy, Rú; R.g...., are approximately drawn from a normal distribution N(,, 1/ where & is a constant. While this assumption is commonly used, other distributional forms could also considered depending on the context. The prior distribution for the parameter j is assumed to follow μ NN 1/do). The posterior estimation of μ then follows the standard derivation below, as detailed in Appendix
(Ri Reg. } Mo, ΦD) ~ M(n, 1/0î),
Pu/10 + no R
1
where μ1
Φ + n
do no
To better interpret the result of Bayesian inference, note that the posterior mean μ1 can be decomposed:
Φ0/400 + no f11
no Φ nø Φ + no
do
No
R.
du + no
shows that the posterior mean μn is weighted average of the prior mean μi and the sample average R. As more samples collected, the weight shifts toward trusting the sample average R. Conversely, when only a few samples are available, the posterior relies more heavily on the prior μo-
2.2 'Fast' Policy Learning
The 'fast' policy, learned from recent data, is critical for mitigating interference in non-stationar MDPs. However, the standard learning paradigm often encounters challenges under these conditions. To address these issues, we propose specific structural designs that significantly enhance the efficiency and adaptability of the 'fast' policy.
Algorithm 2 Training process the 'fast' policy.
1: Input: The 'fast' policy fast (including the dual policies π (1) model M, current epoch reset frequency y
Jast fast) a contextual dynamics
2: Output: The updated fast
3: # Dual-reset mechanism
4: if' mod
0 then
5: π fast: /Jst π (2)
(1)
fast Init( fast)
6: end if
7: # Data relabeling
8: Relabel the recent data by M using the recent trajectories as the context.
9: Training process
10: Train (1)
ffzt: fast using the relabeled data

---

Dual-reset Mechanism A key challenge in continual learning is the performance degradation of neural networks when trained on successive tasks. One common observation that the learning curve for the second task often struggles to converge to an optimal point, even when task difficulty is comparable (as detailed in Appendix C.2) This phenomenon also noted and studied in ITER [14.
To address this, we propose the dual-reset mechanism, as outlined in Algorithm 2 which mitigates performance degradation by periodically reinitializing the model. However, avoid the inferior performance typically observed immediately after reinitialization, we introduce dual-model system. This ensures that during any interaction phase, even directly after initialization, fully trained model is always available for deployment.
Data Relabeling Another challenge lies in the limited recent data available for the 'fast' policy, which is necessary for rapid adaptation in non-stationary environments. This small dataset size may not support learning robust policy, especially over prolonged training periods, in contrast to the "slow' policy that can utilize the entire historical dataset. As a result, the 'fast' policy tends to perform significantly worse, as shown in Figure 5 To address this limitation, we incorporate data relabeling, which significantly enhances the amount of usable data, enabling the learning of stronger policy.
Specifically, the data relaheling process relies on maintaining a context- dynamics model. This model takes the context, state, and action input, and outputs the relabeled next state and reward. Leveraging the context-based property, it becomes possible to relabel historical data from other tasks into the context of the current task. By combining relabeled data with the original dataset, the learning efficiency of the 'fast' policy is substantially improved. Further details can be found in Appendix C.I
2.3 Theoretical Analysis
In this section, we provide theoretical analysis of the sub-optimality bound of the Bayesian Fast-Slow Framework (BFSF) and present Theorem 2.1
Theorem 2.1.
Suboptimality(BFSPP
De, (PM, PM (Tmax Vmax) + rasssl!
+ (UT + VmanDD, (p a), (s)(s,))),
where M, M' denote the original and relabeled MDPs, and pM, PM are their transition functions. H the horizon, Tmaz, Vmax are the maximum reward and value, Tdiff represents the maximum reward gap between M, M', Ur (T)) is defined as E(,,))=3, (s, a) - r(s,a)]. and Di, is the Ll distance.
The following provides a proof sketch and interpretation of Theorem 2.1 First, the suboptimality is defined as the minimum between the fast and slow policies, with the Bayesian estimation serving as an unbiased estimate. We focus on the suboptimality of the 'fast" policy, [Tw ( 77M (x (r)).. This suboptimality can then be decomposed into several components detailed in Appendix(A.) For ease of comprehension, the first term represents the gap in optimal expected return between the relabeled and original MDPs. The second and third terms arise from the performance difference of the same policy under different dynamics. A further analysis of the bound that incorporates optimization error provided in Appendix A.2
Experiments
The Bayesian Fast-Slow Framework (BFSF) is designed to address the complexities of non-stationary MDPs, specifically tackling the interference issue while ensuring generalization across experiences from different tasks. In this section, we focus on two main questions: i) How does BFSF overcome interference in non-stationary environments? ii) How does BFSF perform in real-world scenarios? iii) How do the individual components of BFSF contribute to improving performance?
To answer the first question, we provide experimental comparisons with baselines in Section 3.1 demonstrating the superiority of BFSF in mitigating interference. For the second question, we design an wnfinite-world simulation better approximate real-world conditions and evaluate the performance
5

---

CrosdavCC CV
ArsEx
750
500
201
1100
10:30
5000
25000
40.00
6000
12000
LILA tant comppO
120
Figure 2: The learning curve of BFSF and other baselines, the non-stationary MDPs based 5 MuJoCo locomotion environments and Meta- World environment, For clarity, we only display the curves for BFSF, LILAC, and ITER. Additional curves for CEMRL and CoMPs, along with implementation details, can be found in Appendix
CHEETAH- DIR
CHEETAH- VEL
ANT-DIR
ANT-GOAL
BESF
1209.0 24.0
-99.1 2.8
671.0 + 37.7
-204.9 + 9.3
LILAC
757.7 1 94.9
-1546 2.5
318.5 66.2
-426.6 4.6
ITER
813.4 +: 19.0
-102.0 2.8
-156.8
25.7
511.2 1 50.0
CEMRL
852.6 + 36.1
-196.0 5.2
289.6 + 18
-547.1 67.6
COMPS
277.3 + 40.2
-117.8 6.1
-142.1 6.7
-602.0 72.2
WALKER
REACH
ANT-DIR-INF
ANT-CIR-INE
BESF
530.9 1: 59.8
1543.4 + 98.5
153.3 + 4.2
165.2 + 6.8
LILAC
448.8 1 279.1
1341.0 - 47.7
85.5 18.7
69.5 1.6
ITER
65.6 3.8
1112.1
-338.5 1 64.1
337.0 69.8
CEMRL
577.9 + 68.1
514.3
-11.8 17.4
28.0 13.7
COMPS
80.0 + 34.4
326.9 = 380.0
-151.0 19.4
- + 4.8
Table I: The average return throughout the training process, comparing all the baselines. 'Walker' and 'Reach' are abbreviations for Ralker-Randd Rand-Params and Meta-World Reach, respectively.
of both BFSF and the baselines. Finally, for the third question, we conduct detailed ablation studies on the modules within BFSF, offering evidence their effectiveness.
3.1 Mains Results
We evaluate the Bayesian Fast-Slow Framework (BFSF) on five MuJoCo environments and one Meta-Worls environment. The MuJoCo environments (8 focus on robotic locomotion and are based on the MuJoCo simulator, while Meta- Meta-World (35 is benchmark designed for Multi-Task and meta-RL, specifically with robot manipulation tasks. These environments require adaptation across different reward functions (c.g.. walking direction for Chectah-Dir and Ant-Dir, target velocity for Cheetah- Vel, and goal location for Ant-Goal and Meta-Worle Reach), or across different dynamics (e.g., environment parameters for Walker- Rand- Params). These meta-RL RL environments are widely used in the meta-RL literature and are well-suited for non-stationary MDPs as well. We set the switching frequency of the underlying task 2000 episodes.
We compare BFSF with four reproduced baselines. LILAC (3 an algorithm for non-stationary MDPs that uses meta-RI. techniques. It learns latent variable to discriminate between tasks based on experiences. ITER [14 proposes an iterative approach to relearn the neural network, aiming to overcome non-stationarity. CEMRL [ learns task encoder from the gradients of decoder

---

and provides the task encoding to downstream RL. CoMPS 4 continuously alternates between two subroutines: learning a new task using RL and performing completely offline meta-leaming to prepare for subsequent task learning.
As shown in Figure 2 BFSF outperforms the baselines in non-stationary environments. The task switching is indicated by the gray dashed lines for clarity. While we conduct experiments with fixed switching interval, it important note that our algorithms are designed for the general setting where the task distribution and switching timing are completely unknown to the agent. For comprehensiveness, we also experiment with an unfixed switching interval in Section 3.3
In general, all algorithms show gradual performance improvements within a single task phase but experience a sudden performance drop immediately after task switches. This decay is expected, as the new task is unfamiliar to the agent. However, we observe that the learning curve in the second phase does not increase as quickly as in the first, which we refer to the interference phenomenon.
BFSF effectively mitigates interference, leading to better overall performance across continual phases. LILAC, based meta-RL methods, provides good cross-task generalization. However, it fails to address interference, resulting in slower learning during the second task. ITER's iterative relearning approach is solid defense against interference, but relying solely this approach leads to the learning of elementary policies, which hinders further generalization and improvement. For clarity, we only compare the curves of two baselines in the given figure, while full comparison of average return is provided in Table T. Full experiment results on the learning curve is provided in Appendix [B.2]
3.2 Infinite-Word Simulation
While the main results in Section 3.1 demonstrate the superiority of BFSF in mitigating interference and achieving cross-task generalization, it remains unclear how such methods perform in more realistic scenarios. In the real world, there are typically no explicit task boundaries, no fixed task initializations, and no finite set of predefined tasks. To better approximate these characteristies, introduce an Infinite- World Simulation in MuJoCo, an environment where the agent operates in a non-episodic, continuous manner without resets.
Unlike traditional MuloCo benchmarks, where each episode ends and resets after fixed number of steps (c.g., every 1000 steps), our infinite-world environment allows the agent to move seamlessly through boundless plane without ever being reinitialized. This design leads to a non-episodic interaction flow, closely mimicking the persistent nature of real-world settings. To support this, we implement a dynamic terrain loading module that handles environment generation on the fly, avoiding memory overload while preserving the illusion of endless space.
We design two signature environments to evaluate BFSF and baselines under this setup. Ant-Dir-Inf is a non-episodic, infinite-world variant of the standard Ant-Direction environment, where the agent required to walk alternately left and right across the plane; each time a directional goal is reached, the target direction flips. Inf is derived from Ant-Goal, where the target moves along circular trajectory of infinite radius, requiring the agent to constantly adjust and track it over time.
The corresponding results are reported in Table I In addition, Figure 3 visualizes the trajectories of different methods. BFSF consistently follow the evolving goal direction, while baseline methods react more slowly. This highlights BFSF's effectiveness in non- -pisoodii and task-free environments.
3.3 Ablation Studies
The ablation studies on each module of BFSF are conducted to answer the question: How do the individual components of BFSF contribute to improving performance? The results show that the 'fast' policy, as a whole, alleviates interference and enhances overall performance in non-stationary MDPs. Additionally, the dual-reset mechanism and relabeling further support the 'fast' policy by enabling more effective learning.
Unfixed Switching Interval We conducted experiments with an unfixed switching interval, as shown in Figure 4(a) The overall performance exhibits pattern similar to that observed in the main experiments with a fixed switching interval: the interference issue is mitigated by BFSF, and the agent's 's performance continues to improve training progresses.
7

---

The trajectoyy of postion Art--Di-ind
the trijectory of xy postion Art-Cr-att
SFSF
SFSF
30000
ULAC
ITER
ULAG
IER
CEMILL
CEMPL
CaMIPS
CoMPS
Saggets position
40000
posiion
20000
-10000
100
200
300
400
500
-30000 -20000
-10000
10000
20000
opoch
position
Figure 3: Visualized trajectories for Ant-Dir-Inf and Ant-Cir-Inf are shown. In only the BFSF algorithm successfully adapts quickly to the alternating goals between left and right directions. In Ant-Cir-Inf (right), only BFSF demonstrates rapid adaptation to the continuously moving goal along a circular path.
(left),
Cheetah
Di
2000
2000
1200
1200
0
10 witho toeld polloy
-500
witiout dual rosot rednsweling without Eleyestin
-1000
5000
10000
15000
20000
25003
5000
10000
15000
20000
25003
(a) BFSF with an unfixed switching interval.
(b) The ablation study BFSF.
Figure 4: Ablation studies about the unfixed switching interval and other modules.
The presence of the 'fast' policy, which learns from recent data, enables the agent better adapt changes non-stationary MDPs, as shown Figure [(b) The curve labeled
'aat'' Policy Ablation
"without fast policy' is identical to the baseline LILAC, as introduced in Section 3.1 While LILAC can generalize across tasks, struggles to efficiently learn the optimal policy in second task due to the interference problem. In contrast, BFSF addresses this issue, learning the second task at nearly the same speed as the first task, effectively overcoming interference. This trend also observed in ongoing tasks, where interference does not negatively impact the performance of BFSF.
Dual-Reset Reset Ablation The dual-reset mechanism ensures the effectiveness of the 'fast' policy. As shown in Figure 4bb) without the dual-reset, the learning curve exhibits lower performance due to suboptimal 'fast policy.
Relabeling Ablation As seen in Figure 4(b) relabeling significantly improves the learning effi- ciency of the 'fast' policy. With relabeling, the 'Tast' policy can continuously improve its performance, even on later tasks. In contrast, BFSF without relabeling, as shown Figure 5 struggles achieve similar improvement in later tasks.
Bayesian inference provides a suitable estimate of the expected return for both the 'fast' and 'slow' policies. Without it, the estimation becomes less effective, and proper hyperparameter tuning may required. As illustrated in Figure 4(b) the performance and resistance to interference deteriorate in the absence of Bayesian inference.
Bayesian Inference Ablation As introduced in Section 2.1
4 Related works
Non-Stationary MDPs Research on non-stationary MDPs primarily focuses on the challenge of recognizing potential tasks, as understanding the task transforms the non-stationary MDP into fixed MDP. LILAC [31] first employs latent variable models to learn environment representations based on

---

Cheetah- Dr
Cheelsh-Orr
2000
1900
500
800
BFSF
polncy
-000
-900
-1600
-1000
withost relabwiing skowe polloy
500
1000
2500
5000
1000
1500
20003
Figure 5: A comparison illustrating the performance of the 'slow" and "fast' policies. The main difference that with relabeling, performance the 'fast' policy remains higher throughout the phases, rather than significantly dropping after the initial phases.
current and past experiences, drawing inspiration from online learning and probabilistic inference. Subsequent works identified shortcomings in LILAC and proposed solutions. For instance, FANS-RL.
10 models non-stationarity in terms of individual latent change factors and causal graphs. ITER highlights the impact non-stationarity on latent representations, a form interference similar the one discussed in our work, leading the proposal of Iterated Relearning (ITER). Additionally, several theoretical works have also focused on non-stationary MDPs [ I 2 77. However, none of these prior works simultaneously address both cross-task generalization and the interference problem.
14
Besides, continual RL shares similarities with non- stationat MDPs but focuses on different chal- lenges, particularly catastrophic forgetting [25]. Although non-stationary MDPs and continual RL often treated as distinct problems, their focus differs, primarily due to the assumption that task bound- aries are known agents in continual RI. As result, research in continual RI. focuses designing submodules within the overall algorithm 30. such as replay buffers 6 network architecture (23), representation
22
optimization strategies 21 rather than addressing the non-stationarity itself.
Meta-RL Meta-RL. aims to enable agents to adapt more quickly to new tasks by leveraging prior experience from multiple tasks. It bears strong resemblance to non-stationary MDPs, but assumes that agents have full access all tasks, thus ignoring interference effects. RL learns an agent's learning algorithm, enabling it to adapt quickly new tasks by adjusting its internal update rule based on prior experience. MAML [12. a gradient-basee meta-RL algorithm, seeks a set of model parameters that can be quickly adapted to new tasks with minimal gradient updates. In contrast, context-based meta-RL methods, such as PEARL (24) and VariBAD (37, leverage contextual information to enable more efficient adaptation. While traditional meta- RL assumes access to all tasks during training, recent research has explored meta- leat-lln in the continual task setting which is closely related to our work non-stationary MDPs.
Inference-related Works The issue of interference has been widely explored related areas. In multi-task RL, task interference has been observed, and specialized network architectures have been proposed to mitigate this challenge [17 However, the source interference these works differs from that in non-stationary MDPs, where interference arises from unknown, streaming tasks. In continual RL, which focuses on maintaining high performance across incremental tasks, interference is also recognized and investigated at the representation level [18].
Conclusion
Non-stationarity poses significant challenge when deploying RL agents in real-world environments. In addition cross-task generalization through context-based algorithms, a challenge that has been thoroughly explored in previous works, we have identified that interference can severely hinder performance, especially when tasks conflict with each other. To address this issue, we introduce the Bayesian Fast-Slow Framework (BFSF), which incorporates a 'fast' policy that learns from recent history to prevent interference from previous tasks, and a 'slow' policy that maintains strong cross-task generalization. The use of Bayesian estimation ensures effective and unbiased selection between the fast and slow policies, enhancing the framework's 's adaptability and robustness. We also introduce dual-reset mechanism and data relabeling to further enhance efficiency. Experimental

---

results demonstrate BFSF's effectiveness in resisting interference and show that it outperforms baseline methods in various non-stationary environments.
Although BFSF improves adaptability and efficiency in non-stationary MDPs, the current experiments limited to small set of environments. Other types of non-stationarity, such blurred boundaries or stochastic non-stationary MDPs, have not yet been tested. Additionally, more realistic scenarios are needed to assess its applicability in real-world situations. In future work, we aim develop more realistic benchmarks that align closely with real-world applications of non-stationary MDPs, while further testing BFSF and exploring new challenges.
Acknowledgment
This work was supported by AF Office of Scientific Research (AFOSR) under grant FA9550-25-1- 0318 and with the generous support of the Amazon Research Award program.
References
[1] Alekh Agarwal, Sham Kakade, Mikael Henaff, and Wen Sun. Pe-pg: policy cover directed exploration for provable policy gradient learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20 Red Hook, NY, USA, 2020. Curran Associates Inc.
[2] Reda Alami, Mohammed Mahfoud, and Eric Moulines. Restarted bayesian online change-poine detection for non-stationary markov decision processes. In Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup, editors, Proceedings of The 2nd Conference on Lifelong Learning Agenty, volume 232 of Proceedings of Machine Learning Research, pages 715-744. PMLR, 22-25 Aug 2023.
[3] Andrew Ang and Allan Timmermann. Regime changes and financial markets. Annual Review of Financial Economics, 4(Volume 4, 2012):313-333 2012.
[4] Glen Berseth, Zhiwei Zhang, Grace Zhang, Chelsea Finn, and Sergey Levine. CoMPS: Continual meta policy search. In Deep RL. Workshop NeurlPS 2021, 2021.
[5] Zhenshan Bing, David Lerch, Kai Huang, and Alois Knoll. Meta-reinforcement learning in non-stationary and dynamic environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3476-33) 2023.
[6] Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change online continual learning. ArXiv, abs/2203.037999 2021.
[7] Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non- stationary Markov decision processes: The blessing of (More) optimism. In Hal Daumé III and Aarti Singh, editors, Proceedings the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pages 1843-1854. PMLR, 13-18 Jul 2020.
the
[8] Chuntao Ding, Zhichao Lu, Shangguang Wang, Ran Cheng, and Vishnu Naresh Boddeti. Mitigating task interference in multi-task learning via explicit task routing with non-learnable primitives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7756-7765, June 2023.
[9] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. ArXiv, http//611.027777 2016.
[10] Fan Feng, Biwei Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for non- stationary reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances Newral Information Processing Systems, volume 35, pages 31957-31971. Curran Associates, Inc., 2022.
10

---

[11] Songtao Feng, Ming Yin, Ruiquan Huang, Yu-Xiang Wang, Jing Yang, and Yingbin Liang. Non-stationary reinforcement learning under general function approximation. In Proceedings of the 40th International Conference on Machine Learning, ICMI'22. JoIR.org, 2023.
[12] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, ICML'11, pages 11 1126-1135. JML..orgg 2017.
[13] Weiyu Guo and Mark E. Wohar. Identifying regime changes in market volatility. Journal of Finuncial Research, 29(1):79-99 2006.
[14] Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations, 2021.
[15] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: model-based policy optimization. Curran Associates Inc., Red Hook, NY, USA, 2019.
[16] Leslie P Kaelbling, Michael L.. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Technical report, USA, 1996.
[17] Menelaos Kanakis, David Bruggemann, Suman Saha, Stamatios Georgoulis, Anton Obukhov, and Lue Van Gool. Reparameterizing convolutions for incremental multi-task learning without task interference. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 689-707. Springer, 2020.
[18] Samuel Kessler, Jack Parker-Holder, Philip Ball, Stefan Zohren, and Stephen J. Roberts. Same state, different task: Continual reinforcement learning without interference. 36:7143-7151, Jun. 2022.
[19] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual rein- forcement learning: A review and perspectives. Artif. Intell. Res., 15:1401-, 2020.
[20] Vincent Liu, Han Wang, Ruo Yu Tao, Khurram Javed, Adam White, and Martha White. Measuring and mitigating interference in reinforcement learning. In Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup. editors, Proceedings of The 2nd Conference on Lifelong Learning Agents, volume 232 of Proceedings of Machine Learning Research, pages 781-795 PMLR, 22-25 25 Aug 2023.
[21] David Lopez-Pat Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pages 6479, Red Hook, NY, USA, 2017. Curran Associates Inc.
[22] Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representa- tional continuity for unsupervised continual learning. In International Conference on Learning Representations, 2022.
[23] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning mask weights. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 20/8, pages 72-88, Cham, 2018. Springer International Publishing.
[24] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta- meta-reinforcemeet learning via probabilistic context variables. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Ma- chine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5331-5340. PMLR, 09-15 Jun 2019.
[25] ANTHONY ROBINS. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123-146, 1995.
[26] Amila Thibbotuwawa, Grzegorz Bocewiez, Grzegorz Radzki, Peter Nielsen, and Zbigniew Banaszak. Uav mission planning resistant weather uncertainty. Sensors, 20(2), 2020.
11

---

[27] Amila Thibbotuwawa, Grzegorz Bocewicz, Banaszak Zbigniew, and Peter Nielsen. A solution approach for uav fleet mission planning changing weather conditions. Applied Sciences, 9(19), 2019.
[28] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 0026-5033, 2012.
[29] Michael Wan, Jian Peng, and Tanmay Gangwani. Hindsight foresight relabeling for meta- reinforcement learning. In International Conference on Learning Reprexentations, 2022.
[30] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362-538 5383, 2024.
[31] Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst continual structured non-stationarity. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11393-11403, PMLR, 18-24 Jul 2021.
[32] Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based reinforcement learning with theoretical guarantees. CoRR, abs/1 00088007.35335 03858, 2018.
[33] Tengye Xu, Zihao Li, and Qinyuan Ren. Meta-reinforcement learning robust to distributional shift via performing lifelong in-context learning. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
[34] Zhuoran Yang, Yuchen Xie, and Zhaoran Wang. A theoretical analysis of deep q-learning. 2020.
[35] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation multi-task and meta reinforcement learning. In Conference Robot Learning (CoRL), 2019.
[36] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization, 2020.
[37] Luisa Zintgraf. Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: variational bayes-adaptive deep via meta-lcarning. J. Mach. Learn. Res., 22(1), January 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The introduction mentions all the parts in the text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are included in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
The assumptions and proof are in the main text and Appendix A.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The implementation details can be found in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
The code is in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppulli/guuuiu wwwtiicgguue//eeeee CoodeSubeiiieeee ssi asionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The implementation details can be found in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The error bars are provided in the Table I
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The implementation details can be found in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:
Answer: [Yes]
Justification:
The research satisfies the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creator of the code is mentioned in the code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsippteeeee www.reeewthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
17

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
18

---

Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htte:s/s//eerrippoccc///o/rreerp for what should or should not be described.
19

---

Technical Appendices and Supplementary Material
A About Theorem 2.1
A.1 Proof of Theorem 2.1
Proof. First, since Bayesian estimation provides unbiased estimate, the suboptimality of BFSF is bounded by the better performance between the fast and slow policies.
Suboptimallty(BFFF) & (innSubopttimlllyy//aat)..... Sutp://a/i../uuuopipiittttt/////
We analyze the suboptimalimality of the 'fast' policy ( Tar (Thr )I. the following proof.
Let MDP M have the dynamic function and reward function r. Similarly, let MDP M' have the dynamic function p' and reward function r'. We denote Mμ, as the MDP with dynamics function and reward function r. In Equation 5 we decompose [rad (TM) - Mar (T)))))) and use Theorem A.1
Lemma A.2
A.3
to complete the proof.
(F (T(r))
(T 7as (x + In (T - na (xTre))
-IuM,.r (=Mp.)) "MM (MM, + Imasg, (TMag..)) WN (TMMg...))
SluM,,. (T (Tipp.)) 7a4 (TMM)
+ InMp, (Thsy MMp. (IT (Tisg Myy,.r) + Inss (TiMg),.) 7Dp ( (TMM,)
Dee (PM,, PFs,)) + Vmax) + Taiss] H + (U (TM))) )+ V De, (P) (s, a))
(
Theorem A.1. (Relabeling gap 1)Let M1, M2 be two finite-horizon MDPs with the same reward function Then the distance of qaa (Ths)) and nMa (T),) is bounded by
(M,((TM,)) - TMM, "MM, (TMM)) S ch:
(6)
where en DDe (PM, PM2 (mmar + Vmaz) Tai - h), vh [M], ES
Proof. Since TM, is the optimal policy of MDP Mj and TM, is the optimal policy of MDP M2, Mas, (TM)) = VMr. 7M, (mM,) = VM
We begin our proof from the final horizon, h = H, and use the closeness at horizon h to establish the closeness at horizon h 1.
For the final horizon = H, we have V (s) 0 VM since it is the terminal state. Therefore,
IIV M. (s) - VM..Hlle lo0 S 0 = EH. Suppose
(s) - Vig S ch. Vs E Sp, ||V (s)
(7)
We need to prove
Vs E Sn-1, S1-1,,IVM,,,- Virhh-ille lloc I ch-1-
(8)
It is equivalent to prove
-h--1 S VMazh-I(5) Vi S eh-1-
(9)
20

---

For simplicity, but without loss of generality, we will prove the inequality on the right-hand side of the above equation.
LHS = m aEA
Σ
+ + ( (s)))
s'ES,
max d'ESk
Σ
pM + (s'))
-mxx
(sM + +
s'ES,
+ (d' a) PM, (s a)r + Pma (ss.
PM (s
max
Σ
Pa (s))s.))(((((ss))) (
aEA
s'ES,
S max aEAA
Σ PM, pM, (s1ss + +VM,,
s'ES,,
+ max aEA
(s (
S'ES,
+ PMM (s' pa
PM (M.
Σ
max
S'ES
max
Σ
par, (s
aEA
d'ESn
+ (M a) (5') - PM (s M (s')l}
max AEA
Σ S'ES
(PM - PMr (s' (s ()))r((ss))) a) +
Σ pm, - ri(
s'ESn
+ +MM, (8' (s ( PM, (8' (8'I3
< max aEA
(s'1s, a) Pa (s'ls, a) mmaa aeA
+ maxx
Σ
PMd (8 (d))s,)))aa)}
s'ESA
s'ES,
+ max aEA
(pM₂ (8") ()) pas, (s' a))VV + PMn, (5'18, a ()) (V (8') - v
S'E
<(De, (PM, (lssa),,M,, (+1s, (-1s, a))
+ Vima)) +Taiff (Vib, V (s)))
<(De, (PM, PM2) (Tmaz + Vmax) rdissl + 1.((h)
=[De, (Par, .PMs)) (Fmaz + Vmaz) + De, , (M,)((man + Vmax) + Tais - h)
-De,, (pas PM1z PMM2)))(mn) (Tmax + Vimz) Ta 11.
(10)
Lemma A.2. Let M1, M2 be two MDPs with the same dynamics function, but different reward
functions T1,72. Define (r) = Essayy<,, [r2(s, a) - ri(s,))),, which characterizes how erroneous the model is along trajectories induced by æ. Then
TMa (T) Ma,, (π) (T)
(11)
Proof. We know that M and M shares the same transition dynamics p, but different reward functions i(s,a) a) a(),a) Xu (, a). Therefore,
TMa (T) =E =E [r2(s, a)]
-E (r)(s, a) (r2(s, - r1()))))
(12)
=E(s,)>pBr (s) - E(s,)~pg,, (72(s, (s, = (T) + Un Un)r((
21

---

Lemma A.3. (Telescoping lemma) (36 32/. Let M1 and M2 be two MDPs with the same reward r(s, a), but different dynamics PM, and PM2 respectively. Let
G22 " "(s,a)
(13)
E (1 VMr, (s')] Egrppa (M,
Then,
(x) - nM, (T) = (La GM, (s, For each s E ES,a E A. a l based bound of [ (s,a)) is
(14)
Ghr, (8, a)) - Viaanee (pass (s, a), Par, (s)
(15)
A.2 Bound Considering the Optimization Error
Theorem 2.1 accounts for the error introduced by the relabeling process. To maintain consistency with prior work, we explicitly incorporate optimization suboptimality, following the approach in (34). by considering the policy obtained after K policy-update iterations n M' rather than the idealized optimal policy Tig.
Total Suboptimality
(T3s) 7aa ( K,
(T(M( "ar (-ir))) + (TTe) TM (TH )1 + Ina (TTS)) (M((
(16)
< I1D1 (par, PAss PPas)) + Vnuxx) + Taire H + |U (TTr)) ) + V Dipr
Pu.o Y |A| (log +2) naaa 1)/2 1)/2
47 K+1
+ +C
Rmax:
71-
(1 7)-
where each term explicitly captures different sources of error:
The first 3 terms (matching our original Theorem 3.1) represent the suboptimality caused by the relabeling process, quantifying the error introduced due to differences in dynamics and reward functions between the original MDP M and the relabeled MDP M'.
The newly introduced 4th and 5th terms explicitly quantify the optimization suboptimal- ity, representing errors arising from finite-sample approximations and iterative optimization.
B Experiments
B.1 Implementation Details
The experiments are repeated three times, with the mean and standard deviation shown in the curves and table.
The common hyperparameters are consistent with the original PEARL implementation. Additionally, the context consists of 200 episodes, and the relabeling percentage is set to 50% (i.e., half of the used batch is relabeled). The window of recent data w 100 episodes. The reset frequency v 50 episodes. The discount factor 7 = 0.99.
B.2 Full Learning Curves
A complete comparison of the learning curves for all four baselines is provided in Figure
B.3 Sensitive Studies and Other Experiments
Sensitive study on the relabeling percentage We performed an sensitive study on the relabeling
percentage in the Table 2
22

---

ULAC
200
Cardin
1300
4000
201
REST
200
CPHORT
Figure 6: The learning curve of BFSF with all the baselines including LILAC,, ITER, CEMRL and CoMPs.
Relabel Percentage (%)
0
10
20
30
40
50
Average Return
1055.5
954.9
1046.3
1097.5
1186.4
1039.4
Relabel Percentage (%)
60
70
80
90
100
Average Return
1217.7
1200.9
1148.1
1027.6
1129.1
Table Sensitive study on the relabeling percentage
Sensitive study on the window of recent data and the reset frequency is shown in Table 3 For reference, the average return of the baseline (LILAC) is 757.8. The results indicate that performance of our method is relatively insensitive to the choice of window size w and reset frequency v.
The experiment in gradually changing environments To validate our method in dynamically changing environments, conducted supplementary experiments in Table 4 in 180-task gradually changing Ant-Goal environment, with goals evenly distributed around a circle.
The experiment in stochastically changing environments As to settings with highly stochastic task boundaries, conduct experiments Cheetab-Dir environment in Table This environment consists of two tasks: moving forward and moving backward. A boundary period (one-third of the task length) exists during task switching, with each task having probability of 50%.
C Algorithm Details
C.1 Data Relabeling
To elaborate, relabeling is accomplished by learned dynamics and reward model s', [(s a) 15 which estimates the next state s' and reward r after taking action a in the state s. We can
10
50
100
150
200
BESF
1163.3
1273.6
1209.0
1121.1
945.5
10
25
50
75
V
100
BESF
1250.6
1169.0
1209.0
1132.1
1116.6
Table 3: Sensitive study on the window of recent data w and the reset frequency V.
23

---

BFSF
Baseline: LILAC
Gradual Ant-Goal
-328.5 = 2.1
619.2 3.7
Table 4: The experiment results in gradual Ant-Goal environment. BFSF outperforms the baseline, showing ability to adapt continuously changing environment.
BFSF
Baseline: LILAC
Stochastic Ant-Goal
887.8 +: 21.6
615.4 + 5.7
Table 5: The experiment results in stochastic Ant-Goal environment. BFSF significantly outperforms baseline LILAC in such an environment.
substitute the original next state and reward in the experience replay, even from different tasks, with those predicted by the model, represented as syatabel Tretabel = f(8, a).
Relabeling is widely used in the RL community [29] 33]. The underlying principle is to maximize data reuse for sample efficiency. In our work, given that the environment evolves over time, we leverage a context-basee dynamics model f a, c), which provides different dynamics depending on context c.
As discussed in 29 33 sample efficiency. In work, adopt data relabeling mitigate the issue of performance degradation in non-stationary environment, because the increased amount of relabeled data allow the 'fast' policy better and faster adapt these tasks, which verified by our ablation results.
and confirmed by our ablation studies, data relaheling substantially enhances
C.2 Motivation for the Dual-Reset Mechanism
The dual-reset mechanism was introduced upon observing that RL algorithms tend to encounter performance degradation when alternating between tasks, a trend corroborated by prior research [4]. This mechanism effectively addresses the issue. We present the phase performance for SAC in the alternating Cheetab-De Dir environment (where the two tasks are moving forward and backward) in T
C.3 Bayesian Fast-Slow Framework
We only utilize data within recent window, which keeps the effective sample size small, and we set the prior value dynamically updated upper bound to encourage exploration. As result, if policy has not been sufficiently selected, the prior strongly influences the Bayesian estimate, leading large posterior value that naturally encourages exploration of that policy.
Phase performance
Task
Task 2
Task 1
Task 2
Task
Without dual-reset
954.6
117.8
92.0
8.6
-27.4
With dual-reset
759.7
625.6
697.5
409.8
782.7
Table 6: In the first phase (Task 1), SAC without the dual-reset mechanism performs well, even outpacing the SAC with dual-reset. However, during the alternating tasks, the performance degrades significantly.
24

---

D Posterior Calculation of Normal Distributions
Assume is known.
plut Biz. ),166) p(u)p(( Big, Ha, 1/6)
2
no
x exp {
(1 - mm))² x exp - ( [1))
2
1
a xexpl
{
(00 + =n)) + 2 (2 + 2n
2
00140 + noy )²}
(17)
x exp { 2 (00 no)
@ + no
Normal(M1: of),
00/40 + noR
where μ4
00 + no
@0 + n
E Visualization of Choosing Slow/Fast Policies
We presented visualization in left sub-figure Figure in our paper (attached here in Figure 7). In the left sub-figure, the 'fast' policy predominates in the selection during the initial phases, while the 'slow' policy shows competitive performance as the amount of accumulated data from different tasks increases. Additionally, in the latter part of single phase, the 'fast' policy surpasses the 'slow' policy after learning from relabeled recent data.
Chestah Dr
Cheetah Oir
1500
1900
1000
500
BFBF
klow polcy
-000
1000
-100
-1000
wilhost relaboling sdowi policy
-200
Fast polkoy
500
1000
1500
2000 2000
100
1000
1000
2000
2000
Figure 7: A comparison illustrating the performance of the 'slow" and 'fast' policies. The main difference is that with relabeling, the performance of the 'fast' policy remains higher throughout the phases, rather than significantly dropping after the initial phases.
25


Reviews:
[{"review_id": "i0cQhQPjLn", "reviewer": "Reviewer_fHvL", "summary": "This paper introduces the Bayesian Fast-Slow Framework (BFSF) to address the problem of interference in non-stationary reinforcement learning environments. The key issue is that when RL agents encounter changing tasks over time, learning from previous tasks can negatively interfere with performance on new tasks, especially when tasks are contradictory.\n\nBFSF combines two complementary policies: a \"fast\" policy that learns quickly from recent data to mitigate interference, and a \"slow\" policy that uses meta-reinforcement learning to capture knowledge from all previous tasks for cross-task generalization. A Bayesian estimation mechanism dynamically selects between these policies based on their expected performance using recent return history.\n\nThe framework also incorporates a dual-reset mechanism to prevent performance degradation in neural networks and a data relabeling technique to enhance learning efficiency by augmenting recent data with relabeled historical experiences. Experimental results on MuJoCo and Meta-World environments demonstrate that BFSF effectively mitigates interference while maintaining good cross-task generalization, outperforming existing baseline methods in non-stationary settings.", "questions": "1) How does the computational overhead of maintaining dual policies, meta-RL training, and Bayesian inference scale with the number of tasks and complexity of environments?\n\n2) How sensitive is the framework to key hyperparameters like the window size (w=100), reset frequency (ν=50), and relabeling percentage (50%)?\n\n3) Since the method assumes unknown task boundaries, how would it perform in scenarios with gradual task transitions or when task changes are very subtle?\n\n4) Would BFSF be effective in other domains like discrete action spaces, vision-based tasks, or environments with fundamentally different dynamics than the tested locomotion tasks?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) Addresses the understudied but critical issue of cross-task interference in non-stationary MDPs\n2) The dual-policy approach elegantly balances fast adaptation (recent data) with cross-task generalization (historical data), with principled Bayesian selection between them.\n3) Thorough experimental validation across multiple environments\n\nWeaknesses:\n1) Experiments are restricted to a small set of MuJoCo and Meta-World environments with fixed task switching intervals\n2) The framework requires maintaining dual networks, meta-RL training, Bayesian inference, and data relabeling, which likely increases computational complexity compared to simpler baselines.\n3) Relies on assumptions like normal distribution for Bayesian inference and requires a learned dynamics model for data relabeling", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable review.\n\n**Q1** How does the computational overhead of maintaining dual policies, meta-RL training, and Bayesian inference scale with the number of tasks and complexity of environments?\n\n**A1** Thank you for the question. The computational overhead of our method remains modest and does not scale with the number of tasks, since we always maintain only two policies (fast and slow), regardless of how many tasks are encountered. We do not allocate a separate policy per task; instead, generalization is achieved through a context-based conditioning mechanism.\n\nThe Bayesian inference module only performs selection between the two policies based on recent returns, and thus its overhead is negligible and does not scale with task number or environment complexity.\n\nCompared to context-based baselines (e.g. LILAC), the primary additional cost comes from training and maintaining extra 'fast' dual policies. In practice, this results in only a moderate increase in training time.\n\n**Q2** How sensitive is the framework to key hyperparameters like the window size (w=100), reset frequency (ν=50), and relabeling percentage (50%)?\n\n**A2** Appendix B.3 provides sensitive studies on the window of recent data $w$, the reset frequency $\\nu$ and the relabel percentage. Additionally, we present the following study for reference:\n\n**Sensitivity study on the window of recent data $w$:** (For reference, the average return of the baseline (LILAC) is $757.8$.)\n\n|  | $w=10$ | $w=50$ | $w=100$ | $w=150$ | $w=200$ |\n| ---- | ---- | ---- | ---- | ---- | ---- |\n| BFSF | $1163.3$ | $1273.6$ | $1209.0$ | $1121.1$ | $945.5$ |\n\n**Sensitivity study on the reset frequency $\\nu$:**\n\n||$\\nu=10$|$\\nu=25$|$\\nu=50$ | $\\nu=75$ | $\\nu=100$ |\n|----|---- | ---- | ---- | ---- | ---- |\n|BFSF|$1250.6$ | $1169.0$ | $1209.0$ | $1132.1$ | $1116.6$ |\n\n**Sensitivity Study on the relabel percentage:**\n|Relabel Percentage (%)|0|10|20|30|40|50|60|70|80|90|100|\n|----|----|----|----|----|----|----|----|----|----|----|----|\n|Average Return|1055.5|954.9|1046.3|1097.5|1186.4|1039.4|1217.7|1200.9|1148.1|1027.6|1129.1|\n\n**Q3** Since the method assumes unknown task boundaries, how would it perform in scenarios with gradual task transitions or when task changes are very subtle?\n\n**A3** The experiments in gradual task transitions are provided in Appendix B.3. Additionally, we present the result for reference:\n\n||BFSF|Baseline: LILAC|\n|----|----|----|\n|Gradual Ant-Goal|$-328.5\\pm2.1$|$-619.2\\pm3.7$|\n\n**Q4** Would BFSF be effective in other domains like discrete action spaces, vision-based tasks, or environments with fundamentally different dynamics than the tested locomotion tasks?\n\n**A4** BFSF is designed to address the interference problem in non-stationary MDPs, a challenge that may also arise in domains like discrete action spaces and vision-based tasks. We appreciate the valuable suggestion and agree that exploring these domains will be crucial for further validation."}, {"review_id": "7woD6njisI", "reviewer": "Reviewer_tpda", "summary": "This paper introduces the Bayesian Fast-Slow Framework (BFSF), a novel approach to address the challenge of interference in non-stationary reinforcement learning environments. Unlike previous works that focus solely on cross-task generalization, BFSF dynamically switches between a fast policy trained on recent data for rapid adaptation and a slow policy trained via meta-RL to leverage past experience. A Bayesian inference mechanism determines the optimal policy to use at each point in time.", "questions": "- Could the Bayesian selection mechanism be enhanced with exploration to avoid early commitment bias? The current policy selection is fully deterministic based on posterior mean estimates. This may lead to under-utilization of one policy (e.g., the slow policy) if the other dominates early on. Have you considered incorporating stochastic policy selection mechanisms to encourage balanced exploration of both fast and slow policies, especially in the early phases?\n- How robust is the fast policy’s performance to inaccuracies in the context-based dynamics model used for relabeling? Since relabeling heavily depends on the quality of the learned model, it would be helpful to understand how model errors affect fast policy training. Have you evaluated the impact of model approximation errors on policy performance? \n- What happens when fast policy dominates data collection for extended periods? Although slow policy is updated in each epoch, it depends on data collected through interaction through the past policy. It would be helpful to clarify how such imbalance is mitigated in practice, and whether you considered mechanisms for guaranteeing minimum data exposure for each policy.\n- In Figure 4(b), the ablation results suggest that the fast policy and relabeling have the most significant impact on overall performance, whereas the removal of Bayesian inference seems to cause only a minor degradation. Could you clarify the relative importance of the Bayesian selection mechanism in the overall framework? Specifically, how critical is Bayesian inference to the stability and adaptability of BFSF? If the system performs almost as well without Bayesian inference and slow policy, what unique benefits does it provide in more challenging or uncertain settings?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. Strengths\n- The paper introduces a well-motivated and novel framework (BFSF) specifically designed to tackle interference in non-stationary RL environments.\n- The use of Bayesian inference for dynamically selecting between fast and slow policies is principled and adaptive, allowing the agent to respond to changing task distributions efficiently.\n- The incorporation of the dual-reset mechanism and context-aware data relabeling significantly enhances the fast policy’s adaptability and sample efficiency.\n\n2. Weaknesses\n- The framework deterministically selects the policy (fast or slow) with the higher expected return, without incorporating stochastic exploration, which may lead to overcommitting to suboptimal policies in early stages.\n- If one policy (e.g., fast) dominates selection early on, the other (e.g., slow) may suffer from under-training due to limited data collection, which is not fully analyzed in the paper.\n- The performance of the fast policy partially depends on the accuracy of the learned context-based dynamics model, but the paper does not deeply analyze its limitations or robustness.\n- The framework introduces several new hyperparameters (e.g., reset frequency, window size for Bayesian updates), but sensitivity to these is not extensively studied or theoretically justified.", "was_revised": true, "final_justification": "First, I sincerely apologize for the delayed response. I had intended to submit my updated review during an overseas trip, but it appears that the submission did not go through, most likely due to unstable airport network conditions.\n\nRegarding the paper, after carefully reading the authors’ rebuttal, I found their responses sufficiently address all of my initial concerns.\nGiven the clarifications provided, I have updated my evaluation to better reflect the quality and contribution of the work.", "rebuttal": "Thanks for the valuable review.\n\n**Q1, W1** Could the Bayesian selection mechanism be enhanced with exploration to avoid early commitment bias? The current policy selection is fully deterministic based on posterior mean estimates. This may lead to under-utilization of one policy (e.g., the ‘slow’ policy) if the other dominates early on. Have you considered incorporating stochastic policy selection mechanisms to encourage balanced exploration of both ‘fast’ and ‘slow’ policies, especially in the early phases?\n\n**A1** Exploration is inherently incorporated in our approach. Specifically, we only utilize data within a recent window, which keeps the effective sample size small, and we set the prior value $\\mu_0$ to a dynamically updated upper bound to encourage exploration. As a result, if a policy has not been sufficiently selected, the prior strongly influences the Bayesian estimate, leading to a large posterior value that naturally encourages exploration of that policy.\n\nMoreover, the ‘slow’ and ‘fast’ policies do not compete exclusively as in standard bandit settings. The experience gathered by one policy can be efficiently leveraged by the other. Evidence for this can be seen in existing offline meta-RL research [1], where even fully offline datasets are sufficient to support effective context-based policies.\n\nLastly, our algorithm selects between the ‘fast’ and ‘slow’ policies by sampling from their respective posterior distributions and choosing based on the probability that one sample exceeds the other, which inherently provides exploration capability.\n\n**Q2, W3** How robust is the ‘fast’ policy’s performance to inaccuracies in the context-based dynamics model used for relabeling? Since relabeling heavily depends on the quality of the learned model, it would be helpful to understand how model errors affect ‘fast’ policy training. Have you evaluated the impact of model approximation errors on policy performance?\n\n**A2** Thank you for the question. The context-based dynamics model is used to relabel past experiences to make them more relevant to the current task, thereby accelerating the fast policy’s adaptation. When the model is inaccurate, the relabeled data may become less helpful, but the fast policy can still rely on newly collected (on-policy) data. As a result, the impact of model errors is typically limited to short-term adaptation performance, and does not severely affect long-term learning.\n\nTo evaluate the sensitivity to model quality, we conducted an ablation study on the relabeling percentage (Appendix B.3), summarized below:\n\n|Relabel Percentage (%)|0|10|20|30|40|50|60|70|80|90|100|\n|----|----|----|----|----|----|----|----|----|----|----|----|\n|Average Return|1055.5|954.9|1046.3|1097.5|1186.4|1039.4|1217.7|1200.9|1148.1|1027.6|1129.1|\n\nAs the results show, our algorithm maintains relatively stable performance across a wide range of relabeling percentages, demonstrating that our method is robust and does not heavily depend on precise tuning. \n\n**Q3, W2** What happens when fast policy dominates data collection for extended periods? Although slow policy is updated in each epoch, it depends on data collected through interaction through the past policy. It would be helpful to clarify how such imbalance is mitigated in practice, and whether you considered mechanisms for guaranteeing minimum data exposure for each policy.\n\n**A3** This concern relates closely to the explanation in **A1**: the ‘slow’ and ‘fast’ policies do not compete exclusively, and the experience collected by one policy can be effectively leveraged by the other.\n\nTo better answer your question, we can analyze how the ‘fast’ and ‘slow’ policy functions in a real experiment in Figure 5. In the beginning, ‘fast’ policy dominates as there are not enough multi-task data for ‘slow’ policy to generalize, while in the tasks afterward, the ‘slow’ context-based policy serves as a good initial point and the ‘fast’ policy further improves efficiently. To summarize, the ‘slow’ policy provides reasonable performance when encountering a new task, while the ‘fast’ policy dominates the improvement inside a task, which can also be utilized by the ’slow’ policy.\n\n**Q4, W7** In Figure 4(b), the ablation results suggest that the fast policy and relabeling have the most significant impact on overall performance, whereas the removal of Bayesian inference seems to cause only a minor degradation. Could you clarify the relative importance of the Bayesian selection mechanism in the overall framework? Specifically, how critical is Bayesian inference to the stability and adaptability of BFSF? If the system performs almost as well without Bayesian inference and slow policy, what unique benefits does it provide in more challenging or uncertain settings?\n\n**A4** We appreciate the reviewer’s insightful comment. As shown in Figure 4(b), the fast policy and relabeling indeed have the most direct and substantial impact on performance, with the removal of Bayesian inference resulting in only a minor drop in performance under the current benchmarks. However, it’s important to note that the ablation without Bayesian inference can be viewed as a deterministic approximation of the Bayesian posterior, using the mean. The full Bayesian approach still provides a principled and theoretically grounded framework for decision-making.\n\nWhile the fast policy and relabeling can improve performance independently in simpler tasks or when prior knowledge is strong, the true strength of Bayesian inference shines in more complex and uncertain environments. In such settings, Bayesian reasoning helps mitigate early commitment biases, improving the agent’s ability to adapt over time. By weighing policies based on their long-term effectiveness, Bayesian inference ensures greater stability and robustness, especially in non-stationary or noisy conditions.\n\n**W4** The framework introduces several new hyperparameters (e.g., reset frequency, window size for Bayesian updates), but sensitivity to these is not extensively studied or theoretically justified.\n\n**Answer:** Appendix B.3 provides a sensitivity study on the window of recent data $w$ and the reset frequency $\\nu$. Additionally, we present the following study for reference:\n\n**Sensitivity Study on the window of recent data $w$:** (For reference, the average return of the baseline (LILAC) is $757.8$.)\n\n|  | $w=10$ | $w=50$ | $w=100$ | $w=150$ | $w=200$ |\n| ---- | ---- | ---- | ---- | ---- | ---- |\n| BFSF | $1163.3$ | $1273.6$ | $1209.0$ | $1121.1$ | $945.5$ |\n\n**Sensitivity Study on the reset frequency $\\nu$:**\n\n||$\\nu=10$|$\\nu=25$|$\\nu=50$ | $\\nu=75$ | $\\nu=100$ |\n|----|---- | ---- | ---- | ---- | ---- |\n|BFSF|$1250.6$ | $1169.0$ | $1209.0$ | $1132.1$ | $1116.6$ |\n\nIn the Bayesian update, the prior value $\\mu_0$ is set to a dynamically updated upper bound to encourage exploration. The value of $\\phi_0$ is set to 1 without further tuning.\n\n\n[1] Kate et. al., Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables. ICML 2019."}, {"review_id": "DYPijWDQbM", "reviewer": "Reviewer_DwXB", "summary": "This paper proposes an RL framework (BFSF) for continual non-stationary reinforcement learning. The agent maintains two policies: a “slow” meta-RL policy (implemented with PEARL) that exploits long-term experience for cross-task generalisation, and a “fast” policy trained only on recent data to react quickly to distribution shifts and reduce negative transfer (termed “interference”). The current policy is chosen each epoch via Bayesian model selection on a sliding window of recent returns. The fast policy further uses (i) a dual-reset mechanism that periodically re-initialises one of two identical networks to avoid performance collapse, and (ii) context-conditioned data relabeling to augment its recent replay buffer. Theoretical analysis gives a sub-optimality bound that decomposes errors from relabelling and optimisation. Experiments on six tasks (from MuJoCo and Meta-World benchmarks) plus two “infinite-world” variants show improved average returns over baselines such as LILAC, ITER, CEMRL and CoMPs.", "questions": "Please refer to the weaknesses above. Mainly,\n- Is data relabeling also used for all the baselines with fast policies? If not, please do so. \n- Report PEARL alone (the \"slow\" learner from prior work). Also report the \"fast\" learner used in Algorithm 2 step 10 with experience relabeling. The absence of these two baselines are of major concern to me.\n- Increase seeds to at least 10 and include confidence intervals. Otherwise the claimed gains may not be significant.\n- Please define all symbols in Theorem 2.1 and provide a proof sketch. Under what realistic conditions can the bound be < V_max? If the bound is always loose, consider reframing the theory section.\n- Explain algorithmic timing. Describe precisely how training, dual-reset, and environment interaction interleave.\n- In Fig. 2, what causes the periodic drops in BFSF between switches, and why are the drops absent at 6 k / 10 k steps on Meta-World Reach?\n-  Line 497 says \"As we cannot include images in this rebuttal\" in the appendix of the submission. Please include said images.", "limitations": "There is no broader impacts statement.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n### Quality\n- Really solid engineering effort. BFSF combines a principled fast/slow decomposition, Bayesian model selection, dual-reset, and experience relabeling using learned world model. Each component is at least intuitively motivated and evaluated in ablations.\n- It outperforms strong baselines across eight non-stationary tasks and maintains higher returns through task switches.\n- The empirical investigation of “interference” is interesting, especially with the visualisations in Figure 3.\n- The hyperparameter sensitivity ablations are also very commendable.\n\n### Clarity\n- The High-level intuition of BFSF is easy to grasp and the algorithms are summarised in pseudo-code.\n- Decent related-work section clearly positions BFSF with respect to meta-RL and continual RL literature.\n\n### Significance\n- Addresses an important pain-point in deploying meta-RL agents in streaming settings where sharp task boundaries are unknown.\n- The fast/slow idea could inspire follow-up work that plugs in different meta-learners or fast learners, and the infinite-world experiments push beyond the usual episodic benchmarks.\n\n\n### Originality\nIt provides a novel combination of known components. While fast-adaptation, Bayesian policy selection, and relabeling are known ingredients, their integration as a coherent framework for mitigating cross-task interference in non-stationary MDPs is new. The dual-reset mechanism is also a simple but clever trick that may interest practitioners suffering from representation collapse (although it is unclear how sensitive it is to its hyperparameter).\n\n\n# Weaknesses\n\n### Quality\n- Theoretical section is unfortunately weak. Theorem 2.1 appears vacuous if I understand it correctly. The bound is ≥ V_max when M≠M′, with undefined symbols p_M, η_M, etc. and the promised proof sketch (line 145) is never given.\n- There is no discounting in the MDP problem definition. Is there no discounting in the experiments? In general, critical hyper-parameters like discount factor, window size w, reset frequency ν, RL algorithm for line 10 in Alg. 2) and number of seeds (only 3) are either buried in the appendix or absent, undermining statistical reliability.\n- PEARL is used as the slow policy but is not reported as a standalone baseline. Likewise there is no “fast learner + relabeling only” baseline (since that's not new to this work), so improvements may conflate benefits from the proposed approach.\n- It looks like the authors did not also do data relabeling for all the baselines with fast policies (e.g. ITER). This is a major concern if that is indeed the case.\n- Unlike the authors claim on line 471, Table 3 shows (I assume average returns) that the approach is highly sensitive to the window size w and reset frequency v with a very clear trend (higher w and v lead to poorer performance).\n- For Figure 4.b, the authors say \"without fast policy\" is identical to LILAC. Shouldn't it be PEARL since that's what the authors claim to be using for the slow policy? There's also too much noise in the figure. E.g. It is hard to justify the \"Dual-Reset Ablation\" interpretation given by the authors given the noise. An easier domain like Cheetah-vel or Ant-Goal (from looking at the results in Fig 2) would have been better for this ablation.\n- The Bayesian inference ablation is inconclusive. Figure 4(b) shows almost identical returns on average with and without Bayesian selection, casting doubt on the necessity of the Bayesian component.\n\n\n### Clarity\n- Key quantities like Posterior(...), p_M, η_M, are undefined in the main text. \n- Several typos hinder readability (lines 114, 122, 228).\n- Figures use raster graphics and colour palettes that are hard to distinguish for colour-blind readers; \n- Regular drops in BFSF curves in-between task switches are unexplained (e.g. in Fig 2). Also why does BFSF have no performance drop at 6000 and 10000 in Metaworld Reach (Fig 2)?\n- Algorithmic flow between Alg. 1, Alg. 2, and environment interaction is still ambiguous: when exactly is the environment reset, and how are online interactions interleaved with dual-reset?\n\n### Significance and Originality\n- With only three seeds it is unclear if the results are significant.\n- The main novelty in this work seems to be from the specific Bayesian inference and dual-reset ideas. Unfortunately, Figure 4 does not support the claim that these are useful significant components of the approach.\n\nOverall, the paper tackles an important practical issue and shows promising empirical gains. However, clarity, baseline coverage, and theoretical rigor need strengthening before the contribution can be considered fully conclusive.", "was_revised": true, "final_justification": "The authors really took some time and effort to properly address my concerns and provide a very detailed response. I am happy with their clarification and happy to increase my score to an accept accordingly.", "rebuttal": "Thanks for the valuable review.\n\n**Q1** Is data relabeling also used for all the baselines with fast policies? If not, please do so.\n\n**A1** Thank you for the valuable suggestion. We will include baselines with data relabeling in the final version. Due to time constraints, we currently provide a comparison between our algorithm without relabeling and the baselines:\n\n||BFSF w/o relabeling|LILAC|ITER|CEMRL|CoMPs|\n|----|----|----|----|----|----|\n|Cheetah-Dir|$1053.9\\pm55.0$|$757.7\\pm94.9$|$813.4\\pm19.0$|$852.6\\pm36.1$|$277.3\\pm40.2$|\n\n**Q2, W3, W4, W6** Report PEARL alone (the \"slow\" learner from prior work). Also report the \"fast\" learner used in Algorithm 2 step 10 with experience relabeling. The absence of these two baselines are of major concern to me.\n\n**A2** LILAC, which we use as one of our baselines, is just the non-stationary version of PEARL. PEARL is a meta-RL algorithm that requires the task id, which is not available in the non-stationary problem setting. LILAC transfers PEARL into the non-stationary MDPs scenario.\n\nAnother baseline, ITER, can be viewed as the ‘fast’ learner only, as ITER also adopts the mechanism of periodically resetting the model.\n\nBased on the comparisons with these baselines, our method demonstrates strong performance, addressing concerns regarding the adequacy of baseline coverage. Nonetheless, we recognize the importance of clearer presentation and will improve the baseline annotations and related discussions in the next revision.\n\n**Q3, W13** Increase seeds to at least 10 and include confidence intervals. Otherwise the claimed gains may not be significant.\n\n**A3** Thank you for the helpful advice. We will increase the number of seeds in the next version. For now, we include BFSF results with three additional seeds:\n\n||Cheetah-Dir|Cheetah-Vel|Ant-Dir|Ant-Goal|\n|----|----|----|----|----|\n|BFSF|$1189.4\\pm41.5$|$-97.8\\pm2.8$|$574.9\\pm225.9$|$-369.1\\pm208.1$|\n|LILAC|$757.7\\pm94.9$|$-154.6\\pm2.5$|$318.5\\pm66.2$|$-426.6\\pm4.6$|\n\n**Q4, W1** Please define all symbols in Theorem 2.1 and provide a proof sketch. Under what realistic conditions can the bound be < V_max? If the bound is always loose, consider reframing the theory section.\n\n**A4** The symbol definitions appear on Line 142; we reproduce and briefly explain them below:\n- $M,M’$: the original and relabeled MDPs.\n- $H$: the horizon.\n- $r_{max},V_{max}$: the maximum reward and value.\n- $r_{diff}$: the maximum reward gap between $M,M’$.\n- $U_{r,r'}(\\pi)=E_{(s,a)\\sim\\rho^\\pi_{M}}[r’(s,a)-r(s,a)]$: the average reward difference between two MDPs under the same occupancy measure $\\rho_M^{\\pi}$.\n- $D_{\\ell_{1}}$: the L1 distance.\n\n**Proof Sketch** The bound decomposes the suboptimality into two parts that bridge the true MDP $M$ and the relabeled MDP $M'$:\n1. $|\\eta_M(\\pi^\\ast_M)-\\eta_{M’}(\\pi^\\ast_{M’})|$: the difference between the optimal returns in $M$ and $M’$, corresponding to the first RHS term.\n2. $|\\eta_{M’}(\\pi^\\ast_{M’})-\\eta_{M}(\\pi^\\ast_{M’})|$: the return difference when applying the same policy $\\pi_{M’}^\\ast$ under two different MDPs, $M$ and $M'$. This term can be further decomposed into two parts, corresponding to the second and third terms in Theorem 2.1, because the MDPs differ in both reward and dynamics.\n\n**Tightness:** The bound is not as large as $V_{max}$, as $V_{max}$ can be viewed as a constant multiplied by an L1 distance (e.g. $V_{max}D_{\\ell_1}(p_{M’}(s,a),p_M(s,a))$). The relabeled MDP $M’$ trained to closely approximate the original MDP $M$, so the L1 distance tends to be small after training.\n\n**Q5, W12** Explain algorithmic timing. Describe precisely how training, dual-reset, and environment interaction interleave.\n\n**A5** In each iteration, we first select either the ‘fast’ or ‘slow’ policy. We then collect 5 episodes, with each episode consisting of 200 steps, through environment interaction. Afterward, we perform 2000 steps of RL training. The model reset occurs every 50 episodes.\n\n**Q6, W11** In Fig. 2, what causes the periodic drops in BFSF between switches, and why are the drops absent at 6 k / 10 k steps on Meta-World Reach?\n\n**A6** The periodic drops are caused by abrupt task changes with potentially conflicting objectives — for example, in Cheetah-Dir, the tasks involve moving left versus right. In contrast, in Meta-World Reach, although goals are randomly sampled across the space, neighboring tasks may occasionally result in nearby goals, leading to smoother transitions and less pronounced performance drops.\n\n**Q7** Line 497 says \"As we cannot include images in this rebuttal\" in the appendix of the submission. Please include said images.\n\n**A7** We apologize for the missing image. Since images cannot be included in the rebuttal itself, we will ensure that the final version of the paper includes the missing figure with the performance curves.\n\n**W2** There is no discounting in the MDP problem definition. Is there no discounting in the experiments? In general, critical hyper-parameters like discount factor, window size w, reset frequency ν, RL algorithm for line 10 in Alg. 2) and number of seeds (only 3) are either buried in the appendix or absent, undermining statistical reliability.\n\n**Answer:** Thank you for pointing this out. We will make sure to explicitly present these key hyperparameters alongside the main definitions in future versions for better clarity. Below are the values used in our experiments:\n\n- Discount factor: $0.99$. The MDP definition omits discounting since our environments are finite-horizon tasks where discounting is not strictly necessary. However, we use standard discounted return during training.\n\n- Window size $w$: $100$.\n\n- Reset frequency $\\nu$: $50$.\n\n**W5** Unlike the authors claim on line 471, Table 3 shows (I assume average returns) that the approach is highly sensitive to the window size w and reset frequency v with a very clear trend (higher w and v lead to poorer performance).\n\n**Answer:** We agree that the performance varies across different choices of $w$ and $\\nu$, as shown in Table 3. However, even under suboptimal hyperparameters, our approach consistently outperforms the baseline (LILAC average return: 757.8), demonstrating the robustness and effectiveness of BFSF.\n\n|  | $w=10$ | $w=50$ | $w=100$ | $w=150$ | $w=200$ |\n| ---- | ---- | ---- | ---- | ---- | ---- |\n| BFSF | $1163.3$ | $1273.6$ | $1209.0$ | $1121.1$ | $945.5$ |\n\n||$\\nu=10$|$\\nu=25$|$\\nu=50$ | $\\nu=75$ | $\\nu=100$ |\n|----|---- | ---- | ---- | ---- | ---- |\n|BFSF|$1250.6$ | $1169.0$ | $1209.0$ | $1132.1$ | $1116.6$ |\n\n**W6** There's also too much noise in the figure. E.g. It is hard to justify the \"Dual-Reset Ablation\" interpretation given by the authors given the noise. An easier domain like Cheetah-vel or Ant-Goal (from looking at the results in Fig 2) would have been better for this ablation.\n\n**Answer:** Thank you for the suggestion. We agree that domains like Cheetah-vel or Ant-goal may yield cleaner curves and help highlight component effects. We chose Cheetah-dir because it presents a more challenging setting where interference is more severe, making the benefits of our design more evident. That said, we appreciate the reviewer’s feedback and will consider ways to improve the clarity and readability of the curves in future revisions.\n\n**W14** The main novelty in this work seems to be from the specific Bayesian inference and dual-reset ideas. Unfortunately, Figure 4 does not support the claim that these are useful significant components of the approach.\n\n**Answer:** Thank you for the comment. While Bayesian inference and the dual-reset mechanism are indeed important components of our method, we would like to clarify that the core novelty lies in the dynamic switching between the fast and slow policies, which directly addresses the interference problem in non-stationary environments. The dual-reset and data relabeling mechanisms help these modules function effectively, while Bayesian inference provides a theoretically grounded and adaptive way to select between them.\n\nMore specifically, compared to the ablations without dual-reset or Bayesian inference, BFSF demonstrates a clear advantage at the beginning of the third phase: these ablated versions struggle to leverage experience from the first phase due to interference, while BFSF maintains better policy continuity. This suggests that the full BFSF framework, including Bayesian switching and dual resets, contributes meaningfully to mitigating interference.\n\n**W8-12** Weakness in clarity.\n\n**Answer:** Thank you for the careful reading and constructive feedback. We will correct all typographical errors and improve figure clarity in the final version."}]

Author rebuttal:

Meta-review: This paper proposes a Bayesian Fast-Slow Framework (BFSF) for mitigating interference in non-stationary reinforcement learning. The framework combines a fast policy for rapid adaptation, a slow policy based on PEARL for generalization, and a Bayesian mechanism to select between them. Additional techniques (dual-reset and data relabeling) are introduced to stabilize training and accelerate adaptation. Experiments on MuJoCo and Meta-World benchmarks demonstrate clear improvements over strong baselines, with ablations highlighting the contributions of individual components.

The reviewers appreciated the principled design, novelty of the fast/slow Bayesian integration, and strong empirical results on challenging non-stationary tasks. The rebuttal and discussion clarified initial concerns about missing baselines, limited seeds, and theoretical presentation, with the authors providing additional experiments and improved explanations. While the theoretical component remains relatively weak and the empirical evaluation could be broader, the main ideas are well-motivated and convincingly validated.

Overall, this is a solid contribution to continual and meta-RL, addressing an important problem with a clear and practical solution. The paper is not quite at the level of a spotlight or oral due to its limited theoretical depth and scope, but it makes a valuable and well-executed contribution.
Decision: Accept (poster)