Wavelet Predictive Representations for Non-Stationary Reinforcement Learning
Authors: Min Wang
Xin Li
Ye He
Yao-Hui Li
Hasnaa Bennis
Riashat Islam
Mingzhong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Non-Stationary Reinforcement Learning
Representation Learning
Primary area: reinforcement learning

Abstract: The real world is inherently non-stationary, with ever-changing factors, such as weather conditions and traffic flows, making it challenging for agents to adapt to varying environmental dynamics. Non-Stationary Reinforcement Learning (NSRL) addresses this challenge by training agents to adapt rapidly to sequences of distinct Markov Decision Processes (MDPs). However, existing NSRL approaches often focus on tasks with regularly evolving patterns, leading to limited adaptability in highly dynamic settings. Inspired by the success of Wavelet analysis in time series modeling, specifically its ability to capture signal trends at multiple scales, we propose WISDOM to leverage wavelet-domain predictive task representations to enhance NSRL. WISDOM captures these multi-scale features in evolving MDP sequences by transforming task representation sequences into the wavelet domain, where wavelet coefficients represent both global trends and fine-grained variations of non-stationary changes. In addition to the auto-regressive modeling commonly employed in time series forecasting, we devise a wavelet temporal difference (TD) update operator to enhance tracking and prediction of MDP evolution. We theoretically prove the convergence of this operator and demonstrate policy improvement with wavelet task representations. Experiments on diverse benchmarks show that WISDOM significantly outperforms existing baselines in both sample efficiency and asymptotic performance, demonstrating its remarkable adaptability in complex environments characterized by non-stationary and stochastically evolving tasks.

Published as a conference paper at ICLR 2026
WAVELET PREDICTIVE REPRESENTATIONS FOR NON- STATIONARY REINFORCEMENT LEARNING
Min Wang', Xin Li Ye He', Yao-Hui Li', Mingzhong Wang'
Hasnaa Bennis', Riashat Islam',
Beijing Institute Technology, China
Microsoff Research, Al Frontiers, New York City
University of the Sunshine Coast, Australia (min mminwanges, xinli, heye, 1 liyachui, 3820252027) @bit.edue mwang@usc: edu. au
ri riashat is www.aaammmiisssoooooeeeeeoemosse com
ABSTRACT
The real world inherently non-stationary, with ever-changing factors, such as weather conditions and traffic flows, making it challenging for agents to adapt to varying environmental dynamics. Non-Stationary Reinforcement Learning (NSRL) addresses this challenge by training agents to adapt rapidly to sequences of distinct Markov Decision Processes (MDPs). However, existing NSRL approaches often focus tasks with regularly evolving patterns, leading to limited adaptability in highly dynamic settings. Inspired by the success of Wavelet analysis in time series modeling, specifically its ability to capture signal trends at multiple scales, we propose WISDOM to leverage wavelet-domain predictive task representations to enhance NSRL. WISDOM captures these multi-scale features in evolving MDP sequences by transforming task representation sequences into the wavelet domain, where wavelet coefficients represent both global trends and fine- fine-graine the-graindd variations of non-stationary changes. In addition to the auto-regressive modeling commonly employed time series forecasting, we devise a wavelet temporal difference (TD) update operator to enhance tracking and prediction of MDP evolution. We theoreti- cally prove the convergence of this operator and demonstrate policy improvement with wavelet task representations. Experiments diverse benchmarks show that WISDOM significantly outperforms existing baselines in both sample efficiency and asymptotic performance, demonstrating its remarkable adaptability in complex environments characterized by non-stationary and stochastically evolving tasks. Source code is available at http:s//gghhbb.oo///Maaaaags///s
INTRODUCTION
Humans excel at dynamically adjusting their behaviors to adapt to continually changing environments extended periods. A promising paradigm to realize this adaptability in artificial agents is Non-Staaionaay Reinforcement Learning (NSRL). which focuses on training policies that can adapt sequentially multiple tasks with varying state transition dynamies or reward functions. Meta-RI. enhances sample efficiency and policy generality by identifying shared structures across diverse tasks, making particularly effective for rapid adaptation. Based on meta-RL, recent NSRL methods have improved task inference through Gaussian mixture distributions, which emphasize the independence of different tasks Bing al. (2023b) or distance metric losses that enforce stringent smoothness conditions Sodhani et al. (2022) However, they generally perform poorly on non non- stationtroo tasks due to meta-RL's inherent stationarity assumption and neglect of temporal correlations between tasks.
In NSRL,, the task evolution process is typically modeled as history-dependent stochastic process, implying that task changes follow certain trends or periodic patterns, thus allowing the tracking and prediction of task evolution Xie et al. (202I); Poiani al. (2021); Chen et al. (2022). A line of work Xie et al. (022): Ren et al. (2022) attempts to explicitly model the task evolution process as a first-order Markov chain, which limits its applicability smoothly evolving non-stationary
"Corresponding author.
Tooee-vvolvng changes of tasks in dynamics or rewards that result in non-stationarity.

---

Published as a conference paper at ICLR 2026
tasks, making it ineffective for more complex tasks with rapid changes, where it can result in the accumulation of prediction errors. Bing et al. (2023a) employs recurrent- based neural networks to handle non-stationary sequential data. Poiani al. [2021); Chen et al. (2022); Tennenholtz et al. (2023) assume a history-dependent task evolution process and approximate it with Gaussian Process (GP) (2004) or planning in latent space. The non-stationary kernel function adapts GP to varying covariance, but requires prior knowledge and introduces more parameters, In addition, most existing methods have been primarily limited task evolution processes that exhibit regular pattern with fixed periods. However, practical non-stationary tasks generally exhibit time-varying periods or frequencies, posing a challenge for handling such irregular patterns with stochastic periods.
As in Fig. (a), a noisy non-stationary signal changes with increasing frequency, with its three input stages (A, B, C) having similar means ( 0) and variances (> 2). Although indistinguishable in the time domain, they can be distinguished in the Fourier spectrum with different main frequencies in (ig.[().. However, the Fourier Transform (FT) Cooleyy Tukey (1965) does not reveal when each frequency component appears. For instance, reversing the sequence to create fast-to-slow pattern (C-B-A), thereby generating a different signal. yields the same Fourier spectrum. In contrast, Wavelet Transform Polikar et al. (199)) can resolve this by simultaneously preserving time- tmmefrequency domain information of the non- stationary signal (initial approximation coefficient) and iteratively extracting multi-scale features using different frequencies. Based on sampling theorem [Shannon (1949), each decomposition of the approximation coefficient halves the sequence length, reducing the amount of data without losing fundamental features. High-frequency noise (detail coefficient) in Fig. [ (c-d) is gradually separated from low-frequency components (approximation coefficient), and after the second decomposition in Fig. the smooth low-frequency feature (Fig. (c)) basically reflects the original evolving trend (Fig. (a)).
Inspired by the success of WT theory in handling non- stationay time series by gradually sepa- rating different frequency trends Polikar et al. (1996); Yang et al.) (202)), we propose Wavelet- Induced task repreSentation preDiction for nOn-stationary reinforceMent learning (WISDOM), which leverages wavelet domain information from the task representation sequence z to ex- tract underlying, structural and temporal patterns. Specifically, we start by encoding trajectorics (st, 8441,
r [ to obtain z to imply the changes of tasks Xie et al. (021)); Chen et al. (202)). Each 2; in this z sequence preserves the influence of dynamics and rewards. Each dimension of 2; corresponds to a variate in a multivariate time series. Subsequently, we utilize a wavelet representation network to transform z into the wavelet domain. Iterative decompositions of yield an approximation coefficient that captures slowly changing features and a series of detail coefficients that capture rapidly evolving local features. To remove noise and preserve more rapid task changes, we selec- tively filter detail coefficients through downsampling. We then transform these wavelet coefficients back to the time domain to restore more intrinsic task represen- tations. Moreover, we design a wavelet operator for the explicit temporal difference (TD) update of the wavelet representation network to help capture changes in task structures. Additionally, an auto-regressive loss is incorporated to reinforce long-term temporal dependencies, empowering task evolution predictions. Finally, we integrate restored task representations into policy learning, facilitating prompt policy adjustments and improving policy adaptability across diverse non-stationary tasks. Our key contributions include:
Statimar sinael
Far W Spemin
Frecueccy
Desel Casticiats
350
pest Colfiiaaag
www
Ageroemadir Conmirtts
580 20
350
24
Figure 1: A motivating example.
1. We first propose to address non-stationary RL via perceiving its evolution process in the wavelet domain. By iteratively performing wavelet decomposition, we can capture different evolutionary patterns emerging with stochastic period by utilizing series of varying resolutions or frequencies. This allows WISDOM to dynamically adjust its behavior in response to evolving task conditions. leading to rapid adaptability and improved policy performance.
2. We theoretically demonstrate that wavelet-domain features can serve as informative indicators of policy performance. And our wavelet task representations are demonstrated to lead to improved policies. To better capture structural regularities of the task evolution process, we design a wavelet TD update operator, theoretically ensuring its convergence.
2

---

Published as a conference paper at ICLR 2026
3. We perform a comprehensive evaluation on broader tust-distr Meta- World Yu al. (2020), Type- I Diabetes Xie (2018). and MuJoCo Todorov et al. (2012), and the overall results demonstrate the superior and efficient adaptability of WISDOM compared to baselines. We provide new tasks on these benchmarks and open-source for evaluation in challenging non-stationary settings.
/
2 RELATED WORK
Wavelet Features- Basedd DL. Recent studies primarily investigate the incorporation of wavelet
features for image processing Yu al. (202]): Yao et al. (2022), audio signal analysis Pan al. (2022); Shi et al. (2023); Zhang et al. (2024b), and time series prediction Zhou et al. (2022) y Minhao al. 2021p. In addition, some studies [Boóss al. (2020); Kong al. (2022) have turned wavelet transforms to integrate time-frequency features for handling non-stationary signals. A wavelet neural network Stock & Anderson (2022) has been proposed to learn a specialized filter-bank for non- stationarity modeling. Yang et al. 2024 a graph spectral wavelet to capture the high-frrqueecy components of non-stationary graph signals. To address the issue of online label shift, Qian al.] proposes a streaming wavelet operator that estimates environmental changes for efficient non- stationary learning. Furthermore, a wavelet attention network is introduced analyze heterogeneous time series Wang al. (2023) or even predict seasonal components Wan et al. (024)).
Non-Stationary RL. A significant strand of recent research frames the problem of not-stationarity within the meta-RI. framework. Al-Shedivat et al. 2018 makes an initial endeavor to apply gradient- based meta-RL to address non-stationary tasks. Meta-experience replay Riemer et al. (2019) and parameter learning rate modulation Gupta et al. (2020) are introduced to enhance adaptability. How- ever, policy gradient updates are typically sample inefficient, and they primarily focus on combining context-basse based meta-RL. to achieve faster adaptation in non-stationary tasks. To improve task represen- tation, Gaussian Mixture Models Bing et al.] (2023b), task-metric approaches (Sodhani et al. (2022), causal graph Zhang al. (2024a); [Fng al. (2022), and recurrent neural networks Poiani al. (202T) have been successively leveraged to assist the context encoder in extracting invariant features.
To accurately model the task evolution process, some approaches Xie et al. (2021) >) Ren et al. (2022)
hypothesize the latent context sequence to follow a Markov chain formulation. While such methods
demonstrate superior adaptability in smoothly changing tasks, they struggle to adapt effectively to
complex tasks that involve sudden changes. Therefore, another branch of research Poiani et al. (202T);
Tennenholtz et al.) (2023): Chen et al. 2022) assumes deetoryde of the latent context and seeks to utilize Gaussian Processes Seeger (2004), the Rescorla- Rescorla-Wagner model Rescorla (197 and reward functions Chen et al. (2022) to better characterize the task evolution process. Change-point detection methods represent another line of research for dealing with statioati tasks. They utilize sets of partial models specialized for different environments Da Silva et al.) (2006), ensembles of context-dynamics predictors that capture different modes of task distributions Alegre al. (202T), or Wasserstein distances between behavior trajectories Liu et (2024) to detect when the task has changed. Different from prior methods, we introduce a principled mechanism that tracks and adapts to task dynamics directly in the wavelet domain, rather than relying solely on implicit learning from data. Through iterative wavelet decomposition, our method naturally separates evolving patterns across multiple frequencies, faithfully capturing the underlying structure of non non-stationary tasks.
3 PRELIMINARY
3.1 CONTEXT-BASED META-RL
Meta-RL is defined on a distribution of stationary tasks p(T). where each task is an MDP represented by tuple (S, A. P, R, y, P(so)). in which S denotes the state space, A the action space, P the transition dynamics, R the reward function, P(so) the initial state distribution, and 7 E [0. 1) the discount factor /Sttonn & Barto (1998). In meta-RL, both P and R are assumed to be unknown, with different tasks differing only in P or R. In context-based based meta-RL, the task representation is derived from past transition histories (referred to as context) by a context encoder en. Building on task inference, recent NSRL methods focus modeling the evolution of 2 to improve the accuracy of trend predictions, thereby enhancing the adaptability of the policy π. T[a (ss 2). In a conventional meta-RI. setting, the agent can interact with numerous MDPs, yet the P and R of these MDPs remain
3

---

Published as a conference paper at ICLR 2026
constant over time, indicating stationary MDPs. In contrast, NSRL requires the agent to engage with multiple MDPs where and/or R evolve over time, introducing new challenges for rapid adaptation.
3.2 NON-STATIONARY RL
Non-stationary RL is defined on the distribution of time-evolving tasks (Tr). where an agent interacts with sequence of MDPs : distory-dependenn stochastic process p, i.c. Wh+1 (Wh+1 WW, W1..... ,h.....hh) where w is task ID that regulates the properties of different MDPs (Praan et al. (202T). p. To better adapt to more practical scenarios, w longer undergoes inter-episode changes, nor does it follow equidistant intra-episode changes. Instead, our setting. and consistent with [Bing al. (2023b); [Chnn et al. (2022), w varies within episodes with relatively stochastic period. The overall objective of NSRL is formulated as:
,Mu.. The evolution of these MDPs is determined by
argmaxiB... WA
Σ ET.
Enrr Mwn Lh=0
(1)
For example, consider Walker task in MuJoCo where the agent's mass changes twice during interaction, resulting in three distinct transition dynamics corresponding to MDPs wp. W1, and w2. Their durations are To. Ti and T2, with The goal of non-stationary RI is to find a policy that maximizes the expected cumulative reward over this entire trajectory, with a horizon of To + Ti + T2.
{0, 1.2] and timestep ranging from 0 +Ta
3.3 WAVELET TRANSFORM
The Wavelet Transform (WT) Burrus et al. (1998) analyzes the various frequency components of a non-stationary signal utilizing scalable and translatable mother wavelet function (b). To achieve multi-resolution analysis, the Discrete Wavelet Transform (DWT))Shensa et al. (1992) is introduced to decompose the signal layer by layer different resolutions. Let the initial approximation coefficient un be equal to an input discrete time series æ(n). By substituting a pair of low-pass filter yo and high-pass filter y1 for ((), the following iterative form derived through discrete convolution:
K
K
Umm (12)
Σ (k)kkmm (1kk)mm-122 (2n -),, gm gm (n) =
Σ 1=
y1 (A) Um 12n k),
(2)
A=1
where is the decomposition level and k is the resolution size. Approximation coefficients u. repre- sent low- low-frequency components, whereas detail coefficients gm represent high- freuuerr components. These multi-resolution and localization properties allow the DWT to simultaneously capture both the global features and local variations of the signal, making it well-suited for handling non-stationary signals, where frequency and period change over time. We also provide a derivation of DWT in Appendix A.I to enhance understanding.
METHODOL.OGY
Sec. 4.1 first outlines the setting of non-stationary tasks and describes how conduct task inference. Subsequently, Sec. Sec....2jllaoorates on the implementation of a trainable wavelet representation network that converts task representation sequences terns. Thereafter, Sec. 4.3 devises a wavelet temporal difference (TD) update operator to collaborate with the ruto-regressive (AR) loss optimizing the wavelet representation network. Finally, See. 4.4 explains how to effectively harness this inherent structural representation in policy learning.
the wavelet domain capture inherent evolving pat-
4.1 MODULE A: NON-STATIONARY TASK INFERENCE WITH CONTEXT ENCODER
In scenarios with non-stationary task distributions p P(T),, assuming the agent has collectively un- dergone H instances/times MDP evolution when interacting with the environment, the resulting trajectory, composed of recent history of transitions c0.7 to + 1 MDP segments (i.e. Ma Ma, Mw,..... ".....,,wp" To further enhance applicability, in our non-stationary setup, the period of each Ma,,, denoted Th, is assumed unknown. The current
.0tHf (abbreviated
context C), corresponding
4

---

Published as a conference paper at ICLR 2026
Ao
Transitions
Context Encoder
An
Network
Turget W Network
Task Inference
(s,0,s,7)
Conv1
Conv2/W
W(s, a, a.s.))
Max
MDP, MDP, MDF,
Downsampling Amm
Wavelet TD Loss
+
Period
(s,
KL Loss
D Mur
Modulle C:
a,s
Linear Layer
2
Modalle B:
Policy Leuning
Actor Loss
Critic Loss
Regresenitation Prediction
GELU Activation
Figure 2: The architecture of WISDOM. Module A begins with task inference to derive time-domain task representation 2. Then module B transforms 2 into wavelet domain by a wavelet representation network Ya, jointly optimized by wavelet TD loss and AR loss to derive wavelet task representation 2. Finally, module C integrates 8 to adjust the policy based on predicted evolving trend.
MDP duration Th potentially differs from its predecessors Th-1, and T Σ Th. Each transition (s, a, s', within C encapsulates the current state s, action a, next state s', and the reward r.
To enhance practicality, we assume that the agent lacks prior knowledge of the environment and that the true task ID w is inaccessible. We employ context encoder parameterized by 1) to infer task-relevant information. The encoder ey processes C' to generate a task representation sequence .....,,22 2r] that is generally viewed as approximation of the w sequence Xie et al. (2021); Chen et al. (202)). where each 2 uniquely corresponds to each transition c. The KL-divergence is used variational approximation an information bottleneck RRakllly al. (2019) to train en::
2g, 21.....
Jn = Ec-B [DLL (z(()
where p(z) represents a Gaussian prior, and the context is sampled from the replay buffer B.
4.2 MODULE B: TRACKING TASK EVOLUTION VIA WAVELET REPRESENTATION
In NSRL, the task sequence M Ma is generally modeled as a history-dependent stochastic pro- cess Xie et al. al..((22)))))) (20TT; Chen et al. (2022); Poiani et (202T) From a time series analysis perspective, this sequence can be viewed as a non-stationary signal, with the underlying frequency of each type of task fr. changing over time. WT is widely recognized for its ability to process non-stationary signals by decomposing them into different frequency components (multi-scale features) Polikar et al. (1996). Conceptually supported by Xie al. (202T): Poiani al. (202T), WISDOM is the first address NSRL by leveraging wavelet-domall task representations to track the temporal evolution of tasks.
Inspired by the design of prevalent WT networks Dord (2016); Shi al. (2023), our wavelet representation network Yú consists of two dilated causal convolution networks, Conyl and Conv2, followed by a linear layer to guarantee the capture of underlying task dynamics. Specifically, Y, performs the DWT in Eq. 2 with recursive convolution and transforms into the wavelet domain:
gm = ( U1: M), um Conv2(u No: M), m E MJ, M E Et
(4)
where the learnable convolution kernels yo and y1 can be initialized as classical basis functions such as Haar waveler" Hence, Y. is guided to approximate traditional wavelet behavior while adaptively adjusting the kernel values during training. Yo functions low-pass filter, eliminating high-frequency components, thereby enabling the approximation coefficient um to capture the overall trends of non-stationary task evolution. Conversely, Y1 acts frequency components, allowing the detail coefficients gm to capture rich detail features of tasks. After performing M decompositions on um Yd obtains the Mth uss along with the total Al detail coefficients g1:M- During each decomposition step m. the network downsamples and preserves the most recent detail coefficients g.. Finally, Y, applies a linear transformation to revert g.. and u,s to the time domain, yielding a more expressive and intrinsic wavelet task representation sequence 2.
high-pass filter, gathering high-
2 Since the time steps of changes in the MDP is unknown, superscript is omitted in the following notation. 3 FFor haar wavelet wavelet[Pattanaik Pattanaik Bouatouch 1995). yu [1/V2, 1/ 22 computes the average over adjacent elements, smoothing the sequence, akin to low-pass filter. In contrast, = 11/22, -1/v2) computes the difference, highlighting local changes and extracting high-frequency components, similar high-pass filter.
5

---

Published as a conference paper at ICLR 2026
4.3 OPTIMIZATION OBJECTIVE DESIGN FOR WAVELET REPRESENTATION NETWORK Yo
We propose TD-style optimization of the representation in the wavelet domain and theoretically demonstrate the convergence of the corresponding wavelet TD operator, thereby facilitating learning and search for task-oriented representations.
In wavelet theory, the Com:2 layer (referred as the W network hereafter) integrates a low-pass filter to learn stable and inherent non-stationary features Shensa (1922). We define a TD-style update operator over wavelet-based task features learned by the W network as JW (z+) Z, (2 to explicitly update Ys, where I denotes the discount factor in diagonal matrix form. Similar successor features BBrrreto al. (2017), wavelet features W(z)) satisfy the Bellman equation. Theorem demonstrates that J is contraction mapping (Proof in Appendix A.2) ensuring the convergence of wavelet representation updates and allowing for stable and consistent learning. Unlike most existing work that implicitly updates the representation network with TD loss over the value function, our explicit TD update will not neglect low-reward yet critical features. Although rewards in many RL settings can be sparse or delayed, learned representations tend to be denser and more informative. Intuitively, the wavelet TD update allows Y2 to better capture structural regularities in MDP sequences and enhance sample efficiency.
Moreover, the wavelet TD loss relies solely on the single-step future task representation 24-41 and provides more concise optimization objective than the commonly used auto-regressive (AR) loss in time-series forecasting Shi et al. ): Oord (0066) Our W network is also implemented with dilated causal convolution, ensuring that the i-th output of Yg only depends on the first i input elements, thus maintaining the conditional dependency essential for AR modeling to predict task changes. However, AR objectives often suffer from accumulated prediction errors, where errors in the current representation propagate and degrade future predictions. As remedy, the wavelet TD update with the target W network helps mitigate error propagation and stabilizes optimization through delayed target update.
Theorem Let W denote the set of all functions W S A + CD that map from the time domain to the wavelet domain. The wavelet update operator F :WW + W, defined as
FW(z)) 2u + IW (W(z))))
is contraction mapping, where 2e and Ze+1 represents the current and next task representation sequence. respectively, and I denotes the discount factor in diagonal matrix form.
The overall optimization objective of wavelet representation network Yo is formulated follows:
T
To = ay EewB (Wo(z)) - (z. + TEx [Wa(z++1l)) ESNYo log II P(2l2ed) (5) t=
Wavelet TD loss
AR loss
where ay balances the contribution of each loss. Similarly to DQN Mnih et al. (013). W p is a target network to stabilize the training and is updated separately via an exponential moving average of the parameters of the We network. Complemented to the wavelet TD loss, the AR loss adheres to stricter temporal constraints, preventing the trends captured by Y. from being temporally misaligned due to the relaxed orthogonality of the learnable filters. Moreover, it reinforees long- term temporal dependencies, empowering Yo to make predictions based on the task representation sequence. Together, they facilitate rapid adaptation evolving task structures.
4.4 MODULE C: POLICY LEARNING THROUGH WAVELET TASK REPRESENTATIONS
To establish more intimate relationship between dynamic adjustment of the policy in advance and the non-stationary evolving trends, we further incorporate the predicted wavelet task representation i generated by Y, into the policy iteration. Our WISDOM is based on the Soft Actor-Critic (SAC) algorithm Haarnoja et al. (2018) and can also be integrated with any downstream RL algorithms.
The objective of training the contextual critic network Qu is to minimize the squared residual error:
To = B(e.a)-B.BwY,
(Qu (s,0,,3 Quarget)

---

Published as a conference paper at ICLR 2026
with the target Q value defined as Quaget "y 3. [Q as', 2))] where Ç denotes stopping the backpropagation of gradients of the target critic network Qç. The parameters of Qç updated with exponential moving average derived from the weights of the Qu network. The contextual policy network To can be updated by optimizing the following objective:
Jo Ro [a log (Ta (als, 2)) - Qu (8, a, R)],
(7)
where a is a temperature coefficient. Alg. I Appendix C presents the pseudocode of WISDOM. The following theorems establish a connection between task representations through the wavelet transform and policy performance. representations preserve the ability to indicate policy performance. And in Theorem we prove that the policy is improved with restored time-domain task representations during policy iteration.
Theorem 2 we demonstrate that the wavelet-domain task
Theorem Suppose that the reward function R(2) can be expanded into a Bth- Taylor series for 2 E R then for any two policies Ti and T2. their performance difference is bounded as:
R R
(
11m Jual
W (dg.88g) W (dg. 8a) (8) 1895 d, 20
max sup
6!
where wit) (d, 3) denotes the wavelet transform of the bth power of the task representation se- quence 2bb) [20, E1,..., 2T (b) for any integer E [1, B). and Jx represents the average policy performance over the entire task space within a single episode.
See Appendix A.3 for the proof. Theorem 2 describes how the performance differences of the wavelet-domain features control the performance of the corresponding policies. We prove that wavelet-domain features can serve as an informative indicator of policy performance, aiming at more efficient policy iteration find the optimal policy and exhibiting fast convergence in experiments.
Theorem WISDOM returns a policy T(-|s, 2) conditioned on the wavelet task representation that improves upon its intermediate policy produced during policy iteration Th, satisfying JWISDOM 2 Jrn under certain assumptions, where JwISDOM denotes the final policy performance of WISDOM.
The assumptions and proof are provided in Appendix A.4 Theorem 3 describes that our restored in the time domain after wavelet transform leads to improved contextual policy during iteration. The wavelet transform is well known for enhancing the signal-tar iignal-o-noise ratio (SNR) by separating different frequencies Polikar et al. (1996). And é is expected to filter out mask-irreeevant information and helps the policy focus on essential non-stationary characteristics, providing a clearer optimization direction.
5 EXPERIMENTS
Task settings. Following SeCBAD Chen et al. (2022), the stochastic period Tý, of an MDP Mws, is sampled from Gaussian distribution with mean of 60 and variance of 20 time steps. We mainly performed experiments on the following three benchmarks, and more details are in Appendix B
1. Meta- World Yu et al. 2020). consisting of 50 robotic manipulation tasks with a broader distribu- tion that is generally considered challenging, exhibits non-stationarity the continuous variation of the target position for the robotic arm movement, which is correlated with the reward function;
Type-1 Diabetes Xie (2018) aims to test the control of blood glucose levels (observation) type-1 diabetic patients by injecting insulin (action). The con-stationarity is reflected in the continuous variation of food intake, which affects blood glucose levels and the dynamics of tasks;
3. MuJoCo Todorov et al. (2012), widely adopted in NSRL, has only parametric diversity. We considered two MuJoCo scenarios: a) Evolution tasks with changing reward functions (e.g.. Walker-Vel); b) Evolution tasks with changing dynamics (e.g., Cheetah-Damping).
Baselines. We compared WISDOM with four competitive NSRL baselines: CEMRL Bing et al. (2023a). TRIO Poiani et al, (2021), SeCBADCCher et al. (2022). and COREP ZZang et al. (2024a), which utilize the Gaussian Mixture Model, Gaussian Process, reward function, and causal graph model the task evolution process, respectively. Additionally, we compared SAC Haarnoja et al.
7

---

Published as a conference paper at ICLR 2026
2 1.0
Door-nlock
Poucet-Close Cloze
Button Press
R30
DoorLockk
1.0
00.8
TN
0.8
0.8
0.0
0.6
0.6
0.6
Augrage
0.4
0.2
0.4
0.2
0.4
0.3
Avergge
0.4
0.0
0.0
0.0
0.0
0.00
0.35
0.70
1.05 1.40
0.00
0.35
0.70 1.05
0.0
0.2
0.4
0.6
0.9
1.0
0.6
Time Staps
Time Steps
Time Staps
Time Steps
3.0
Dace-Close
1.0
Plate-Side
1.0
Handle-Press
Rote
1.0
Plate Seze-Siiaee Si Back
Average Succer
0.8
0.0
0.8
0.6
0.6
0.2
0.4
0.2
K0.2
0.4
Avcrage
0.4
0.0
0.0
0.0
0.00
0.35
0,70
0.00
0.35
0.70
0.30
0.35
0.70
Steps
Stops
Stops
Figure 3: Testing average performance on Meta-World over 6 random seeds. Our WISDOM achieves rapid convergence and exhibits excellent asymptotic performance.
20188) PEARL Rakelly et al. (2199) and RL2 Duan et al. (2016) highlight the importance of task inference and modeling of the task evolution process, respectively.
Meta-World results. Fig. 3 and Table D illustrate the testing curves and converged performance,
respectively. As in Fig. 3 WISDOM consistently demonstrates exceptional adaptation efficiency
across all environments, surpassing all baseline methods. As in Table | WISDOM achieves the highest average success rate with minimal variance in most environments, significantly contributing to effective noise filtering and the retention of fundamental -reefreuyncy components. Although SeCBAD outperforms PEARL in Door-Unloct and Door-Lock, it falls behind PEARL. regarding rapid adaptability in most environments. In Meta- WoradWorld, where the evolutionary factors are relatively complex due to the potential variations in the three- diccesiimnsiinall coordinates of the target position, agents require substantial trial and error during the early training due to fluctuating rewards. Conse- quently, SeCBAD faces challenges in relying solely on rewards to identify whether task evolution has occurred. Surprisingly, PEARL and RL have impressive overall converged performance, indicating their potential to adapt non non-stationary tasks by solely meta-learning a policy, even without explic- itly modeling the inherent changing characteristics of task evolution. The competitive SAC further illustrates that inaccuracies in the inference and modeling of non-stationary tasks in more challenging environments may undermine the adaptability of the policy. When CEMRL attempts to cluster complex task distributions into few categories, representation collapse often occurs. Conversely, clustering into a larger number of categories significantly increases the complexity. The selection of kernel functions in TRIO has similar dilemma, causing wholly poor performance.
Table 1: Converged average test success rate + standard error (%) on Meta- Meta-World.
Doer-Unlock
Faucet-Close Button-Press
Door-Lock
Door-Close
Plate-Slide
Handle-Pross Plate-Slide-Back
CEMRL
4.0828.71
0.17+0.37
1.83+1.00
6.67=11.11
0.42::993
0.00+0.000
45.17=38.5 58
0.00:20100
TRIO
3.92:6.46
20.171:22.1
10.42:16:.6
20.75
0.001::00
6.421110.3
33.25-26.92
0.20::028
PEARL
10.25
96.33116.88
39.42.132.54
50.67 1:3.74
73.50:11.118
85.33 20.73
$2.50:113.73
SeCBAD
11.58+17.01
93.42-8.99
36.58+332.46
72.50=31.05
05
89.33-16200
71.50+19.93
93.92-12.37
79.53++11.39
COREP
67.50:145.96
98.001:2211
96.83314224
98.67-1188 89
100.0010.000
64.174.688
0.00
43.004-10.19 10.19
SAC
1.6743.14
98. 38.33-33.3
62.834439.22
62.67+390.8
33
100.00
55.00++37.15
61 61.25-46.66 63
90.03+55.8
RL3
5.75.19.38
81.00
19.8331.26.40
70.67
15
56.92.115.77
94.75.1..65
79.23
WISDOM
91.5819.36
99.92.10.19
99.42:11.30
99.67.10.75
100.0010.000
96.50.1.044
100.00:10.00
90.57.17.66
Type-1 Diabete results. We further examined the adaptability of all models in a more real- istic environment that investigates the relation- ship between the control of blood glucose levels and insulin injections in diabetic patients. As observed in Fig. 4 WISDOM demonstrates no- tably more efficient adaptability, indicating that wavelet task representations can effectively cap- ture the trend of changing blood glucose levels and aid in selecting appropriate insulin dosages to maintain normal levels. Moreover, SeCBAD and CEMRL struggle to adapt to enviromments with changing food intake, such as environments where both lunch and dinner (2 meal) quantities vary over time. This difficulty may stem from the limited distinctiveness of the trajectory data, making it challenging to rely on rewards to determine the evolution moment to cluster the trajectories into limited categories.
Adut-Imoal
Adul:-2moal
Aieerage Retum
4000
PeraggeRe Retur
8000
1500
4750
-1000
Average
-1750
6000
-5000
0.8 24 3.2 Time Stops 141
4.0
0.0
1.0
3.2
Time Stops
Figure 4: Testing average return on glucose control environments over 6 random seeds.
8

---

Published as a conference paper at ICLR 2026
MuJoCo results. Following CEMRL, the non- stationarity in MuJoCo is simulated by dynam- ically adjusting parameters, such target ve- locity. However, these adjustments are rela- tively small, leading narrower distribution of non-stationary tasks. Consequently, these tasks tend to be less challenging compared to those in Meta-World. Besides, the lower state dimension Figure 5: Testing return on MuJoCo over 6 seeds.
Average Retur
400
WalkerkKal
NNv Mwerrop Return
10000
Cheata'> Dammping
250
7500
100
5000
-50
2500
220
0.0
0.0
1.2
24
3.0
0.0 0.2 0.4 0.8 0.0 1.0
Time Stops
Time Staps
of MuJoCo results in relatively simple relationships between the graph nodes, which explains why COREP's advantage over other baselines is not as significant as in Meta- World. In Fig. [ CEMRL demonstrates enhanced capability in clustering the task distribution, which facilitates precise classifi- cation of data in the replay buffer and effective extraction of task-specific representations. Since both TRIO and ScCBAD explicitly model the task evolution process, and TRIO additionally incorporates a recurrent encoder to capture and leverage history information, they converge faster than PEARL and RL... Signifieantly, WISDOM shows consistently rapid adaptability and enhanced final performance.
Ablation studies. We conduct ablation studies to validate the role of the wavelet representation network (Y' network) and its optimization objective, and the structure of the context encoder. In [()] the Y network provides significant gains, further indicating that the wavelet task repre- sentation reflects non-stationary trends. The AR loss accelerates convergence and improves final performance, and the wavelet TD loss stabilizes training and reduces variance. In contrast, the RNN encoder tends to forget changes over time and is susceptible to gradient vanishing, whereas the MLP encoder (WISDOM) shows greater stability. Since each in is generally multidimensional, each dimension of can be viewed as variate in the multivariate time series. In addition to perform- ing feature concatenation at each time step when performing DWT as WISDOM, we investigate a Variable- Wise Encoding (VWE) strategy which applies DWT to the sequences constructed from each variate independently. The VWE strategy leads our model to converge more slowly and yields inferior final results. We attribute this to the loss of cross- variate interaction modeling caused by disrupting the dependencies among variates. In the time series domain, this VWE strategy often necessitates additional structural components to re-establish inter-variate relationships.
1.0
0.8
Mltte-Side
1.0
0.8
Doos-Unlack
0.8
1.0
Buston-Press
Anerage Return
400
Wallarr Vall
0.4
0.6 Haviee
0.6
0.4
100
corD
0.2
18 0.2
-50
0.0
0.0
-200
0.64
1.2
Steps
140
Non-stationary Degree
16
(a) Ablation study of WISDOM.
(b) Impact of N-S degrees. (c) Impact of different RL.
Figure 6: Ablation study and evaluation of different non-stationary (N-S) degrees and RL backbones.
Adaptability analysis of different non-stationary degrees and RL backbones. The non-stationary d set in Meta- World, MuJoCo, and Type-1 Diahetes are 0.99, 0.97, and 0.7, respectively. In Fig. 6(b) most models' final performance shows a downward trend as non-stationarity increases. Contrastively, our WISDOM demonstrates remarkable and consistent adaptability, highlighting the more expressive wavelet task representation and robust policy. In Fig. [(C) WISDOM can quickly adapt and converge utilizing various RL algorithms. Due to limited exploration, DDPG often converges to a local optimum. For fair comparisons, all models employ SAC as the backbone.
Robustness to noise for WISDOM and baseline methods. We con- ducted experiments by injecting Gaussian noise (O. 1) into states to evaluate robustness. As illustrated in Fig. 7 all baselines exhibit slower
Button Press
1.0
0.8
Sectao
0.6
0.4
convergence and reduced performance under noisy conditions. In contrast,
0.2
our WISDOM maintains the highest success rate and rapid convergence. We credit this to the wavelet-based representation learning process, which not only suppresses noise but also retains fast-varying and resk-relevant signals more effectively, thereby improving the signal-to-noise ratio.
0.0
0.0
2.2
4,4
6.6
8.8
Time Sneps
Figure 7: Evaluation of
robustness to noise.
'Measured as (T (T-T7)) Th)/T E [0, 1), where 7' is the total period and T% denotes the mean of the stochastic period of an MDP Mas,- Larger values of degrees indicate severer non-stationarity.

---

Published as a conference paper at ICLR 2026
Case study. We visualize the changes of representations tracked by the Yo
Rowere d
Appixoraan
Docal cart
Changin May
network to validate its ability to cap- ture intrinsic non-stationary changes. As depicted in the yellow lines of Fig. [8 the velocity of the agent in In- verted Double Pendulum undergoes 7 changes over 440 consecutive time steps, generating a non-stationary se- quence MMwi. with varying state tran- sition dynamics.
2
100
200
400
130
200
430
Appnamral on
Uste Coet
182
250
360 400
180
250
400
130
200
420
Time Steps
Figure 8: The case study shows that the wavelet task repre- sentation 2 accurately reflect the true changes task M
the middle subfigures, the approximation coefficients u that represent low- frequency trends match the true underlying task changes and become smoother with deeper decom- position. In the right subfigures, the detail coefficients g that represent high-freeg fegh-freuencc details show sharper, more localized changes and may appear inverted due to the orthogonality of the initialized Haar filters. As shown in the left subfigure, when we combine the final approximation U2 with selected details, obtain wavelet task representation 2, which closely aligns with the true trend of the non-stationary task. Overall, this illustrates that the wavelet-based representation successfully captures the underlying structure of how tasks change over time.
CONCLUSION, LIMITATION AND FUTURE WORK
We introduced WISDOM, novel approach address the challenge of non-stationarity in RL by leveraging a learnable wavelet representation network to capture the trends of task evolution in the wavelet domain, facilitating flexible adaptation to complex non-stationary tasks with stochastic evolving periods. WISDOM captures more intrinsic evolving features of non-stationary task changes, learns to predict the evolving trend, and ensures efficient and improved policy learning. Experimental results on Meta- Meta-World, MuloCo, and Type-1 Diabetes benchmarks demonstrate its superb adaptability and performance, highlighting its robustness and efficiency in handling non- non-stationary tasks.
Although WISDOM is effective and easy to implement, it presents certain limitations. The compact- ness of task representations may be influenced by decomposition levels in the wavelet representation network, necessitating tuning appropriate decomposition levels based specific environments. Fu- ture directions include: 1) Developing adaptive mechanisms for dynamically adjusting decomposition levels; and 2) Exploring effective selection of detail coefficients for rapid adaptation.
ACKNOWLEDGMENT
The authors would like thank the anonymous reviewers for their valuable feedback. This work was partially supported by the National Natural Science Foundation of China (Grant No. 62276024), the Beijing Natural Science Foundation (Grant No. 4262066), the Fundamental Research Funds for the Central Universities, Jilin University (Grant No. 93K172025KO1), and the Fundamental Research Funds for the Central Universities (Grant No. 2025CX01010).
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure that our work is fully reproducible. All experiments are conducted on publicly available benchmarks, with complete descriptions of these benchmarks and their configurations provided in Section 5 and Appendix B Additional details regarding the tasks and baselines are available in Appendix B To support reimplementation, we present pseudocode for our methods in Appendix C The neural network architectures, experimental protocols, and hyper- parameters are thoroughly described in Appendix and Appendix [ For our theoretical results, all proofs are provided in Appendix A All the code required to train and evaluate the proposed methods has been spenesourced, ensuring full accessibility for the reproduction of our results.
ETHICS STATEMENT
We place strong emphasis adhering to ethical standards throughout this research. This study raises no notable ethical concerns, as all experiments were conducted within simulated and well-controlled environments. This research relies exclusively data generated from simulations, therefore, it does
10

---

Published as a conference paper at ICLR 2026
not raise privacy concerns involve experiments posing risks human subjects. Additionally, followed strict ethical standards throughout the development of our method, and remain committed to addressing any unintentional enrors or lapses in a timely manner.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, llya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In International Conference on Learning Representations (ICLR), 2018.
Lucas N Alegre, Ana LC Bazzan, and Brumo C da Silva. monimum-delay adaptation in non- stationary reinforcement learning via online high-confidence Slange-point detection. arXiv preprint arXiv:2/05.00452, 2021.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
Sumana Basu, Mark A Legault, Adriana Romero-Sorian and Doina Precup. On the challenges of using reinforcement learning in precision drug dosing: Delay and prolongedness of action effects. In AAAI Conference on Artificial Intelligence, 2023. URI / //api semant icscholar
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pp. 531-5400 PMLR, 2018
Zhenshan Bing, Lukas Knak, Long Cheng, Fabrice O Morin, Kai Huang, and Alois Knoll. Meta- reinforcement learning in nonstationary and nonparametric environments. IEEE Transactions Neural Networks and Learning Systems, 2023a.
Zhenshan Bing. David Lerch, Kai Huang, and Alois Knoll. Meta-reinforeement learning in non- stationary and dynamic environments. IEEE Transactions on Pattern Analysis & Machine Intelli- gence, (500)))3476-3499 2023b.
Vicente Bolós, Rafael Benítez, and Román Ferrer. new wavelet tool to quantify non-periodicity of non-stationary economic time series. Mathematics, 8(5):844, 2020.
Sidney Burrus, Ramesh A Gopinath, and Haitao Guo. Wavelets and wavelet transforms. rice university, houston edition, 98, 1998.
Xiaoyu Chen, Xiangming Zhu, Yufeng Zheng, Pushi Zhang, Li Zhao, Wenxue Cheng, Peng Cheng, Yongqiang Xiong, Tao Qin, Jianyu Chen, et al. An adaptive deep rl method for non-stationary environments with piecewise stable context. Advances in Neural Information Processing Systems, 35:35449-35461 2022.
James W Cooley and John W Tukey. An algorithm for the machine caleulation of complex fourier scries. Mathematics of computation, 19990)) 1965.
Bruno C Da Silva, Eduardo W Basso, Ana LC Bazzan, and Paulo M Engel. Dealing with non- stationary environments using context detection. Proceedings of the 23rd international confer- ence on Machine learning, pp. 217-224, 2006.
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02777 2016.
Fan Feng, Biwei Huang, Kun Zhang, and Sara Magliacane. Factored adaptation for non-stationary reinforcement learning. Advances in Neural Information Processing Systems, 35:31957-3199 31971, 2022.
11

---

Published as a conference paper at ICLR 2026
Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. Advances in Neural Information Processing Systems, 33:11588 11598, 2020.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, P. Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. ArXiv, abs/1812.005055, 2018. URL
CorpusID:55703665
Pouya Hamadanian, Arash Nasr-Esfahany, Malte Schwarzkopf, Siddartha Sen, and Mohammad Alizadeh. Online reinforcement learning in non-stationary context-driven environments. arXiv preprint arXiv:2302.00 02222, 2023.
Fan Kong, Yixin Zhang, and Yuanjin Zhang. Non-stationary response power spectrum determination of linear/non-ling -inerar systems endowed with fractional derivative elements via harmonic wavelet. Mechanical Systems and Signal Processing, 162: 108024, 2022.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. 3519-3529. PMIR, 2019.
Zihe Liu, Jie Lu, Guangquan Zhang, and Junyu Xuan. A behavior-awand approach for deep rein- forcement learning in non-stationary environments without known change points. arXiv preprint arXiv:2405. /42 14, 2024.
LIU Minhao, Ailing Zeng, LAI Qiuxia, Ruiyuan Gao, Min Li, Jing Qin, and Qiang Xu. T-wavenet: A tree-structured wavelet neural network for time series signal analysis. In International Conference on Learning Representations, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXhv:1312.5602, 2013.
Aaron van den Oord. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 2016.
Wenwen Pan, Haonan Shi, Zhou Zhao, Jieming Zhu, Xiuqiang He, Zhigeng Pan, Lianli Gao, Jun Yu, Fei Wu, and Qi Tian. Wnet: Audio-guided video object segmentation via wavelet-based cross-modal denoising networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Sumanta N Pattanaik and Kadi Bouatouch. Haar wavelet: A solution to global illumination with general surface properties. In Photorealistic Rendering Techniques, pp. 281-294. Springer, 1995.
Riccardo Poiani, Andrea Tirinzoni, and Marcello Restelli. Meta-reinforcemment learning by tracking task non-stationarity. In International Joint Conference Artificial Intelligence, 2021. URL :234
https://ap1 http:///ppp..semttttccecee www.ntttmttttiineee.... ar.
Robi Polikar et al. The wavelet tutorial, 1996.
Yu- Yang Qian, Peng Zhao, Yu-Jie Zhang, Masashi Sugiyama, and Zhi-Hua Zhou. Efficient non- stationary online learning by wavelets with applications online distribution shift adaptation. In Forty-first International Conference on Machine Learning, 2024.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient oof-po meta-reinforcement learning via probabilistic context variables. International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Hang Ren, Aivar Sootla, Taher Jafferjee, Junxiao Shen, Jun Wang, and Haitham Bou-Ammar. Reinforcement learning in presence of discrete markovian context evolution. arXiv preprint arXiv:2202.06557, 2022.
Robert A Rescorla. A theory of pavlovian conditioning: Variations in the effectiveness of rein- forcement and non-reinforcemmet. Classical conditioning, Current research and theory, 2:64-69, 1972.
12

---

Published as a conference paper at ICLR 2026
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference Learning Representations (ICLR), 2019.
Zohar Rimon, Tom Jurgenson, Orr Krupnik, Gilad Adler, and Aviv Tamar. Mamba: an effective world model approach for meta-reinforcemmnt learning. arXiv preprint wxXv::2403.08859, 2024.
Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniilidis, Igor Mordatch, and Sergey Levine Model- reinforcement learning via latent-space collocation. In International Conference on Machine Learning, pp. 9190-9201. PMLR, 2021.
Matthias Seeger. Gaussian processes for machine learning. International journal of neural systems, 14(02):69-106, 2004.
Claude Shannon. Communication in the presence of noise. Proceedings of the IRE, 37(1):10-21, 1949.
Mark J. Shensa. The discrete wavelet transform: wedding the trous and mallat algorithms. IEEE Trans. Signal Process., 40:2464 2482, 1992. URR.https://apg semant iommanticsoon icscholar
arg/CorpusID:: 97911992.
Mark J Shensa et al. The discrete wavelet transform: wedding the a trous and mallat algorithms. IEEE Transactions on signal processing, 40(10):2464 2482, 1992.
Jiaxin Shi, Ke Alexander Wang, and Emily Fox. Sequence modeling with multiresolution convo- lutional memory. International Conference on Machine Learning, pp. 31312-31327. PMLR, 2023.
Shagun Sodhani, Franziska Meier, Joelle Pineau, and Amy Zhang. Block contextual mdps for continual learning. In Learning for Dynamics and Control Conference, pp. 608 623. PMLR, 2022.
Jason Stock and Chuck Anderson. Trainable wavelet neural network for non non-stationary signals. arXiv preprint XXXx:2205.033555 2022.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. JEEE Transactions Neural Networks, 955) 1998.
Guy Tennenholtz, Nadav Merlis, Lior Shani, Martin Mladenov, and Craig Boutilier. Reinforcement learning with history dependent dynamic contexts. In International Conference on Machine Learning, pp. 34011-34053. PMLR, 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. URL prttps://appe www.nt
wwwp///rposoooor/eo 5DD0::223622
Jiashan Wan, Na Xia, Yutao Yin, Xulei Pan, Jin Hu, and Jun Yi. Tedformer: A transformer framework for non-stationary time series forecasting based on trend and change-poit -ointee--onttoeo detection. Neural Networks, pp. 106196, 2024.
Jingyuan Wang. Chen Yang, Xiaohan Jiang, and Junjie Wu. When: A wavelet-diw hybrid attention network for heterogeneous time series analysis. In Proceedingy of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 23612361222222 23733, 2023.
Alan S. Willsky. Multiresolution markov models for signal and image processing. Proc. JEEE, 90:1396-1455 2002. URL httt::/:/ap..smmmnicccco...... wwt.scchhh.aaaaooooouss..e.. 122692461
Annie Xie. James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity. International Conference Machine Learning (ICML), 2021.
Jinyu Xie. Simglucose v0. 2.1. Avaible: https://github. com//xx123/simglueen Accessed on: Jan-20-2020, 2018
13

---

Published as a conference paper at ICLR 2026
Zhirui Yang, Yulan Hu, Ouyang Sheng, Jingyu Liu, Shuqiang Wang, Xibo Ma, Wenhan Wang, Hanjing Su, and Yong Liu. Wavenet: Tackling non-stationary graph signals via graph spec- tral wavelets. In AAAI Conference on Artificial Intelligence, 2024. URL https://api semant cscho ar. org/ ID:
Ting Yao, Yingwei Pan, Yehao Li, Chong-Wan Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In European Conference on Computer Vision, pp. 328-345. Springer, 2022.
Mingxuan Ye, Yufei Kuang, Jie Wang, Yang Rui, Wengang Zhou, Houqiang Li, and Feng Wu. State sequences prediction via fourier transform for representation learning. Advances Neural Information Processing Systems, 36, 2024.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Yingchen Yu, Fangneng Zhan, Shijian Lu, Jianxiong Pan, Feiying Ma, Xuansong Xie, and Chunyan Miao. Wavefill: A savelet-based generation network for image inpainting In Proceedings of the IEEF/CVF international conference on computer vision, pp. 14114-14123, 2021.
Wanpeng Zhang, Yilin Li, Boyu Yang, and Zongqing Lu. Tackling non-stationarity in reinforcement learning via causal-origin representation. In Proceedings of the 41st International Conference Machine Learning, volume 235, pp. 59264-59282 PMLR, 2024a.
Xiangyu Zhang, Daijiao Liu, Hexin Liu, Qiquan Zhang, Hanyu Meng, Leibny Paola Garcia, Eng Siong Chng, and Lina Yao. Speaking in wavelet domain: A simple and efficient approach to speed up speech diffusion model. arXiv preprint arXiv:2402.10642, 2024b.
Xinyu Zhang, Wenjie Qiu, Yi-Chen Li, Lei Yuan, Chengxing Jía, Zongzhang Zhang, and Yang Yu. Debiased offline representation learning for fast online adaptation non-stationary dynamics. arXiv preprint arXiv:2402.11311 2024c.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning, pp. 77268-27286 PMLR, 2022.
A PROOF
A.1 1 PROOF OF THE DISCRETE WAVELET TRANSFORM
The Wavelet Transform (WT) Burrus et al. (1998) analyzes and represents the various frequency components of a signal utilizing a scalable and translatable mother wavelet function tb. Given time series xf), the continuous wavelet transform (CWT) extracts its frequency by performing the continuous convolution:
3( f a(1)bb (B-+((
The translation position d allows the & function to move along the signal æ(t) in the time domain, enabling analysis different time points. The scaling factor 3 controls the degree of stretching and compression of z function, which in turn affects its resolution properties and the range of extracted frequencies. Specificallyy, a smaller 33 corresponds to higher resolution and a smaller timescale, facilitating the capture of finer details, while a larger 3 corresponds lower resolution and larger timescale, suitable for analyzing overall trends. CWT typically analyzes signals at a fixed resolution, either fine coarse. To achieve multi-resolution analysis while simplifying computations, the discrete version of CWT, the Discrete Wavelet Transform (DWT) Shensa et al. (1992). is introduced decompose the signal layer by layer at different resolutions, formalized in Lemma
14

---

Published as a conference paper at ICLR 2026
Lemma DWT. Let the initial approximation coefficient uy be equal to input discrete time series r(n). By substituting a pair of low-pass filter yo and high-pass filter y1 for the mother wavelet function following iterative form is derived through discrete convolution operations:
(m) = - (
k),, (1)
i 14 -)),
k),
(10)
where m represents the decomposition number and & denotes the resolution size. The approximation coefficients um represent low frequenyy components, whereas the detail coefficients gm represent fiquecyy components.
Proof For the continuous time series x(t), t E [O,T), the coarsest approximation of x(t) is:
(1) - (n,o vb(1) - 1(0 < <),, Uo.0 = x<()dl,
(11)
where (b)) denotes the scaling function and the coefficiemt uo,0 is the average value x(() in this interval. The superscript implies the initial approximation of x(t). To obtain more accurate estimation of r(t), the approximation can be refined by halving the intervals into smaller parts:
(t) (2,) + 11,1 1), u1,0 -o /2 xtt)dt,
u1.1
JT z(t)dt.
(12)
Define the mother wavelet function as:
(t) A 2 (2 - k)
(13)
By repeating this procedure, x²() can be approximated with finer precision, given as:
(mm) (t) Σ Lmm ((),
Um
x(t))mm
(14)
KEZ
where m N represents the m-th interval bisection, represents series of resolutions, and indicates the convolution operation. After the m-th bisection of the original time interval, the scaling functions in the subspace Um & span (19m, k bsez) are capable of capturing local structures in ar() at timescale no longer than T/2"
Inspired by mulli-resolution analysis (MRA) Willssy 2002), discrete wavelet transform (DWT) Shensa al. (992)); Shi el al. (2023) introduces the orthogonal subspace G,, - span ((5m,k) ) U o where indicates orthogonal scaling functions Burrus et al. (998). UM can be decomposed into the orthogonal sum of the lowest resolution subspace Uo and a series of orthogonal complement G:
UM -M @ GM-1 Uo Go ..... GM-2 A GM-1-
(15)
Accordingly, x(l) can be represented by series of different resolutions:
M-1
x (1) (t) +
Σ Σ gm,kkm.. (1),
(16)
TR=
KEN
where u and g are referred to as the approximation coefficient and detail coefficient, respectively. Therefore, for discrete time series z(n), we can rewrite Eq. the following iterated form:
Fm
Σ
Um 1 (n)
(17)
Σ ( + Σ 8m-2()))0m22(m
where n represents the sequence length, and (n) and 5(n) the discrete forms of vb(1) and 5(1), respectively. As result, by performing the discrete convolution operation (+), the discrete wavelet transform can be obtained as follows:
Um (n) 11 4/m (n)
Σ N n= Um (n)
E K 1k00 k))
(18)
Σ
K -= Mo
K), and similarly.
8m (m) = 1mm-1 * 8m (n)
= Σ
191 Um (2n ( k):
which concludes the proof of lemmal)
15

---

Published as a conference paper at ICLR 2026
Non-stationary RL is defined the time-evolving task distribution ((T) where agent interacts with a sequence of Markov Decision Processes (MDPs) Sutton Barto (1998)
M ,M. The evolution of these MDPs determined by history-dependent stochas- tie process p. i.e., wh+1 ~ p (Wh-1 |w,,, w1...,......44 wa), where w is a task ID that regulates the properties of different MDPs. Each MDP Ma represented by a tuple (S, Pw: Rw: 7: P((s)), in which S
.May.
denotes the state space, A the action space, P(s'|s, a) the transition dynamics, R(s, a, s') the reward function, (5)) the initial state distribution, and y E [0, 1) the discount factor.
A.2 PROOF OF THE CONVERGENCE OF THE WAVELET TD LOSS
Theorem Let W denote the set of all functions W : S A + CD that map from the time domain to the wavelet domain. The wavelet update operator F W W, defined as
FW( Zt + TE, (W((())))
(19)
contraction mapping, where be and Ze41 represents the current and next task representation sequence, respectively, and I denotes the discount factor in diagonal matrix form.
Proof The norm on W is defined as [[ == sup max [[[(z))]<l)) where K. denotes the =EB REK.
sequence length of ze. For any W1, W22 E W, we have
| W2 - F" W2llw = sup max [[2 + + [[W (z<+1)lk]
ZEB KEA
|ze|K EE [W₂ [W2l(z+1]]lkd Ill
S y max sup KEK.
| 11 [[ (2c+1]kx [W2 (241)
(20)
S 7 max sup KEK: xcB
[| (z++1) W2(z(+1)kllD
7. || W2llw:
where I' = diag(T, " xxxx and y E [0, 1). proving that F is a contraction mapping.
A.3 PROOF OF THE POLICY PERFORMANCE DISTINCTION VIA WAVELET DOMAIN FEATURES
Lemma Following the previous definition of state distribution AAchimm et al. (2017) we define the discounted task representation distribution as follows:
00
(=)=)
Σ
Pwn ( (w0
(21)
The neural network maps the trajectory to the task representation, denoted as z. Then the expected discounted total reward under policy can be expressed
J
F Eun.ropTT RRwa (T)]
R RT(T)P (1i)))nnn
Rs
P wy (
t=0
h=0
RJ (z)
Pun (1/7)
2m7((T)
1=0
K=
Rs
1-7
sny(s,a,e")
E, Eg-D" amst(1), [R(x)]
(22)
7
s' C/PP(( P(-,,,a,))
16

---

Published as a conference paper at ICLR 2026
Theorem 2 Suppose that the reward function R(2) be expanded into Bth-degree Taylor series for E RD. then for any two policies Tı and T2. their performance difference is bounded as:
1Ja, J VD
B
(RI((())
WWh (dy. BB) W (dy. Ba)) (23) B,ER,
max 1S45D
sup
b!
where W (d. denotes the wavelet transform of the bth power of the task representation se- quence bb²) [20, F1y...;2Taa (b) for any integer E [L, B), and Jx represents the average policy performance over the entire lask space within single episode.
Proof 3 First, the reward function can be written R(2)=En R (0)
bi 2d based on the Taylor
series expansion Ye et al. 2024). According to lemma 2 for any integer E [1, B}, we have
1
1am1 Jxzl
R (2)
R (2) d2
|
(0)
(>* (2) 2 D' Dz
$
B
(
(0)
E
[2]]
E
[>a]
(24)
b!
D
Since the inverse wavelet transform Burrus et al. representation sequence z (b),, ) we have
1998) of W (d, B) is the reconstructed task
('=ak) dg Bg
E Ze
W (dg Say0
dddd3..
4q 1,2,..., (25)
where T is the dual function form of the mother wavelet function tb and has the normalized orthogonal property. The dimensions of translation position d and scaling factor B are identical to those of 2.
Then we have the following derivation:
E DerDi1
E
bb 24g
<
1-
W(()
(dg.
(dg.
-WDb 8q) 1 (1-dg Bg
(1
-dd
dd
D
JU
1/00 /
("=
t (t-dy dg
W(b (dg. (q)
-W() (dy: Aq))
dd
sup 2,
Bg
sup d, 2,
w (dg. (dg. 3q) w (dy. Bq)
(26)
Then the policy performance difference bound is derived follows:
[kx, Jual 1
VD
R
(O)
Wilh (dg, Ba) w (dg (dg.,8))
sup
1505 d.ER,A,20
6=1
which concludes the proof of Theorem
2
A.4 PROOF OF THE POLICY IMPROVEMENT WITH WAVELET TASK REPRESENTATIONS
Suppose that the function class for the contextual policy æ of WISDOM defined as TWISDOM - {N,, s.t. 30 with r-|ss, 2) = fe (s, (e,e,()))) Vs). where fo is neural network characterizing the policy conditioned on top the context encoder en and the wavelet representation network Yo-
Assumption (1.1) Access to the true history policy: Th = The (1.2) The context encoder on and the wavelet representation network Yg allows to represent the estimate of the history policy : (als, (C)) = Fú (a,a,, 2) E IIWISDOM (1.3) Access to the true performance of policies: ( = (T) for all policies æ. (1.4) The non-stationary RL algorithm performs perfect optimization on top of en and Ya: JWISDOM maxxellwoom (
17

---

Published as a conference paper at ICLR 2026
Theorem WISDOM returns policy T(-|s, conditioned the wavelet task representation that improves upon its intermediate policy produced during policy iteration The satisfying JWISDOM 2 JRn under Assumption 1.1-1.4, where JWISDOM denotes the final policy performance of WISDOM.
Proof 4
WISDDO = J
arg max ((T]
(Assumption 1.4)
FEHHWIDOO
max
(Assumption 1.3)
# HIwIDOM
J (TT))
(Assumption 1.2) (Assumption 1.1)
2 J (TT)
which concludes the proof of Theorem
B EXPERIMENTAL DETAILS
B. BASELINES
We compared WISDOM with seven prevailing and competitive baselines. Initially, SAC, a standard RL method, serves as the backbone validate the importance task inference. Conversely, PEARL and RL?, which are classical meta-RL. methods, perform task inference using an encoder, emphasizing the significance of modeling the task evolution process in ota--sationary environments. Moreover, the last four baselines (CEMRL, SeCBAD, TRIO, and COREP) are specifically designed to address RL tasks non-stationary settings, further substantiating the advantages of modeling and predicting the task evolution process through wavelet transform in WISDOM. Their introductions
follows:
(1) SAC Haarnoja et al. 2018 a standard model-fres off-policy reinforcement learning method that aims to maximize both the expected return and the entropy of the policy;
(2) PEARL Rakelly et al. (2019) a classical context-based meta-RL method that enhances sample efficiency and policy performance by disentangling task inference and control, employing online probabilistic filtering of latent task variables for effective adaptation to new tasks from limited experience;
(3) RL. Duan et al. (2016) employs RNNs to encode states for meta-learning, storing learned prior knowledge in the hidden states. This prior knowledge is then applied across multiple tasks, and RL algorithms are used to train the weights of the RNN, addressing the challenge of rapid learning;
(4) CEMRL Bing al. (023b) leverages Gaussian Mixture Model to represent non-stationary task distributions with cluster structure and learns a more compact latent space through reconstructing tasks;
(5) SeCBAD Chen et al. (2022) perceives and adapts to new evolving tasks through reward functions, leading to policy that can adapt to rapid variations;
(6) TRIO Poiani et al.
2021 meta-trains a variational module for inferring distributions over latent context and leverages Gaussian Process (GP) model the task evolution process;
(7) Zhang et al. (2024a) utilizes a dual graph structure to retrospect causal origin of non- stationarity, with core graph emphasizing stable representation and general graph compensating for overlooked edge information.
For fair comparison. WISDOM and all baseline models, except TRIO and RL which emphasize the use of a recurrent encoder, employed the same MLP network architecture for their context encoders.
B.2 EVALUATION METRICS
All models were trained with equal time steps, and subsequent policy evaluations were performed with the same number of time steps. For evaluating the policy's performance Meta-World. the average success rate was computed across six random seeds. On MuJoCo and Type-1 Diabetes control, the average return computed across six random seeds.
18

---

Published as a conference paper at ICLR 2026
B.3 NON-STATIONARY EXPERIMENTAL SETTINGS
We evaluated our proposed model WISDOM on various non-stationary tasks with uncertain evolution periods, using three benchmarks: Meta-World, Type-1 Diabetes and MuJoCo. Then we describe the details of these three environments.
Train tasks
Testtasks
Pick and place with goalg1 Pick and place with goal
Pick and place with goal In Fick and place with unseen goel
Figure Visualization of MLL evaluation protocol.
Meta-World Meta- World is an open-source simulated benchmark for meta-RL and multi-task learning, consisting 50 distinct robotic manipulation tasks. As shown in Fig. [ we experiment with the Meta-Learning 1 (ML1) evaluation protocol, which evaluates meta-RL. algorithms' few-shot adaptation goal variation within one task. ML1 uses single Meta-Word World tasks, with the meta-training tasks corresponding to 50 random initial object and goal positions, and meta-testing conducted on 10 held-out positions. The non-stationarity in Meta- World is evidenced by the continuous fluctuation of the target position for the robotic arm's 's movement, which is correlated with the reward function. We selected different types of tasks, and the detailed descriptions are listed in Table 2
Table 2: Meta- World task descriptions.
Task
Description
Button- Bueson--Pesss
Randomize button positions and press a button.
Door-Close
Randomize dour positions close door with revolving joint,
Door-Lock
Randomize door positions and lock the door by rotating the lock clockwise.
Randomize door positions and unlock the door by rotating the lock counter-clockwise. Randomize faucet positions and rotate the faucet clockwise.
Faucet- Close
Handle-Press
Randomize handle positions and press handle
Slie-SSlic
Randomize the plate and cabinct positions and slide plate into cabinet.
Plate
Slide Plate-Slide-
Back
Randomize positions for both the plate and cabinct retrieve a plate from the cabinet.
Type-1 Diabetes In addition to the MuJoCo and Meta-World physical control tasks, we also evaluate our proposed approach on more challenging real-world task: Realistic Type-1 Diabetes, which aims to control an insulin pump to regulate the blood glucose level of patient. Each step of an episode corresponds to a minute in an in-silico patient's 's body, and each episode consists of 1440 day. When patient consumes food, the blood glucose level will increase. Extremely high or low blood glucose levels can lead to life-threatening conditions such as hyperglycemia and hypoglycemia, respectively. Therefore, insulin dosage required mitigate the risks associated with these conditions.
time steps, corresponding
In the glucose control task, the states hidden patient parameters, the observations continuous blood glucose readings, the actions consist of discrete insulin dosages within the range [0, 5]. and the transition dynamics are specific to the patient but unknown to the agent. Following the previous work Basu et al. (2023), we design a biologically inspired custom zone reward that incentivizes the time spent the target zone and penalizes hyperglycemia and hypoglycemia:
20
8, 70 or s; 200 (episode termination)
< 100 or 8; 9t-1 < 0.5 (hypoglycemia)
(8(8 =
8, >>>>> 150 8; 81-1 > 0.5 (hypergiycemia)
(27)
50
100 < < 150 (target blood glucose)
where r, encourages the agent to maintain the target vital statics within healthy range.
We induced non-stationarity by oscillating the body parameters, such the consumption amount of total glucose. Specifically, we simulate non-stationary fluctuations in blood glucose levels by varying the daily intake of lunch (1 meal) both lunch and dinner (2 meals) in both adolescent and
19

---

Published as a conference paper at ICLR 2026
adult populations. Specifically, the meal plan (meal time and size) is set follows: 7:00 AM: 45 (breakfast), 12:00 AM: 70 (launch), 16:00 PM: 15 (snack1), 18:00 PM: 80 (dinner), 23:00 PM: 10 (snack). The meal plan for adults is exactly the same as that for adolescents. The range for the size of the adults' meals for lunch or lunch and dinner is 60-80, while the range for adolescents for lunch lunch and dinner is 50-80. The goal of this system is responsibly update the doctor's initial prescription, ensuring that treatment continually improves.
MuJoCo We also adopted 4 classic MuJoCo control tasks for comparison. In the setting of non- stationary RL, we primarily considered two different types of MuJoCo evolution tasks: changes in reward functions and dynamics. The environment details and the number of tasks for meta-training and meta-testing in different tasks are shown in Table
Changes in reward functions. the Hopper-Vel and Walker- Vel tasks, the target velocity of the agent is sampled from the uniform distribution U [0.5, 3.0] every 60 + 20 time steps to simulate sutsstaiionarit.
Changes in dynamics. In Cheetah-Damping, the damping parameters are sampled from the set [0.85, 0.9, 0.95, 1.0]. In Walker-Rand-Parane the physical parameters of the agent, such as body mass, damping, and friction, are randomized. The variation in these physical parameters 60 + 20 time steps results in non-stationary environmental dynamics.
Table 3: Classical MuJoCo details.
Task
Observation dim
Action dim
Training tasks
Test tasks
Cheetah-
17
6
60
10
Hopper- Vel
Walker-Rand Kand-Params
17
100
30
Walker-Vel
17
100
Hyperparameters
The hyperparameters utilized in the experiments are outlined in Table 4
Table Hyperparameters.
Hyperparameter
Value
Encoder training steps
200
RL layer size
300
Target entropy factor
1.0
Learning rate
3e-4
Dims. of task representation z
Coef. of soft update
5e-3
Replay buffer size
10,000,000
Batch size of the policy
MuJoCo & Type-1 Diabetes: 256, Meta-World: 512
Batch size the encoder Maximum trajectory length
MuJoCo & Type-1
Diabetes: 256, Meta-World: 512
MuJoCo & Type-1
Diabetes: 800, Meta-Workde 200
Evaluation trajectories Policy training steps Training tasks per episode Initial trainsitions Trainsitions per episode Decomposition level M
MuloCo & Type-1 Diabetes: 2, Meta-Worlds
MuJoCo: 2,000,
Mera-World: 4,000, Type-1 Diabetes: 200
MuloCo (Cheetah: 20), Type-1 Diabetes: 25, Meta-World: 50
MuJoCo & Meta-Worid: 200. Type-1 Diabetes: 400
MuJoCo: 800, Meta-World: 600, Type-1 Diabetes: 200
MuJoCo (Walker-Ranl Baikeer-Ran-PParams 3). Type-1 Diabetes:
Meta- Meta-World (Button- Press & Handle-Press: 5):
Coef. wavelet TD loss ay
MuJoCo (Hopper-Ved Vel: 0.5): 0.9
Type-1 1 Diabetes Adolescent--meen 0.5): 0.1
Meta- World (Faucet-Cloo
Door-Close: 0.9,
Door-Loce 0.8): 0.1
20

---

Published as a conference paper at ICLR 2026
C
PSEUDOCODE
Algorithm 1 WISDOM algorithm
Input: training tasks Fraain from P P(T), replay buffer B, context encoder (z(())
wavelet representation network y₂ 2). W network Wa (ulul, e), entropy term H. contextual policy (a(a|s, 2), critic (s 2)
Collecting Training Data
for training task Ttrain do
Generate a task representation sequence z e,,((CC Roll-out policy To (als, z) and add transitions ($j, aj 84 ri) fa..g to B
end for
Training Context Encoder and Wavelet Representation Network
for each context encoder training step do
Sample c (s,, a,, s r1) B for context encoder
Generate a task representation z ~ (=(c))
Obtain wavelet task representation sequence: 2 Yς(z)
Train context encoder e f- 7/ - X, J, (Eq.
Train wavelet representation network Ys: Φ + Ø
A Js (Eq.
Train W network We: yp & X2 V Jo (Eq. 5
Train target W network W μ + op + (1 F
end for
Training Policy
for each policy training step do
Train contextual policy ma: +- 0 X V.Jo (Eq. 7
Obtain a wavelet task representation sequence: à Yo(z)
Train contextual critic Qui 1 ut \o V Ju., for [1, (Eq.
Train temperature coefficient a: a + a - Aa V.Bi-a mo (a s, 8) - aH]
Train target contextual critic Qç: G <- ou + (1 oi, for l E (1,2)
end for
Testing on Unseen Tasks
Initialize transitions et O
Sample test tasks Frest from P(T) for g = 1,..., G do
Generate task representation
(2(gg)
Roll out policy (a 2) to generate transitions DJ r5)
Store the transitions: ct ct UD
end for
D IMPLEMENTATION DETAILS
Model Architecture All components of WISDOM are implemented as MLPs, We report the total
number of learnable parameters for our model initialized for the Walker Vel task (S E S!7 A E A9). We summarize the architecture and model size WISDOM using Py Torchh-like notation Alg 2
Training Details We evaluate our proposed method and other baselines on the NVIDIA GeForce RTX 3090 GPU. And PyTorch-like pseudo-code for training model is illustrated in Alg. 3
21

---

Published as a conference paper at ICLR 2026
Computational Cost WISDOM adds only modest time cost. As depicted in Table WISDOM stands out with the lowest memory usage (Mb) among all baselines and a shorter runtime per episode (s) compared competitive NSRL methods such SeCBAD and COREP. WISDOM is also more efficient than the classical meta-RI. method PEARI., despite PEARL not modeling task changes.
Algorithm 2 WISDOM Network Structure Pytorch-like like Pseudocode
Encoder parameters: 90, 810 Q1, 62, target Q1, target Q2 paramsters: 189, 602
Policy parameters: 191, Alpha network parameters: 251
Rncoder(
(EcO): (in
u_dim a did + r_dim, out Featuueeee,,000 bias-True) (fel): Linear (i__feerrree out cut_foatures=20e bias=True)
(Ec2):
Linear (int fut__esturee=200, biag-Truc)
( Linear featureeee,,,,0, out fest_eesteee=1 bias-True)
21, 92, target Q1, target Q2
(fe0):
(o_dii dim latent a_dim, ors_features-300, bias-True) Linear (in_featurer=300, out fostures=== bias=frue) (fc2): Linear (in
(fel): :
out_festures-300, bias-Irue!
(last_fa) Fo) Linearlin (ieerrrineet out fettteesurese,, bias=Trua)
Policy(
(EcD): Linear Feature==2,, out features=300, biss-True)
(fel): (in (in_featuree-300 out out_features-300 bias-Irue)
(fc2): Linear (infarrin_fetturee-3303
out_festurrs-300, bias-True!
(last_fa); Linear ciarrin_feee fon_uuuures--000 out festures= aias-Trus)
(last_ fo____t_log__tde (in (tn_ out_featurea-6, bias-True]
Alpha network
(FcO): : Lincar (in, foatures-S, fut__featare=500, bias-True)
Laat__fe); Linear (in_ features=50, out_feeaures=,, bias-True]
Algorithm 3 WISDOM Training Pytorch-like Pseudocode
def (self):
collecting initial samples ... if aelf. init.initall.n trtattaaaiioon > savsteev_steps +- cedf.ccilett.Gsaaeeel....ae
self. int.iiitts__ttaassttoo
for epoch An gt.timed_ for (range (aeit.nun_eee hewoche_eeesss save_itrs-Truel:
4 1. Collect data with rollout coordinator collection_tase self.env_stepe +- sel..
ap, ra.dom. por (self (trai__ttssss)
taa__ttraa__aaka
self.num_transitione
# #22 Replay buffer stats seif.rreieypbuffeeeettt
welf.reeiay eel...eg..pp.fffe..eee.tttt
# 3. Train context encoder and wavelet sepresentation netwerk wew.costtuutionntreaiiee tww.aere..aiaiiieee...... wwwe:/ffimmrecossstrroooooeaeeee
# 4. Train policy with data from the replay buffer
temp, sac_stats
Evalustion
evai_outpat self.evaluate (10G, 'train', collection self. eval outp_t self.evaluste (LOS, 'test', average_test__em std_test_rewaat eval_output
self.eval
Table 5: Computational cost comparison among different methods.
Task
WISDOM
PEARL
SeCBAD
RL
TRIO
COREP
CEMRL
Button-Press
432M
455M
457M
459M
499M
695M
741M
251.32s
255.968
409.86s
185.84s
170.66s
1207.51s
171.44s
Walker- Vel
406M
421M
425M
449M
443M
501M
787M
175.34s
183.21s
285.37s
80.12s
108.05s
348.94s
327.98s
22

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EXPERIMENT RESULTS
E.1 ADDITIONAL ABLATION RESULTS
1.0
Plate-Siide
Average Success Rate
Door-Unlock
Average Success
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.00
0.25
0,50
0.75
1.00
0.00
0.32
1.28 Time Steps
0.64
0.96
Time Steps
leb
Figure 10: Ablation the design of the wavelet representation network.
E.2 EVALUATION RESULTS ACROSS DIVERSE NON-STATIONARY DEGREFS
SAC
RL?
PEARL
CEMRL
1.0
1.0
0.8
0.4
0.6
0.6
0.1
0.6
fverage
0.2
039
628
Avcrage
0.1
0.50
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Time Steps
Steps
Steps
Time Steps
1.0
0.15
0.8
0.8
0.8
0.6
045
0.05
0.4
0.2
0.4
0.4
0.00
0.0
0.0
0.0
0.2
1.0
0.0
0.2
0.4
0.6
0.8
3.0
0.2
0.4
0.6
0.8
1.0
0,2
Time Steps
Time Steps
Time Steps
Time Steps
Figure 11: Testing average performance on button-press with varying non-stationary degrees.
E.3
PERFORMANCE DISPARITIES OF META-RL IN STATIONARY AND NON-STATIONARY
TASKS
Average Success Rate
1.0
Button-Press
1.0
Door-Unlock
Average Success Rate
1.0
Handle-Prees Press
Non Stationary
Non Stattonary
Statienary
0.6
0.6
0.6
0.6
0.2
0.2
Non-Stationary Stationary
0.0
0.0
0.0
0.2
0.4
0.8
1.0
0.00
0.35
0.70
1.05
1.40
0.00
0.35
0.70
1.05
1.40 166
Time Steps
366
Time Steps
Time Steps
Figure 12: Testing performance of the classical meta-RL algorithm PEARL Rakelly al. (209 in stationary and non-stationary environments on Keta-World] Yu et al. 2020)
E.4 SENSITIVITY ANALYSIS OF WAVELET DECOMPOSITION LEVELS AND WAVELET TD LOSS COEIFICIENTS
An appropriate setting of wavelet decomposition levels M allows for more precise separation of frequency components corresponding to distinct evolutionary periods, thereby facilitating the extraction of more critical features. In Fig. 13 (left), increasing M moderately can filter out task- irrelevant information and improve adaptability. However, excessive decomposition may lose vital low-free fre.uencynnyy components, thereby hindering the learning of non-stationary trends. In Fig. B (right), ay typically influences the convergence speed. When wavelet TD loss or AR loss predominates, the convergence accelerates; even so, the final performance remains insensitive oy.
23

---

Published as a conference paper at ICLR 2026
Walker-Rand-Params
Average Success Rate
Door-Lock
Average Return
5000
K=2
1.0
4000
0.8
3000
Km5
0.6
0--01
2000
0.4
0:-0.3
0,50.5
1000
0.2
0.0
-0-0.9
3
0.00
0.24
0.48 0.72
0.96
1.20
Time Steps
Time Steps
lef
Figure 13: Sensitivity to wavelet decomposition levels M (left) and TD loss coefficients ay (right).
E.5 ADDITIONAL EXPERIMENTAL RESULTS ON MuJoCo BENCHMARK
Hopper-Vel
Walker-Ran Params
Average Return
2000
PEARL
Average Return
5000
1500
T3O
4000
SeCBAD
CEMRL
3000
1000
WEDDO
500
1000
0.0 1.2
3.6 4.8
1.2 2.4 3.6
Time Steps
Time Steps
Figure 14: Testing average return on MuJoCo over 6 random seeds.
E.6 ADDITIONAL EXPERIMENTAL RESULTS ON GLUCOSE CONTROL BENCHMARK
Adolescent-Ime
Adolescent
5000
4000
Average Return
Average Return
3250
2250
1500
500
-250
-2250
-2000
-3000
0.0
0.8 1.6 2.4
3.2 4.0
0.0
0.8
3.2 4.0 Time Steps
1.6 2.4
Time Steps 106
1n4
Figure 15: Testing average return on 2 glucose control environments over 6 random seeds.
E.7 EVALUATION ON MORE DIFFICULT TASK
We conducted additional experiments more complex Reach, Door-Open, and Hammer to evaluate the performance WISDOM. As illustrated in Fig. [6] WISDOM consistently outperforms popular non-stationary baselines these tations can effectively capture the underlying trends of complex non-stationary changes and remain robust across broader range of task difficulties.
challenging tasks, indicating that wavelet-based task represen-
E.8 SENSITIVITY ANALYSIS OF WAVELET BASIS FUNCTIONS
We evaluated the sensitivity of WISDOM to the choice of wavelet basis by conducting additional experiments on Button- Press using three representative wavelet bases (Biorl.1, .1, Sym2, and Haar). As illustrated in Fig. [17 different bases lead to slight differences in convergence speed, but they have minimal impact on the final performance. In our wavelet representation network, the wavelet basis functions serve only as initialization to approximate traditional wavelet behavior. Because the filters are learnable, the model progressively reduces reliance on the initial basis and is able to extract more expressive and task-adaptive features from non-stationary task sequences without relying on hand-crafted designs or a fixed wavelet basis. This explains why different bases influence convergence only in the early stages while having minimal impact on the final performance.
24

---

Published as a conference paper at ICLR 2026
Average Success Rate
Door-Open
1.0
Hammer
Average Success Rate
0.4
VISDOM
TRUO
0.3
0.6
0.6
COREP
SACIAD
0.2
0.4
CEMAIL
0.2
0.2
0.1
0.0
0
4
8 1n5
0.0
0.2
0.4
0.6
0.8
1.0
2
6 1e.
Steps
Steps
1n6
Steps
Figure 16: Testing success rate on more difficult tasks over 3 random seeds.
Among the three wavelet bases, Sym2 converges the fastest, followed by Haar, whereas Biorl. ..3 converges the slowest. This can be attributed to the strict orthogonality of the Haar wavelet, which reduces feature redundancy and enhances learning efficiency, leading to faster convergence compared the biorthogonal Bior1.3. The longer filters of Bior1.3 incur greater computational cost and slower convergence. In contrast, the Sym2 wavelet, with its smooth and near- Gaar-Gans shape, more effectively captures stable trends and suppresses high-frequency noise, thereby accelerating convergence and yielding superior final performance. Overall, the comparable results across different bases demonstrate that WISDOM largely insensitive to the choice of wavelet basis functions.
Average Success Rate
Button-Press
1.0
0.8
0.6
0.4
SymZ Bior1.3 Haar
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0 1e6
Time Steps
Figure 17: Sensitivity analysis of different wavelet basis functions.
E.9 EVALUATION ON TASK WITH SPARSE REWARD
Following the sparse-reward setup LatCo Rybkin et 2021). 1. the agent receives reward only when it successfully completes the task. As shown in Fig. 18 WISDOM consistently outperforms the baselines, and the wavelet TD update further enhances both adaptability and stability. This demonstrates that our wavelet representation can handle non-stationary tasks with sparse reward signals, enabling rapid adaptation. Importantly, the explicit wavelet TD update helps the learned representations become denser and more informative, while also highlighting low-reward yet critical features. Thus, WISDOM also exhibits exceptional adaptation efficiency on tasks with sparse rewards.
Average Success Rate
Sparse-Door-Close
Average Success Rate
Sparse-Handle-Press
1.0
VISDOM
1.0
WISDOM
0.8
Wavelet
0.8
Wavelet
TRIO
0.6
SeCBAD
0.6
SeCBAD
0.4
CEMRL
0.4
CEMAL
0.2
0.2
0.0
0.0
0.0
0.6
1.2
1.8
2.4
3.0
0.0
0.7
1.4
2.1
2.8
Time Steps
Time Steps
1a5
Figure 18: Testing success rate sparse tasks over random seeds.
25

---

Published as a conference paper at ICLR 2026
E.10 ADDITIONAL EVALUATION TASK WITH TRANSITION DISTRIBUTION DRIFTS
We conducted an additional experiment on Walker-Mass to further evaluate the wavelet TD update. As depicted Fig. 19 WISDOM consistently outperforms the baselines, and the wavelet TD update further enhances both adaptability and stability. This demonstrates that our wavelet representation can handle non-stationary changes not only in reward functions but also in state transition dynamics, enabling rapid adaptation.
Walker-Mass
4000
Average Return
3000
WISDOM
2000
TRIO
1000
CORED
CENRL
0
wio Wevelet TD
0
1
2
Time Steps
Figure 19: Testing average return Walker-Mass with transition distribution drifts.
E.11 ADDITIONAL EVALUATION OF WISDOM'S PLUG-IN CAPABILITY
Our wavelet task representation can be viewed as a plug-in in module and can connected with non-stationary policy methods. We conducted additional experiments validate that WISDOM can be integrated with online/offine Non-Stationary RL (NSRL) methods as well as model-based RL.
The testing curves on unseen tasks are provided in Fig.
20
Average Success Rate
Handle-Press
Average Success Rate
Door-Close
0.8
0.8
0.6
MAMEA+WISDOM
MAMBA-WISDOM
0.4
LCPO
0.4
LCPO
LCP0+WI5DOM
LCPO-WISDOM
0.2
DORA
0.2
DORA
DORA-WISDOM
0.0
DORA-WISDOM
0
0.0
0.2
0.4
0.6
0.8
10
1.2
Env Steps
165
Env Steps
167
Figure 20: Testing success rate Non-stationary Handle Press and Door-Close tasks for validating WISDOM's plug-in capability over random seeds.
LCPO Hamadanian al. (023) is an online non-stationary policy learning method that does not incorporate additional representation learning; it mitigates catastrophic forgetting by constraining policy optimization using experience samples that lie outside the current context distribution. Building on LCPO, our wavelet- based task representation further accelerates adaptation and improves the final success rate. This further validates the generality and effectiveness our wavelet representation and shows that it can facilitate policy improvement, consistent with our analysis in Theorem 3
DORA Zhang et al. (2024c) is an offline NSRL. method that incorporates mutual information to learn task representations capable of differentiating between changes in the environment and the behavior policy. By filtering out spurious high- nigh-frequency components and learning stable multi-scale features, our wavelet- based representations reduce misleading correlations in the data and mitigate the distribution shift, effectively preventing policy performance degradation, as observed in CooreClose.
Using the model-based meta-RL method MAMBA Rimon et al. (2024), which is built on world model, we show that incorporating our wavelet representation significantly improves adaptation efficiency non-stationary tasks. The potential reason may be that the wavelet representation
26

---

Published as a conference paper at ICLR 2026
explicitly separates the non-stationary changing patterns, making it easier for the world model to capture the true environmental dynamics and thereby improve its prediction accuracy.
E.12 ADDITIONAL EVALUATION OF WISDOM'S SS LEARNED REPRESENTATIONS
We conducted experiments using Centered Kernel Alignment (CKA) Kornblith al. 2019) and Mutual Information Neural Estimation (MINE) Belghazi et al. (2018) metrics to further assess the quality of the learned representations. Using the Inverted Double Pendulum as an example, measured the mutual information between & and MMo... with MINE. As shown in Fig. (1 as training progresses, the mutual information (polyline) steadily increases alongside the episode return (bar chart), demonstrating clear positive correlation between their performances. This indicates that the wavelet representation progressively captures the underlying dynamics the non-stationary tasks and thereby facilitates policy improvement.
In our framework, we utilized two different convolutional layers to learn the approximation and detail coefficients, respectively, followed by the same final linear layer. We computed the representation similarity across layers for the CKA analysis. The heatmaps in both Fig. 21(b) and Fig. [ reveal a clear hierarchical structure, with similarity decreasing layer distance increases, showing that the model does not collapse into identical mappings. This trend is consistent with observations from the CKA literature Kornblith et (2019), where early-layer similarity in well-trained neural networks tends to saturate more easily than in deeper layers.
Inverted Double Ponduum
Mutuel illorsation
Retarn
1:26
Leer'
8.75
6:2
$8500
Cokocce tatum
0.05
Leaa2
0.33
1.30
EX3
0.91
3.83
0.25
4002
0075
8.22
Leven
825
20000
Train
33003
3000
4000
(a) MINE analysis
(b) CKA of approximation coeffi-
(c) CKA of detail coefficients
cients
Figure 21: Qualitative analysis of the wavelet task representations.
27


Reviews:
[{"review_id": "T12tQs0MzB", "reviewer": "Reviewer_hSXg", "summary": "This paper introduces WISDOM, a context-based non-stationary RL that models sequences of latent task representations in the wavelet domain. A learnable wavelet representation network, composed of two dilated causal convolution layers functioning as adaptive low- and high-pass filters, decomposes context embeddings into approximation and detail coefficients, selectively downsamples detail terms, and reconstructs an enhanced representation used by the policy and critic. \nTraining jointly applies a contraction-guaranteed wavelet TD operator and an autoregressive likelihood objective. The theoretical analysis shows that wavelet-domain features can bound policy performance differences and that reconstructed representations improve contextual policy learning under certain assumptions.", "questions": "- There is an interesting sharp performance increase pattern around 4e5 time steps in Figure 3 (Meta-World experiments) for WISDOM (except Plate-Slide-Back). Likewise, in Figure 5 (MuJoCo experiments) around 0.2 million steps, all baselines show a sudden performance improvement. What factors cause these abrupt jumps in performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper evaluates non-stationary RL scenarios across Meta-World, MuJoCo, and Type-1 Diabetes environments. The inclusion of multiple benchmarks and difficulty levels provides a strong empirical basis for assessing adaptability and generalization. \n\n- WISDOM shows particularly superior performance on Meta-World, while achieving comparable or better results in MuJoCo and other environments, supporting the claimed contributions.\n\n- The paper is easy to read and follow.", "weaknesses": "- Discussion is lacking on how the proposed wavelet-based approach for tracking fast and slow changes connects to prior methods. This omission makes the contribution somewhat unclear; it is not well explained why alternative approaches could not be applied or how the proposed one provides more than an implementation variant.\n\n- Among the baselines, COREP also models non-stationarity by disentangling stable and varying causal factors through a dual-graph structure, conceptually related to WISDOM's multi-scale decomposition. A clearer discussion on why WISDOM's frequency-based modeling achieves stronger results than COREP would further strengthen the paper.\n\n- To further enhance reproducibility, it would be helpful to include additional details on how hyperparameter tuning was performed for each baseline, particularly in environments not examined in their original studies. In the same context, offering clarification on why CEMRL exhibits lower performance on Meta-World compared to other methods would further enrich the empirical analysis and provide valuable insights for replication and interpretation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wygsis54h8", "reviewer": "Reviewer_8h4R", "summary": "This paper introduces a novel reinforcement learning (RL) method for non-stationary domains. In particular, it is considered the setting in which the agent interacts with a sequence of MPDs, and the evolution of such MPDs is determined by a history-dependent stochastic process. To tackle this setting, the authors propose to track task  evolution via a Wavelet representation network. The proposed method, named WISDOM, is based on a Soft Actor-Critic (SAC) agent that also learns a wavelet representation network, which predicts a vector representation of the current task, and is optimized via a TD-learning loss combined with an auto-regressive loss function. The paper introduces two theorems that characterize policy performance with respect to wavelet-domain features. The experiments considered Meta-World, Type-1 Diabetes, and MuJoCo benchmarks, and compared the proposed approach with many competing methods in non-stationary RL.", "questions": "- Section 3.2 would benefit from more details on the problem formulation. In particular, I suggest discussing the intuition behind the formulation in Eq. 1. For instance, $\\mathcal{T}$ is a set of tasks, and $p(\\mathcal{T})$ is a distribution of tasks. Then, the authors introduce $p(\\mathcal{T_t})$, whose definition is not very clear. In Eq. 1., it is not clear how an expected value over $\\mathcal{T}_t)$ relates to the expected value over $\\omega_h$. Providing an example of a real-world scenario where Eq. 1 makes sense would improve the clarity of this section.\n\n- In line 257, $\\Gamma$ is not defined. Why does this operator not have an expected value over $z_{t+1}$? Moreover, why does this operator not depend on the policy being followed?\n\n- How does the policy being followed impact the Wavelet representations? How does the model differ from non-stationarity of the task, to non-stationarity caused by policy being updated and changing\n\n- “Stabilize the training and is updated separately and softly.” Please define what is meant by “softly”. Did you mean with polyak averaging?\n\n- In Theorems 2 and 3, the variable $J_\\pi$ was not defined. Is it the performance in a single episode? An average over the entire space of tasks? Please be precise.\n\n- There is also a class of non-stationary RL methods that track context changes in the MDP, which could have been at least discussed in the related work. See [1-3].\n\n[1] Silva, Bruno C. da, Eduardo W. Basso, Ana L. C. Bazzan, and Paulo M. Engel. ‘Dealing with Non-Stationary Environments Using Context Detection’. Proceedings of the 23rd International Conference on Machine Learning, 2006.\n\n[2] Alegre, Lucas N., Ana L. C. Bazzan, and Bruno C. da Silva. ‘Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detection’. Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, 2021.\n\n[3] Feng, Fan, Biwei Huang, Kun Zhang, and Sara Magliacane. ‘Factored Adaptation for Non-Stationary Reinforcement Learning’. Advances in Neural Information Processing Systems 35, 2022.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using a Wavelet representation to track non-stationary in RL is interesting and novel.\n- The experiments are extensive, considered challenging tasks, and have evaluated many competing methods. Moreover,  the authors provided the source code for reproducing the experiments.", "weaknesses": "- The paper can benefit from a more rigorous/precise mathematical notation in some parts of the text (see Questions below).\n- The description and discussion of Theorem 3 require significant improvements in terms of clarity. For instance, the variable $J_{WISDOM}$ is not defined, nor is it clearer what is meant by $its iterated history policy$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mTN188Sh6k", "reviewer": "Reviewer_78sm", "summary": "This paper proposes WISDOM, a method for Non-Stationary Reinforcement Learning (NSRL) that leverages wavelet-domain predictive task representations. The key idea is to perform a discrete wavelet transform (DWT) on latent task representations to capture both low-frequency trends and high-frequency variations in evolving MDPs. The authors introduce a wavelet temporal-difference update operator, theoretically prove its contraction property and convergence, and integrate it into a meta-RL framework for improved adaptability to dynamic environments. Experiments on Meta-World, MuJoCo, and Type-1 Diabetes benchmarks demonstrate improved sample efficiency and final performance compared to strong baselines such as PEARL, SeCBAD, CEMRL, and COREP.", "questions": "1.How sensitive is WISDOM to the choice of the wavelet basis?\n\n2.Does the wavelet TD update improve stability when the reward signal is extremely sparse, or mainly when the task transition distribution drifts?\n\n3.Could the proposed representation be integrated with model-based RL frameworks (e.g. world model-based RL) to further leverage the predictive nature of wavelet transforms?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The paper presents a creative use of wavelet decomposition for temporal abstraction in non-stationary task representation. Unlike conventional Fourier- or RNN-based modeling, the proposed approach explicitly encodes multi-scale temporal information.\n\n2.The authors rigorously define the wavelet TD operator and prove its contraction mapping property (Theorem 1), establishing convergence guarantees rarely seen in representation-learning-based NSRL methods.\n\n3.The appendices include detailed analyses on the effects of decomposition levels, TD loss coefficients, and architectural components, which substantiate the design choices.", "weaknesses": "1.While wavelet theory is well-motivated for time-series decomposition, the link between wavelet coefficients and latent task dynamics could be more clearly articulated.\n\n2.The paper does not discuss the additional computational cost introduced by recursive wavelet decomposition and reconstruction. An empirical runtime comparison (e.g., training time per step or GPU hours) would make the work more practically relevant.\n\n3.Although WISDOM outperforms baselines on the six reported Meta-World tasks, the evaluation scope remains narrow. Meta-World environments vary significantly in difficulty (e.g., reach vs. door-open vs. hammer) [1], and the paper only tests on relatively moderate-difficulty tasks. It remains unclear how WISDOM performs in harder manipulation environments or more complex, partially observable settings. Expanding the evaluation to a broader range of task difficulties would help establish the robustness and applicability of the proposed approach.\n\n[1] Seo Y, Hafner D, Liu H, et al. Masked world models for visual control[C]//Conference on Robot Learning. PMLR, 2023: 1332-1344.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XtJathck2R", "reviewer": "Reviewer_SUaC", "summary": "This paper aims to address an important problem: the non-stationary environment in deep reinforcement learning.\nTo this end, this paper proposes WISDOM to leverage wavelet-domain predictive task representations to enhance non-stationary RL.\nExperiments show the effectiveness of the proposed method.", "questions": "1. Can you connect your work with the non-stationary policy methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is overall well-organized and easy to follow.\n2. The non-stationary environment in deep reinforcement learning is an important and interesting problem.\n3. The authors provide sufficient theoretical support, and the proposed method shows decent performance in numerical experiments.", "weaknesses": "1. Compared to existing methods, the proposed method seems to perform decently in the numerical experiments, but the improvement is not that significant, especially for the final performance in Fig.3. Without additional analysis on the quality of the learned representation, it is not clear if the performance benefits indeed come from the proposed wavelet-domain representation.\n\n2. I think the major contribution of this work is the proposed new representation in NSRL. Thus, it would be good to discuss the metrics that can directly evaluate the \"quality\" or \"informativeness\" of the learned representations. Such metrics might include Centered Kernel Alignment and Mutual Information Neural Estimation.\n\n3. The time cost of different methods should be given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)