Learning to Reuse Policies in State Evolvable Environments
Authors: Ziqian Zhang
Bohan Yang
Lihe Li
Yuqi Bian
Ruiqi Xue
Feng Chen
Yi-Chen Li
Lei Yuan
Yang Yu
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Open Environment
State Evolution
Policy Reuse
Adaptation
Primary area: reinforcement_learning

Abstract: The policy trained via reinforcement learning (RL) makes decisions based on sensor-derived state features. It is common for state features to evolve for reasons such as periodic sensor maintenance or the addition of new sensors for performance improvement. The deployed policy fails in new state space when state features are unseen during training. Previous work tackles this challenge by training a sensor-invariant policy or generating multiple policies and selecting the appropriate one with limited samples. However, both directions struggle to guarantee the performance when faced with unpredictable evolutions. In this paper, we formalize this problem as state evolvable reinforcement learning (SERL), where the agent is required to mitigate policy degradation after state evolutions without costly exploration. We propose **Lapse** by reusing policies learned from the old state space in two distinct aspects. On one hand, Lapse directly reuses the *robust* old policy by composing it with a learned state reconstruction model to handle vanishing sensors. On the other hand, the behavioral experience from the old policy is reused by Lapse to train a newly adaptive policy through offline learning, better utilizing new sensors. To leverage advantages of both policies in different scenarios, we further propose *automatic ensemble weight adjustment* to effectively aggregate them. Theoretically, we justify that robust policy reuse helps mitigate uncertainty and error from both evolution and reconstruction. Empirically, Lapse achieves a significant performance improvement, outperforming the strongest baseline by about $2\times$ in benchmark environments.

Learning to Reuse Policies in State Evolvable Environments
Ziqian Zhang
Bohan Yang 1 Lihe Li Yugi Bian 12 Ruiqi Xue 12 Feng Chen 12 Yi-Chen Li
Lei Yuan 123
Yang Yu" 123
Abstract
1. Introduction
The policy trained via reinforcement learning (RL) makes decisions based on sensor-derived state features. It is common for state features to evolve for reasons such as periodic sensor main- tenance or the addition of new sensors for perfor- mance improvement. The deployed policy fails in state space when state features are unseen dur- ing training. Previous work tackles this challenge by training a innsor-invariant policy or generat- ing multiple policies and selecting the appropriate one with limited samples. However, both direc- tions struggle to guarantee the performance when faced with unpredictable evolutions. In this pa- per, formalize this problem state evolvable reinforcement learning (SERL), where the agent is required to mitigate policy degradation after state evolutions without costly exploration. We propose Lapse by reusing policies learned from the old state space in two distinct aspects. On one hand, Lapse directly reuses the robust old policy by composing it with learned state reconstruc- tion model to handle vanishing sensors. On the other hand, the behavioral experience from the old policy is reused by Lapse to train newly adap- tive policy through offline learning, better utiliz- ing new sensors. To leverage advantages of both policies in different scenarios, we further propose automatic ensemble weight adjustment effec- tively aggregate them. Theoretically, we justify that robust policy reuse helps mitigate uncertainty and error from both evolution and reconstruction. Empirically, Lapse achieves a significant perfor- mance improvement, outperforming the strongest baseline by about 2x in benchmark environments.
Deep reinforcement learning (RL) has shown tremendous potential in various applications (Wang et al., 2022), such as sequential recommendation systems (Lin al., 2023) and robotic control (Singh et al., 2022). However, deploying RL agents in the real world faces major challenge: policies trained under a fixed state feature space often fail to gen- eralize to new, unseen ones open environments (Zhou, 2022; Yuan et al., 2023). For instance, autonomous driving agents utilize heterogeneous sensor sources for information gathering (Kiran et al., 2021). However, the sensors might wear out due to limited lifespan or require maintenance, and new sensors with different camera views may be deployed to improve performance from richer informatioin (Shukla et al., 2023). Such common scenarios lead to continuous changes (evolutions) in the state space during deployment, requiring agents adapt the state evolvale environments.
This presents critical challenge: adapting spaces demands extensive interactions to learn a policy, which is costly in online deployment. Existing methods learn a sensor-invariant policy that is robust to changes in the state space (Chen et al., 2021; Li et al., 2022; Hansen et al., 2021), or generate multiple source policies beforehand and select the appropriate one using hierarchical high-level policies value function estimation (Cheng et al., 2020; Yang et al., 2020; Zhang et al., 2022: Chen et al., 2023), However, learning censor-invariant policy requires deter- mining which aspects of state features are task-specific and which are shared. This is complex and previous methods of- ten assume prior knowledge about new state spaces (Laskin et al., 2020a: Zhou al., 2023). Meanwhile, enumerating all possible state spaces is daunting task, making it im- practical to generate sufficiently diverse source policies for reuse (Cheng et al., 2020; Barekatain et al., 2021). Conse- quently, both directions fail to guarantee the performance when faced with unpredictable state evolutions.
state
'Equal contribution National Key Laboratory for Novel Soft- ware Technology. Nanjing University "School of Artificial In- telligence, Nanjing University "Polixir Technologies. Corre- spondence to: Lei Yuan <yuanl @aleeaaaaj..m...cc.... Yang Yu <yuy@nji Syuy@njj.edu..n>..
tackle the challenges posed by evolvable state features, we formalize the problem as state evolvable reinforcement learning (SERL), where unknown evolutions map the old state space to a new one, leading to changes in transition and reward functions as well. To mitigate performance degra- dation resulted from state space evolutions, we propose
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning to Reuse Policies for State Evolvable Environments
Learning reuse policies in state evolvable environments (Lapse) by reusing the old policy in two different directions. On one hand, by assuming the prior knowledge of sensors designated for maintenance, agents can simultaneously ob- serve features from old and new state spaces for a short period. Using a state reconstruction model trained with Con- ditional Generative Adversarial Networks (GANs) (Isola et al., 2017), the old policy can be directly reused by recon- structing the old state features from the new ones, thereby handling issues like vanishing sensors. However, uncer- tainty in state evolution and reconstruction errors may cause unbounded performance gaps, we minimize discrepancies in action distributions under perturbations, thereby bound- ing performance degradation. From another perspective, an adaptive policy is trained using the behavioral knowledge carried out by its experience via offline RL, so as better utilize new sensors. Finally, to benefit from the advantages of two policies in different scenarios, we aggregate them with automatic adjustment of ensemble weights, ensuring better performance as the state space evolves.
mapping Jn : :S, - A(S governs how the state space evolves from S,, to Sn+1, where 2()) is the probability sim- plex. N E -_ U [+oo} is the number of evolutions. The goal of SERL is to maximize the adaptation performance of policy na during deployment, without the need for costly -erroor Given that the evolution of the state space is driven by changes in sensors, we can collect limited experi- ences D, {( Sn+1, a. T2 S/,, s' s1+1) via the policy Tns We make a mild assumption that prior knowledge of which sensors will undergo maintenance and which new sensors will be deployed is available. This allows agents to perceive paired states (s,, S(+1) in a short period by simply masking the corresponding state features.
As our focus is on the evolution of the state space, the reward and transition functions remain consistent. However, due to unpredictable noise, fn may be stochastic, non-bijective, and multivalued. To manage this uncertainty, we define Bayesian-style inverse f-1 : Sa+1 - A(S,), where In (1m and quantify the uncer- f (s ()+1)) Σin (a 5.)
tainty by defining én EP consistency in Section 3.4.
Our theoretical study discloses the gap caused by the un- certainty of state evolution and justifies the use of policy robustness. Extensive experiments are performed both in MuJoCo control tasks (Todorov et al., 2012) with vec- torial state features and Atari games (Bellemare et al., 2013) with pixel-based images. To evaluate the adapta- tion capability, we test the policy in environments with multiple unknown state space evolutions. The empiri- cal results showcase remarkable improvement in our method compared to existing adaptation and transfer ap- proaches, achieving about 2x performance enhancement for state evolvable environments. Our code is available at bott LLapse
Definition 2.1. Let Mn A,, Ra, T) be an evolving MDP, with evolution fn Sn + A(Sn+1)
and inverse fm1 : Sat1 + A(S,). We define Mn+1 = (Snt1, Pat1, R1 where Sat1 = (sn+1 : 8ut1 ~ [(188),,8 8n E Sn). Ruti (8n-1, a) E BenN ((1m)11) [nn(s a)l, and Pn+i($ (441(54219n+1,0 (s++1 (1 E Sa (a. (lompr) (1 ) [Pa [s),
3. Method
https: / wit..b..hhtu.to.gg com///g-bot/Laateecoocooo
In this section, we will describe our proposed Learning to reuse policies in state evolvable environments (Lapse). As shown in Figure 1, Lapse learns to reuse the old policy in two directions. On one hand, we learn state reconstruction model and directly reuse the old robust policy. On the other hand, behavior knowledge contained in limited experience is reused to learn a new policy with robustness regulariza- tion. Moreover, we aggregate them with automatic adjusted ensemble weight. Finally, we give a theoretical analysis of how uncertainty of evolution brings about performance degradation and a bound of the performance gap.
2. Problem Formulation
The standard reinforcement learning (RL) problem (Sut- ton& & Barto, 2018) is formally defined by Markov deci- sion process (MDP), denoted as MM- = {S. A, P,,, r). Here& S and A represent state and action space, respec- A S + [0, 1] is the transition func- tion, R R:SS x A [0, Rmax] is the reward function, and y E [o. signifies the discount factor. At each time step t, the agent perceives the state s, and decides an action based on its policy at ~ T()1s:). This leads the next state S141 P(-Ist, at) and a reward R(st, ac). The primary objective is to find a policy that maximizes the expected discounted return (TT) 1 7¹ R(ss, dz)) so, ,, P).
tively. P S x
3.1. Robust Policy Reuse with State Reconstruction
When a policy w,, effective in Mn, encounters Mn-1, fails due to the evolution of the state space. However, if old state features s, can be reconstructed from new 8n-1, we can reuse nn by composing it with the learned reconstruc- tion model. Leveraging paired state features (sn, 8n+1), we can utilize supervised paired GANs (Isola et al., 2017) to learn a state reconstruction model In S + A(S)), alongside a discriminator d, Sat1 x Sn - [0, 1]. The
In this work, we consider an environment where the state space evolves over time. This scenario is formulated as state evolvable reinforcement learning (SERL) problem: {(Mn; Sn; Dw) Each evolving MDP is denoted by Mn = (Sn, A, Pn, Rn, 7). and the unknown multivalued
2

---

Learning to Reuse Policies for State Evolvable Environments
Sensor Set pd
policy network
collected experience
RL
Tt
b+ br
Mo
TTo
L robbut
(a) Robust Policy Reuse
(b) Behavior Knowledge Reuse
S
fn
Sn
maintenance
(DDBC OQL...
time
b+
b
Mnn
5
Ln+1
In
d
Smp++
off 7E
divergence
addition
Lapse
On (fr+1))
tecon xtp++1
perturbations
actions
robust
off
Rg+1
M
(c) Automatic Adjustment of Policy Ensemble Weight
AJA
Kn+1
n+1
recon
Ra+1
Kn+1
A
Figure 1. The overall framework of SERI. and Lapse
objective of the conditional GAN can be expressed as:
distributions. Concretely, we incorporate robustness regu- larization when training Tn under Mn:
GGAN = lE[log dn (an(((+)) (9n+1, sn) + log(1 (1 - (sn+1,
(1)
Lj = E
max DTv(Tn(-18p)) (r (.. (T.E))))))
(4)
where 8n 9n (8n+1). On aims to minimize this objec- tive against dn which seeks to maximize it, i.c., Ih arg ming, maxd, GAN- Additionally, we impose an 4p-
where (sn) = (K/lllsh -..122 S 6,,9% E S. is a c- neighborhood of sn. With the reconstruction model only, we could define n, = mgogue... Because my is trained in Mo before deployment, when trial-and-error acceptable, we can use any robust RL algorithm by optimizing:
based reconstruction loss to ensure the generated 8, closely matches sn:
Lip B., 8m+1 Sn wtw..tt........eeeeee.. (-1em41) 8n [p]
(2)
Lo LR.. + Ormbust
(5)
where is a predefined parameter. Concretely, we set to encourage less blurring in pixel-based environment and
Here, LRL is the standard RL objective (c.g.. TD-target for DQN (Mnih et al., 2013)), and Oroimnst weights the robust- ness term Pi Any robust RL method (e.g.. RADIAL- DQN (Oikarinen et al., 2021), Wocar-PPO (Liang et al., 2022)) can be employed for this step. We defer the theoreti- cal performance-gal analysis under robustness guarantees to Section 3.4. When extending Lapse to environments with different state action spaces, we simply adapt the method to each environment's 's standard RL backbone. Further im- plementation details can be found in Appendix B.
p = 2 in vectorial environments. The overall reconstruction objective is thus defined as follows:
ca+1
GAAN + ACT
(3)
where À is tuneable hyper-paramettr It is also feasible to instantiate the reconstruction model by diffusion networks, but we here select GAN for its faster inference time consid- cring time-delay during deployment.
By composing the reconstruction model with the old pol- icy, obtain the policy with a T - (-18a41) ((n)((()))))))) (-18a))), where 8, ~ (-(1)x++1) Since g is trained to approximate f-1, we show in Section 3.4 that this reconstruction-bue based policy can transfer effectively to Mn+1 - In practice, however, limited data coverage dur- ing the short period and reliance on a fixed behavior pol- icy can hinder gn.s 's ability to generalize to unseen states. Additionally, standard RL policies tend to be sensitive to perturbations of their inputs (Zhang et al., 2020a), so the re- construction error may lead to erroneous actions, that drive the agent into unohserved states, further compounding the reconstruction error.
3.2. Offline Policy Learning with Knowledge Reuse
Despite the effectiveness of robust policy reuse when old sensors vanish for maintenance or power failure, it fails to take advantage of the information provided by newly added sensors. Meanwhile, the reconstruction error from the composite function 9. := 9o o gI... on. will accumulate as the state evolution continues. Relying solely on direct policy reuse will lead to increasing discrepancy in the action distribution E[Drv(To) (-150), mo In (-1s.++1 ))}, and the compounding error from the composition of policy and reconstruction function becomes problematic again.
We mitigate these issues by learning a robust policy that reduces the impact of small input perturbations on action
To mitigate the issues, we propose to complement policy reuse with the reuse of behavioral knowledge to further
3

---

Learning to Reuse Policies for State Evolvable Environments
promote adaptation in the new MDP. Although the recon- struction policy may deteriorate over time, its performance is stable for the earlier stages, suggesting decent qual- ity of offline dataset D, = {(5,, Sn+1, a, 7, S/y: 1)) A straightforward method is to apply behavior cloning which attempts to simply clone the actions observed in the dataset, but it also fails to leverage the additional information gained from the newly added sensors. Therefore, we turn to offline RL for policy learning, as suggested by Kumar et al. (2022).
where Jn (T,) is the expected return of policy ah under M,. and D(p,q) is divergence metric (e.g., KI. divergence). no, then Kn+1 emphasizes whichever policy is closest to TT. Otherwise, the newly adaptive policy, which is expected to exceed the behavior policy via offline RL, takes precedence.
If T. performs comparably to
To summarize the overall training and testing processes un- der the SERL framework, Lapse first learns robust initial policy To via Equation. (5). During deployment, nn makes decisions in M,, Prior to evolving into Mn+1,, Wn could perceive both state feature from S, and Sx+1 for a short period and collects limited amount of experience. The re- construction model and the offline adaptive policy are then trained to obtain =o = Wn o In and = 1n+1, respectively. Our aggregation method automatically adjusts the ensem- ble weight Ka++1 of Tn+1. facilitating transfer to Mn+11 For memory efficiency, one may also prune past policies with negligible performance loss. Detailed pseudocode is provided in Appendix C.
For environments with vectorial state features and con- tinuous action space such as Mujoco, we employ the TD3+BC (Fujimoto & Gu, 2021) to learn the offline policy rff The loss function is optimized as follows:
#n+1-
off
(T (Sn+1) - a)²+ D1ss
(6)
Bn+1Qn-1 (s To (sn+1))l.
where Quti is learned via the TD3, and Bu+1 is coef- ficient controlling conservatism. We schedule the coef- ficient dynamically with state feature evolution, setting Ba+1 = =mmxx(1 exp n)), where T is a decay co- efficient. Intuitively, when reconstruction error is small (in early stages), we rely more on imitating the behavior pol- icy. As uncertainty grows, we tilt toward more optimistic updates. To further enhance robustness, we incorporate robust regularization term:
3.4. Theoretical Analysis
We now analyze how a robust policy, combined with a learned state reconstruction model, can limit performance degradation under evolvable state space. To characterize the uncertainty introduced by evolution Sn. we first define ER Ep consistency:
Ca+1 off
cn+11 1 + Grouust robust
Definition 3.1. Given M,, and fn. we say fn is ER - EP consistent, if for any 8n E Sn, Sn-1 E Snt1,a E A, with [n( (8,) O, the following holds:
where a robust n+1 balances the regularization, and C is given
in Equation (4). Different from online training, we imple-
Rn [n a)
In EE
" (5n (5n|8n41 1n (5n,a)) S ER:
ment the robust regularization via function smoothing based on robust offline RL proposed by RORL (Yang et al., 2022). For environments with pixel-based image inputs and dis- crete action space, we adopt suitable offline RL backbones tailored to such settings. Detailed architectural and algorith- mie choices are provided in Appendix C.
8n
[Pa( a) - -F, FFriPatt PPtti((nnt) Sep.
Here, Pu(sn; a) tion for (sn, u), and F= is |Sp X |Sm+1| ma- trix with F Sn+1) = fn (sx1))+1).. Thus, F FP'Put "Pn+i($nt1,0) collapses the transition distribution over Sa+1 to one over S,,.
RIS is the transition distribu-
3.3. Policy Aggregation with Automatic Adjustment of the Ensemble Weight
The two distinet ways of policy reuse are proficient in dif- ferent scenarios. Finally, to leverage their advantages, we combine them via an ensemble whose weight is adjusted automatically (Lee et al., 2021). Specifically, for Mn+1, the policy 1n+1 is recurrently defined as:
Intuitively, ER - ep consistency controls the discrepancy of reward and transition functions in Mn, and Mn+1, thereby bounding the uncertainty of the evolution fn and fz
Our starting point is the theoretical effectiveness of the state reconstruction model. If we have direct access to the inverse f. of the evolution fa. we can immediately reuse n, by composing it with the f to get ग s-= The following proposition shows that the performance loss in this scenario only results from the E - EP consistency of fn-
roonn In+1 = K T 1n + (1 ) n+11
2 0,
where T Tn Ir. no. In and Tel are learned via Equations (5). (3), and (7), respectively. We set
Proposition 3.2. Let fo be CR - cp consistent, and f be its inverse. Suppose fr (sn,olsn410)) = 1 for the ini- tial state 8n,0 and 8,+1.0 in Mn and Mn+1. respectively.
Jn (TT))
D(TT, ofll
Kn+1 = Jo( D(T,, m0000)) + D(TT, moff
4

---

Learning to Reuse Policies for State Evolvable Environments
In (n = Ei I' I 8 Pal. When applying Tn 1 in Mu+1,, the performance gap is bounded by
and Atari games with pixel-based image inputs and discrete action spaces. Our experiments are designed to address three critical questions: (1) Whether Lapse achieves supe- rior adaptation capabilities in state evolvable environments compared to existing methods (Section 4.2)? (2) How does the learning process of Lapse proceed in evolving stages, assessing its adaptability (Section 4.3)? (3) What contribu- tions do the different components and hyper-parameters of Lapse make to its overall performance (Section 4.4)?
|T+t(TT o fT Jn((n -
ER_ +
YCp yep-mmax
1 - 2(1 7)2
The full proof can be found in Appendix A. We disclose the policy performance gap when encapsulating nn with the oracle inverse of evolution fn' This quantifies how the uncertainty in the evolution, as captured by ER and ep, affects the performance under anfnite-horizon setting.
For a comprehensive evaluation, Lapse is compared against multiple baselines. All results are averaged over five random seeds and are presented with their corresponding standard deviations. Detailed descriptions of the experimental setups, including environmental conditions and network architec- ture parameters, are provided in the Appendix C.
In practice, we only have access to learned reconstruc- tion function In. that approximates fn² Because the data D, come from trajectories with specific distributions, compounding errors may occur if In is inaccurate in off- distribution states. The next proposition shows that a robust policy n and help mitigate such errors.
4.1. Baselines and Environments
Proposition 3.3. Let fi-1 be the inverse of evolution fn that transforms Mn into Mn+1. Suppose we have both the policy an and the reconstruction model gn- If
To thoroughly assess the performance of Lapse, we com- pare it against the following adaptation baselines: (1) RL- GAN (Gamrian & Goldberg, 2019) simply employs fea- ture reconstruction model to reuse the old policy, but with- out robust training. (2) LUSR (Xing et al., 2021) focuses on extracting disentangled state representations and trains agents in the latent space. (3) PAD (Hansen et al., 2021) optimizes RL and self-supervised objectives in the initial MDP (Ma) and adapts by fine-tuning the representation using self-superrvsed signals. (4) Offline learns new pol- icy for each Mn+11 using data D,,, employing offline RL. (5) Few-shot Policy Transfer (FPT) (Shukla et al., 2023) reuses old policies to guide the transfer of policies under new environments. (6) CUP (Zhang et al., 2022) generates multiple source policies in the initial MDP (using random- ized domains) and selects the appropriate one based on critic value, which then guides the learning of the new policy.
E [Drv(t o f-2 (-1s"tt), ) Th o (-18" 11]] S 7, (10)
where Drv(p.g)= = En lp(x) q(x) E [0, denotes to- tal variation divergence for discrete probability distribution
p, 9. then:
(JntiTTn o f - Jnti(En o gn)l S 2Rmax
(1 (1-
Moreover. if fn is En- Ep consistent, it follows that
- tR (7EP + 44) Rmax
(J)((T,) - JnT o gn)l
1 1-7
2(1 7)
The intuition behind Proposition 3.3 is that if the old pol- icy n, is robust enough to restrict the differences in action distributions under perturbations, then composition with a slightly inaccurate 9n causes limited performance degra- dation. The detailed proofs are available in Appendix A. We also validate theoretical results through experiments in Section 4.3. Accordingly, when learning on using dataset D wo under Mo. we could focus on minimiz- ing E[Div (T,, (- (-Isx), =-)(=)))) where s,, ~ (-(ssa++)) While we fail to obtain In. heforehand, we can assume that the reconstructed state 8n is bounded by (s,) = (s/ |ll snll2 I e, 6, E S, as 9n. is optimized to minimize reconstruction errors. This motivates the robust regularization in Equation (4), discussed in Section 3.1.
For evaluation, we select four tasks from the Gym Mu- JoCo suite (Todorov et al., 2012): Ant, Hal fCheet ah, Hopper, and Wa l ker. In these tasks, agents receive vecto- rial state features and output continuous actions. We utilize PPO (Schulman et al., 2017) as the backbone and train the initial policy (Tg) via Wocar (Liang et al., 2022). In the context of Atari games (Bellemare al., 2013), which in- volve pixel-based image inputs and discrete action spaces, we employ DQN (Mnih et al., 2013) as the backbone to train the robust RADIAL agent (Oikarinen et al., 2021) To for games including BankHeistt F Freeway, Pong, and RoadRunner, following the setup outlined in Oikarinen et al. (2021). The specifics of the state space evolution process in these environments in Appendix B. To simulate the addition and removal of sensors, we split pixel-based images into multiple patches. The addition or reduction of patches or dimensions is then applied. Furthermore, we introduce more complex evolu-
detailed and illustrated
4. Experiments
In this section, we present our experimental analysis con- ducted across eight diverse tasks, including continuous con- trol tasks from Mujoco, featuring vectorial state features,
5

---

Learning to Reuse Policies for State Evolvable Environments
tions including the replacement of physical units, rotation of view, introduction of distracting objects and arbitrary linear mapping. Noise is added to simulate the uncertainty of the evolution. We restrict the number of trajectories in the dataset D, to 10 and 15 in Mujoco and Atari, respectively.
control over this decline. While RL-GAN's performance de- teriorates gradually or breaks down at the first stage, Lapse effectively reuses the old policy, due to the application of the enhanced robustness. LUSR, PAD, and FPT are con- strained by the performance of the last stage's policy, yet Lapse transcends these limitations, especially notable in the final stage of HalfCheetah, where even gains perfor- mance improvement in the last stage. This is attributed to the potential of offiin adaptive policy learning. Offline and CUP, despite observing similar trends, cannot match Lapse's s performance, largely due to their absence of policy aggre- gation mechanism with dynamically adjustable ensemble weights. The complete result is provided in Appendix D
4.2. Competitive Results
Overall Adaptation Performance For a fair evaluation of adaptation performance, we analyze all methods across five evolving stages with various orders of evolutions. The performance is normalized against the initial stage return: 5
100. i E Jo(mo)
I_(sn), with 10 episodes per stage to estimate J. This metrie reflects how well each method retains performance relative to its initial stage, demostrating the agent's 's adaptability.
4.3. Analysis of Policy Reuse
Validation of Theoretical Analysis Section 3.4 provides a theoretical discussion on the robustness of policy reuse through state reconstruction. To further validate the the- oretical results, we demonstrate the detailed learning pro- cess for single evolving stage in Figure 3. We compare robust policy and vanilla policy, each coupled with reconstruction model trained in the same way. Both the reconstruction models' training losses, depicted by the dot- ted lines, decrease and stabilize rapidly, indicating mini- mal reconstruction errors of the old states. This allows the robust policy to be effectively reused in the new state space, delivering satisfactory performance as shown by the red solid line. In contrast, the vanilla policy without prior robust training breaks down, even with well-trained state reconstruction model. To quantify the total variation divergence [[Drv(TA)) (s*/,=( T. f-19.. (s*+1)))) we measure the action differences produced by the policy when perceiving real and reconstructed states under the testing environment. The blue dash-dotted line that represents the action difference of the vanilla policy has illustrated that the performance gap is mainly caused by the distribution shift and compounding errors. Conversely, a robust policy is capable of mitigating these issues by restricting action distribution discrepancies under perturbations beforehand, thus being reused effectively.
Table 1 reveals that all algorithms experience performance degradation as the state space evolves, underscoring the need for specialized strategies. Domain adaptation methods like RL-GAN and LUSR, which rely on substantial train- ing data to generalize or translate states, struggle because limited samples of D,, exhibiting low adaptability. FPT shows only marginal gains compared to RL-GAN, indicat- ing that few-shot fine-tuning is not always sufficient. PAD, intermediate representa- tions through self supervisiii,, fails to cope with state space evolution. It learns latent features from Mo alone and it will inevitably be affected by domain specificity, verifying the difficulty to learn a sensor-invariant policy. In addition, PAD fine-tunes the features via limited data with a shallow distribution collected during a short period, this leads to ineffective adaptation. Offline demonstrates decent adapt- ability tasks like Ant and Hopper and is on par with Lapse in these scenarios. However, its capability diminishes in Atari games with nigh-dimensional inputs and complex evo- lution mappings. CUP shows promise by guiding source policy selection through critic value estimation. However, it is prone to performance degradation when the divergence in state space between the source and target environments becomes too large. This highlights the limitation of previ- ous policy reuse approaches: they are primarily suited for transfer learning and rely on costly trial-and-error handle significant state space changes. In contrast, Lapse stands out, consistently delivering the best adaptation performance across all tasks. It nearly doubles the effectiveness com- pared to the strongest baseline in average, evidencing the superiority and efficiency of our proposed method.
despite learning dynamic-informen
Analysis of the Learning Process We further analyze the learning process of Lapse in HalfCheetah, where the recon- struction model and the policy are updated for 10K steps in each evolving stage, as depicted in Figure 4. First, the reconstruction policy क exhibits high efficacy in early stages, but declines over repeated evolutions. The com- pounding reconstruction error inevitably accumulates and erodes the initial policy's robustness with the increase of the evolving stages, widening the performance gap. Further- more, even with an optimal state reconstruction model, the reconstruction policy T fails to surpass the old policy Tne The offline adaptive policy, off shows better long-
Continuous Adaptation Capability Figure 2 illustrates the continuous adaptation of various methods to the evoly- able state space by comparing the performance over 5 differ- ent stages. Although all methods exhibit increased perfor- mance degradation over successive stages, Lapse maintains
6

---

Learning to Reuse Policies for State Evolvable Environments
Table Average test return + std across various tasks. The highest performance each row emphasized bold. Results are averaged over 5 evolving stages, 10 episodes per stage, and 5 distinct seeds for robustness, Values are normalized against the initial stage return, indicating the performance retained relative to the initial stage.
Task
RL-GAN
LUSR
PAD
Offline
FPT
CUP
Lapse
Ant
28.10 3.29
39.57 1.90
21.17 1.45
82.63 4.95
38.39 + 5.31
69.83 + 9.57
84.02 + 2.69
HalfCheetah
12.26 3.06
14.55 3.35
28.31 1.07
15.01 2.71
13.13 4.36
12.01 + 2.02
82.56 + 4.48
Hopper
16.43 5.96
15.42 6.40
34.02 3.73
74.16 9.33
78.94 23.18
66.62 + 20.39
98.89 3.09
Walker
3.03 + 1.29
4.38 + 1.75
10.58 1 6.61
18.91 + 6.27
1
13.42 + 4.59
83.85 + 7.49
BankHeist
25.38 + 10.58
0.00 + 0.00
41.05 + 10.43
61.07 + 37.43
20.79 14.78
0.00 - 0.00
95.34 + 0.89
Freeway
43.86 + 16.26
61.64 1.22
78.86 + 3.46
71.85 4.47
78.10 : 5.77
48.82 : 24.74
98.32 + 1.34
Pong
37.41 9.68
0.06 + 0.08
6.54 + 4.55
33.56
18.06
33.56 8.33
0.11 + 0.23
100.0 0.00
RoadRunner
14.66 4.82
2.33 + 0.08
28.67 + 11.28
6.92 + 4.3.0
24.46 + 9.00
1.25 + 1.04
85.65 + 9.33
Overall
22.64
17.24
31.56
45.51
38.41
26.51
91.09
Lapse
RL-GAN
LUSR
PAD
Offline
FPT
CUP
HalfCheeeth
Hopper
Freeway
Pong
Normalized Average Return
100
100
100
100
40
40
40
40
20
20
20
20
2
2
2
2
Evolving Stage
Evolving Stage
Evolving Stage
Evolving Stage
Figure 2. The adaptation performance of the Icarned policy as the state space evolves
strengths of offline RL based on Equation (9), even without prior knowledge of the individual test performances of the two policies. Incorporating these mechanisms, Lapse real- izes superior adaptation performance, outpacing standard approaches. Meanwhile, we illustrate how the two aspects of policy reuse contribute effectively in different scenarios.
Normalized Retum Looss
Robust
1.00
Vanilla
Test Performance
0.75
Training Loss
Test Action Difference
0.50
0.25
When the sensor on the backward shin of the HalfCheetah robot wears out, our method learns to reconstruct the miss- ing information from available features, enabling the reuse of the robust policy network. As the reconstruction loss converges, the robust policy can be effectively deployed. Moreover, we visualize the saliency map of offline adaptive policy wlll's state features via SmoothGrad (Smilkov et al., 2017). The added sensor on the forward shin provides ad- ditional state features. The offline policy not only reuses behavior knowledge from the old policy but also leverages the information for improved decision maasin..
0.00
0.0K
2.0K
4.0K
6.0%
8. B.O
10.0K
Training Steps
Figure 3. Comparison of training loss, test return, and test action difference when reusing robust policies versus reusing vanilla poli- cies through state reconstruction. The type of policy (robust and vanilla) is indicated by color, and different metrics (performance, loss and action difference) represented by line styles.
term stability and can potentially exceed the performance of sub-optimal old policy and better utilize new sensors. To benefit from both policies, we aggregate them via an tomatically adjusting ensemble weight k which reflects the dependency on the reconstruction policy. As is shown by the dash-dotted line, Kn-1 initially favors the reconstruction policy due to its closeness to the well-performer old policy. As the state space evolves, Ka+1 is reduced to harness the
4.4. Ablation and Parameter Sensitivity Studies
Ablation Study To dissect the impact of each component within Lapse, we execute ablation studies on the Pong task, as depicted in Figure 5(a). This analysis uncovers their con- tributions to Lapse in evolvable state space. First, W/o recon

---

Learning to Reuse Policies for State Evolvable Environments
bshin-sensor wears out learn roconstrict
new fehin-sensso learn to leverage
5. Related Work
Domain Randomization and Domain Adaptation Do- main randomization is a widely used technique for enhanc- ing policy generalization (Laskin et al., 2020a; Peng et al., 2018; Zhou et al., 2023). It generates diverse training scenar- ios by introducing visual or dynamic changes in simulators, with the hope that the unknown test environments can be covered. However, this approach has limitations: the train- ing complexity scales with the number of variations, and in- sufficient randomization may lead to inadequate coverage of target domains. In contrast, target domain data is assumed to be accessible in domain adaptation (Zhu et al., 2023). RL-GAN employs unaligned GANs to translate target do- mains into familiar states in the source domain (Gamrian & Goldberg, 2019). CURL extracts high-level features from raw pixels using contrastive learning (Laskin et al., 2020b), and LUSR adopts Cycle-Consistent VAEs for unified state representation learning (Xing et al., 2021). Unlike these approaches, Lapse does not presume access to unknown target domains during deployment. We focus on adapting to evolvable state space via policy reuse.
biki
boita-
lshin-
Festure Salieney Stip
scnsor
1.00
Nomaiizzd Retum
0.75
0.50
0:25
Resouttcctee octaae
Lapa
OK
10K
20K
30K
40%0
50K
Training Steps
Figure 4. The learning process of Lapse in HalfChectah environ- ment. Displayed are the test returns for the reconstruction policy, the offline adaptive policy, and the combined Lapse policy. Also plotted the ensemble weight (1,,4-)). ), illustrating its dynamic adjust- ment throughout evolution. When sensor placed on the backward shin wears out, the model is optimized to reconstruct the missing feature. When a sensor is added the forward shin, the policy can utilize new feature, as shown in saliency map.
Normalized Average Return
100
Normalized Retum
Policy Reuse Policy reuse learns multiple policies each specialised to their own subset of tasks to improve transfer to a new unseen task Fernández & Veloso (2006); Rosman et al. (2016); Li et al. (2019). One branch of meth- ods train hierarchical high-level policies over source poli- cies. Another branch of works aggregate source policies with the guidance of value functions on the target task Bar- reto al. (2018); Cheng et al. (2020); Barekatain al. (2021). (Barreto al., 2018) focus on the situation where source tasks and target tasks share the same dynamics, and aggregate source policies by choosing the policy that has the largest Q value at each state based on successor features. CUP (Zhang et al., 2022) chooses a guidance policy that has the largest one-step improvement by querying the critic, and guides learning by regularizing the target policy to imitate the guidance policy. Bossens & Sobey (2024) adaptively assigns policies to tasks based e-greedy bandit learning over lifetime reinforcement learning. Continual Reinforce- ment Learning (CRL) (Kessler et al., 2022; Li et al., 2024) focuses on adapting to new tasks quickly by reusing pol- icy knowledge learned from old tasks, but they still require costly trial-and-error. All these approaches focus on effi- ciency on transfer to tasks with dynamics or reward gaps. We emphasize the evolution of state space and do not allow costly trial-and-error.
Lapse
80
20
Wil kappa
80
25
16
0.1
30
100
1000
Evolving Stages
Parameter Value
(a) Ablation study
(b) Parameter sensitivity study
Figure 5. Ablation and parameter sensitivity studies on Pong and
Ant, respectively.
and W/ offline relies solely Toff and m respectively. Both of their adaptation performance declines in later stages, especially W/o offline, emphasizing the necessity of the pol- icy aggregation mechanism. Moreover, W/o robust learns policies without robustness regularization and this results in unbounded performance gap for the reconstruction policy. Its inferior performance compared to W/o recon also sug- gests that robust training aids in offline learning. W/o kappa ensembles two policies with a static weight Kx+1 = 0.5. It suffers the least performance degradation among all abla- tions but still performs worse than Lapse, manifesting the robustness of the policy aggregation and the benefit of the automatic ensemble weight adjustment.
Parameter Sensitivity Study We probe the influence of hyperparameters on adaptation efficacy in the Ant. Here we examine variations in X, which balances the GAN objective and the fidelity of state reconstruction. An excessively low A leads to a misalignment between state spaces, while an overly high À could hinder the ability to fool the discrimi- nator. We find that À= = 10 is the best choice in Figure 5(b). Additional experimens are detailed in the Appendix D.
Novelty Detection and Adaptation Focusing on the open- world decision making problem, line of research works refer to abrupt changes in environments as novelty (Peng et al., 2021; Balloch et al., 2023). They propose to detect the novelty via knowledge graph or neuro-symbolic world model and achieve policy adaptation with imagination tra- jectories generated from the updated world model. These
8

---

Learning to Reuse Policies for State Evolvable Environments
methods fail to be directly applied high-dimensional, con- tinuous and complex environments, where dynamics and rewards are difficult to be depicted with simple rules. Zolli- coffer et al. (2023) addresses this problem by extending it to latent-based novelty detection via DreamerV2 (Hafner et al., 2020) but does not study how to achieve policy adaptation in complex environments, mainly because the accuracy of imagination trajectories cannot be guaranteed. Our method takes a further step for the high-dimensional, continuous and complex environments via novel solution.
Foundation of Jiangsu (BK2024119, BK20243039). We thank Tencent AI Arena for their support, and the anony- mous reviewers for their support and helpful discussions on improving the paper.
Impact Statement
This paper presents work whose goal is to advance the field of Reinforcement Learning (RL). Our method is intended to more effectively adapt RL policy to possible changes in state space during deployment. The work presented does not raise any additional ethical concerns, and thus no special discussion on ethical issues is required.
Learning with Evolvable Features The concept of Fea- ture Evolvable Streaming Learning (FESL) was introduced by Hou et al. (2017), focusing on an overlapping pe- riod where features from different stages are observable. OLVF (Beyazit et al., 2019) learns to classify the feature space and the instances simultaneously to handle arbitrarily varying spaces. FDESL (Zhang et al., 2020b) further ex- tends the setting by assuming that the data distribution also changes. Our work is related to FESL paradigm, but they focus on online learning with data streams. We first consider the problem in RI. and formulate it into SERL problem.
References
Balloch, J. C.,, Lin, Z., Peng, X., Hussain, M., Srinivas, A.,
Wright, R., Kim, J. M., and Riedl, M. O. Neuro-symbolic world models for adapting to open world novelty. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 2848- 2850, 2023.
Barekatain, M., Yonetani, R., and Hamaya, M. Multipolar: multi-source policy aggregation for transfer reinforce- ment learning between diverse environmental dynamics. In IJCAI, pp. 3108-3116, 2021.
6. Conclusion
In this work, we present the Learning reuse policies in state evolvable environments (Lapse) approach, address- ing the challenge of evolvable state space in reinforcement learning (RL) with the formulation of state evolvable re- inforcement learning (SERL). Lapse reuses policies from two aspects by combining a robust policy and an adaptive policy through state reconstruction and offline learning, thus avoiding the need for extensive and costly trial-and-error during deployment. Theoretical and empirical results adaptation capability underscores Lapse 's potential in ef- ficiently handling dynamic real-world scenarios. However, the requirement of D, can restrict the application scenarios in real world, and Lapse may be less effective in environ- ments with drastic state space evolutions. Future work could focus on enhancing Lapse to handle more rapid state evolu- tions and exploring its applicability to more complex tasks. Another appealing direction is extending Lapse to multi- agent embodied intelligence settings (Feng et al., 2025), where multiple physically situated agents must coordinate and adapt in a jointly evolving state space. Furthermore, relaxing the need of behavioral experiences through Large Language Models (LLMs) (Kim al.. 2024) and human instructions is also promising and valuable direction for future research.
Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A., and Munos, R. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In ICML, pp. 501-510, 2018.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-279 2013.
Bellemare, M. G., Dabney, W., and Munos, R. A distribu- tional perspective reinforcement learning. In ICML, pp. 449 458, 2017.
Beyazit, E., Alagurajah, J., and Wu, X. Online learning from data streams with varying feature spaces. In AAAI, pp. 3232-3239, 2019.
Bossens, D. M. and Sobey, A. J. Lifetime policy reuse and the importance task capacity, Al Communications, pp. 1-34, 2024.
Acknowledgements
Chen, A. S., Chada, G., Smith, L., Sharma, A., Fu, Z.,
Levine, S., and Finn, C. Adapt on-the-go: Behavior mod- ulation for single-life robot deployment. arXiv preprint arXiv:231 arXiv:2311.0005, 2023.
This work is supported by the National Science Founda- tion of China(62495093, U24A20324), the Natural Science

---

Learning to Reuse Policies for State Evolvable Environments
Chen, X.-H., Jiang, S., Xu, F,, Zhang, Z., and Yu, Y. Cross- modal domain adaptation for cost-efficient visual rein- forcement learning. In NeurIPS, pp. 12520-12532, 2021.
Kumar, A., Hong, J., Singh, A., and Levine, S. When should we prefer offline reinforcement learning over behavioral cloning? In ICLR, 2022.
Cheng, C.-A., Kolobov, A., and Agarwal, A. Policy im- provement via imitation of multiple oracles. NeurlPS, 33: 5587-5598, 2020.
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. In NeurlPS, pp. 19884-19899 2020a.
Feng, Z., Xue, R., Yuan, L., Yu, Y., Ding, N., Liu, M., Gao, B., Sun, J., and Wang, G. Multi-agent embod- ied ai: Advances and future directions. arXiv preprint arXiv:2505.055 108, 2025.
Laskin, M., Srinivas, A., and Abbeel, Curl: Contrastive unsupervised representations for reinforcement learning, In ICML, pp. 5639-5650 2020b.
Lee, K., Laskin, M., Srinivas, A., and Abbeel, P. Sunrise: simple unified framework for ensemble learning in deep reinforcement learning. In ICML, pp. 6131-6141, 2021.
Fernández, F. and Veloso, M. Probabilistic policy reuse a reinforcement learning agent. In AAMAS, pp. 720-727, 2006.
Li, D., Meng, L., Li, J., Lu, K., and Yang. Y. Domain adaptive state representation alignment for reinforcement learning. Information Sciences, 609:1353- 2022.
Fujimoto, S. and Gu, S. S. A minimalist approach to offiin reinforcement learning. In NeurlPS, pp. 20132-201444 2021.
Li, L., Chen, R., Zhang, Z., Wu, Z., Li, Y.-.., Guan, C., Yu, Y., and Yuan, L. Continual multi-objective reinforcement learning via reward model rehearsal. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp. 4434-4442, 2024.
Gamrian, S. and Goldberg, Y. Transfer learning for related reinforcement learning tasks via image-to-image to-image transla- tion. In ICMI, pp. 2063-2072, 2019.
Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. Mas- tering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
Li, S., Gu, F., Zhu, G., and Zhang, C. Context-aware policy reuse. In AAMAS, pp. 989-997, 2019.
Hansen, N., Jangir, R., Alenyà Ribas, G., Abbeel, P., Efros A, A., Pinto, L.,, and Wang, X. Self-supervised policy adaptation during deployment. In ICLR, 2021.
Liang, Y., Sun, Y., Zheng, R., and Huang, Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning. In NeurlPS, pp. 22547- 22561, 2022.
Hou, B.-J., Zhang, L., and Zhou, Z.-H. Learning with feature evolvable streams. In NeurlPS, pp. 1417-1427, 2017.
Lin, Y., Liu, Y., Lin, F., Zou, L., Wu, P., Zeng. W., Chen, H., and Miao, C. A survey on reinforcement learning for recommender systems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Isola, P., Zhu, J.-Y. Zhou, T., and Efros, A. A. Image-to- image translation with conditional adversarial networks. In CVPR, pp. 1125-1134, 2017.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint 2013.
Jiang N., Kulesza, A., and Singh, S. Abstraction selection in badel-based reinforcement learning. In /CML, pp. 179-188, 2015.
Kessler, S., Parker-Holder, J., Ball, P., Zohren, S., and Roberts, S. J. Same state, different task: Continual rein- forcement learning without interference. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pp. 7143-7151, 2022.
Oikarinen, T., Zhang, W., Megretski, A., Daniel, L., and Weng. T.-W. Robust deep reinforcement learning through adversarial loss. In NeurlPS, pp. 26156-26167, 2021.
Peng, X., Balloch, J. C., and Riedl, M. O. Detect- ing and adapting to novelty in games. arXiv preprint N.T::2106.022044 2021.
Kim, W. K., Lee, Y., Kim, J., and Woo, H. Llm-based skill diffusion for zero-shot policy adaptation. In NeurIPS, 2024.
Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Sim-to-real transfer of robotic control with dynamics randomization. In ICRA, pp. 3803-3810, 2018.
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Al Sal- lab, A. A., Yogamani, S., and Pérez, P. Deep reinforce- ment learning for autonomous driving: survey. IEEE Transactions on Intelligent Transportation Systems, 23 (6):4909-499) 4926, 2021.
Puterman, M. L. Markov decision processes: discrete
stochastic dynamic programming. John Wiley & Sons, 2014.
10

---

Learning to Reuse Policies for State Evolvable Environments
Rosman, B., Hawasly, M., and Ramamoorthy, S. Bayesian policy reuse. Machine Learning, 104:99-127, 2016
Yuan, L.,, Zhang, Z.,, Li, L., Guan, C., and Yu, Y. A vey of progress cooperative multi-agent reinforce- ment learning in open environment. arXiv preprint arXiv:2312.01058, 2023.
Schulman, J., Levine, S., Abbeel, Jordan, M., and Moritz, P. Trust region policy optimization. In ICML, pp. 1889- 1897, 2015.
Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. Robust deep reinforcement learning against adversarial perturbations state observations. In NeurlPS, pp. 21024 21037, 2020a.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 1rX1v:170..06347, 2017.
Zhang, J., Li, S., and Zhang, C. Cup: Critic-guided policy reuse. NeurIPS, 35:27537-27548, 2022.
Shukla, Y., Kesari, B., Goel, S., Wright, R., and Sinapov, J. A framework for few-shot policy transfer through observation mapping and behavior cloning. In IROS. pp. 7104-7110) 2023.
Zhang, Z.-Y., Zhao, P., Jiang, Y., and Zhou, Z.-H. Learning with feature and distribution evolvable streams. ICML, pp. 11317-11323 2020b.
Singh, B., Kumar, R., and Singh, V. P. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review, pp. 1-46, 2022.
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., and Loy, C. C. Do- main generalization: A survey. IEEE Transactions on Puttern Analysis Machine Intelligence, 45(04):4396- 4415, 2023.
Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Watten- berg, M. Smoothgrad: removing noise by adding noise. arXiv preprint wrXiv:1706.03255. 2017.
Zhou, Z.-H. Open-environment machine learning. National Science Review, 98):nwac 123, 2022.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, Cambridge, 2018.
Zhu, Z., Lin, K., Jain, A. K., and Zhou, J. Transfer learning in deep reinforcement learning: A survey. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2023.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033, 2012.
Zollicoffer, G., Eaton, K., Balloch, J., Kim, J., Riedl, M. O., and Wright, R. Novelty detection in reinforcement learn- ing with world models. arXiv preprint arXiv:2310.08731, 2023.
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., and Miao, Q. Deep reinforcement learning: survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Wong, E. and Kolter, Z. Provable defenses against adver- sarial examples via the convex outer adversarial polytope. In ICMI,, pp. 5286-5295, 2018.
Xing, J., Nagata, T., Chen, K., Zou, X., Neftci, E., and Krichmar, J. L. Domain adaptation in reinforcement learning via latent unified state representation. AAAI, pp. 10452-10459, 2021.
Xu, T,, Li, Z., and Yu, Y. Error bounds of imitating policies and environments. In NeurIPS, pp. 15737-15749, 2020.
Yang. R., Bai, C., Ma, X., Wang. Z., Zhang, C., and Han, L. Rorl: Robust offline reinforcement learning via conserva- tive smoothing. In NeurlPS, pp. 23851 -23866, 2022.
Yang. T., Hao, J., Meng, Z., Zhang, Z., Hu, Y., Chen, Y., Fan, C., Wang, W., Wang. Z., and Peng, J. Efficient deep reinforcement learning through policy transfer. In AAMAS, pp. 2053-2055, 2020.

---

Learning to Reuse Policies for State Evolvable Environments
A. Proofs for Theoretical Analysis
A.I. Proof for Proposition 3.2
We first introduce the following lemma for the proof of Proposition 3.2.
Lemma A.1. Let In be an - evolution of Mn. which leads to Mat1. f is the inverse of the evolu- tion. As is defined in Definition 2.1, Re+1(5n+1,04 a) = E., (1 Ba (8n, a)}. and Pat Pu+i($) 11 a) E B So "H ((1))4))) PP (P a)]. Then for any Sn+1 Sn+1, A.
- (15m (1811)) (8n, a)] - Ru+i (Sn+1, a0] SR.
Proof.
E en (1 (R ( a)] ( Σ
( (8n (Rn))) a)] Ra+i (8441,))
8n
fo (sn 9 (an a) Rn+1(5n+1,
Σ I (8m 9 Rn( Sn () (n(n
(11)
Σ ((nn 0)f7 (s 18n Re (Sn; a) E
a)]
Σ I(fn(Sn+1155) > (sm//s44)) 8n
=E
According the bayesian inverse (s.1(51(4441)) Σn, Ia (#n4118,)) Sn fn(sn+ only if S (sal844))) 0. This allows for the application of Definition 3.1, which bounds the discrepancy of reward function when f fn 0. fn(sn4a|sm) also noticeable that we define fo- ((nl))+1)=== Es,, (#n411885)) by assuming uniform prior distribution pr, over
fo (1.4t1ls))
S.. We can also introduce occupancy measurement induced by uniform policy as prior, but it will not change the conclusion of the proposition.
Based on the above conclusion, we show that the performance gap between Fn and n. fn' within Mn and Ma+1 can be bounded by en and cp.
Proof. First of all, state value function VM= S > IR defined as V&((s) = E 7're/7,,00 = s). Notice that, since Tr E [0, Rmax], we have
R 1-7
(s S Rmax
(12)
Furthermore, Bellman policy operator Th is defined as follows, when applied to some vector f E RSS)
(TM)(s) =E E (+l)) R(s a) ((
(13)
Here (:--) is dot product. The Bellman policy operator is a 7-contraction under to0 norm (Puterman, 2014): for any f. E R|
|TE - Th rllff f'llo.
(14)
VM is the fixed point of TM meaning TMVA VM.
12

---

Learning to Reuse Policies for State Evolvable Environments
Inspired by Jiang al. (2015), first prove that
ER M VM. o fo llo 1
+ 7ep 7epRomxx
(15)
1 y 2(1 - -7)
Here, (VMi, f (8n-) = f. (1s.. M. (sn)) is defined over Sa+1. To prove this, we first apply 7-contraction and fixed-point property of Bellman policy operator. For simplicity, let T, = TM.
" - VM, o fii'loo || Mn11
= =ll(Tare V
Ma+i -TT+41 (V,,, o fn + (Trit (V f Vi. o 11
(16)
I|(Tatt vinel -Tatte (V o lo =I((Tut1 " (V o fm - VM, o ll
$ V Ma+1 - VM, o fa + +11(TT+] (V o so - -Vir, o 11o-
This implies that
| 1
Vāa fn 1bo
- (Tu+1
1
( 1
(V fT Mr, fall
(17)
1
(TTTe (V o (ntt (T VM, smx 11=l
For notation simplicity, let (s) E (a R(s, a), PT()) == (n and on fr² are abbreviation for a ~
(-15)) and 5n 1T- (.184t)) Then, for any 8841 E Sn+1,
(T (V fn (Sn+2) (Til VMa ) o fn (sn+1)l
=I(T (( Mn o fo ( - (TTT M.. )o (f (f=²)) (sn+1)]
I (sn+1) + 7 X (8n+1), VM, o f - E E 1 11t (sn)) - (B, [P (sn)), Vūr
S EawNo4i Ru+1 (2x+(1n+1,a) - E san 1 (nn a)]l + (P (Fr - (1: 1 (P (sn)], VM,,)
ER + 11F=11p (sn+1), Min) (E (0,,,,,'' fa [P (s V
n+1
= + [1 n+1
B s, (Pa (8n));
(Pan(s))), Virn)
Rmax
ER + 7
(B (B (Pn"( Vū.
2(1 7) 1)1
n+1
Rmax
<R + yE, 0,3, - Patt Pn a) |//|||1/1 VV 2(1 (1) 11100
Rinxx
I CR TCP 2(1 7))
(18)
In the steps above, we expand bellman update by definition and use cauchy-schwarz inequality to split dot-product with AP and value function. Lemma A.I and ép consistency applied to derive upper bound of the discrepancy of reward and transition functions.
Finally, according to the definition of the policy performance Jn(TT) MM, (sm.0).
E In 8n Pal. Jn (TT)
(Jn o 15 [x,))
of Mott (8m+1,0) - Vūl, (sn.o)
of (8n41.0) - -VV,, fir (5
(19)
Vi of - Van o f -
Mat1
+ 7ep Peax
< 7
2(1 7)2²
13

---

Learning to Reuse Policies for State Evolvable Environments
Proof for Proposition 3.3 We first introduce the following lemmas for the proof of Proposition 3.3
Lemma A.2. With the definition of state occupancy ds( - (1- 2 Pr(ss = s/T)). for any two different policies
n. x under the MDP M, we have:
dzz) S
7 1- Band [ (1B), (-1s),
(20)
Proof. We prove the lemma based on permutation theory presented in Schulman et al. (2015) and Xu et al. (2020). With (s((s)) = EeEa ls, ())(a)s) and do specifies the initial state distribution, we have:
d (1 -1)
Σ 7 Pr(#t = s|x))
(21)
1=0
=(1 - (I- 7p7
Let GA = (P=7 GB = (I- xp= ) then we obtain:
-1
GA GD G - GB
=GAlyPaA GB
(22)
=GGA(P - -Pan)GB-
With the equation, we have:
Drv(dG,, 1 / | d=
= 11 7) GR)ddoli
1(1 (11>2)Ga(PPA - - P
(23)
2
= - p
-
$ |GBll||PPA - Pr )d7a
Pre)daall 1|1.
Here GB is bounded as:
|GBll1 =1 7Px 8 -1l 1
8
pt.
(24)
| P
1=0
1 1-7
0=
We can also show that:
[[(P - -Pasddall
<
P (s'(ss)ss) -
8,8'
Σ P(s' ((s - dsn(s)
(25)
$ Σ
- (a(ss))dd))s)
8,0,8'
Σ (a)((e)
n (als) (a)ss)
2E 2E (-1s)
We complete the proof by substituting Equation (24) and Equation (25) into Equation (23).
14

---

Learning to Reuse Policies for State Evolvable Environments
Lemma A.3. With the definition of state-action occupancy Pa (8,0) - (dlss)d for any two different policies xA under the MDP M, have:
Div(Paa.Pxe) $ 1- Band, (Drv(T (11),
(26)
Proof.
Div(PTA,Pre) 2
Σ Pana -
(0,0)
π (als)dda x (a(s)d=nn())
(T (TT (als) (alas)))d + (4))))))
(27)
π (als) s) π (als)) ( (s) Σ ((d dx (s)) Σ
x (als)
= Bgnd A DrV (TT (-1s), π (-18))] + Div(dxA,
- Esndea [rrv ( (-1s); म " (-18))]-
We complete the proof with the last inequality following Lemma A.2.
Based on the above conclusions, we show that the performance gap can bounded by divergence of two policies. For simplicity, assume f-= and In deterministic and the conclusion holds by expanding the case of expectation.
Proof. Following Schulman al. (2015), we can rewrite the expected return as:
J(x)
R(se, at)
Pr(s: = s/T) Σ T(als)
(28)
Pn (s, a(s
t= 8.a
E p. (s.a) R(s,, a.)..
Combining Prop. 3.2, Lemma A.3 and the condition, we have that:
Ja+1 f Inti(Tn o 9nn)
Σ
(P (8nt1 a) Px ogn (snt1, (a) a)l
gm+i
2Rmax DTv Pr
2Rmax
(29)
E EDDTV((TT o so (-1s0t)) (n o gn g (-16
(1 7)²
2Rmax
E (LDTT(TT no (-1g
(1 7)²
2Rmax
7)2
15

---

Learning to Reuse Policies for State Evolvable Environments
Finally, combining Equation (29) and Proposition 3.2, we have
(Tn) Jn+1 (Tn o gn)l
= ( Jn+1 o fn Inti(Tnn fn J Jn+I( gn)l
- Jn (T))
- Jn+1(Tn o fn ) + |Jnt1(TT o fn Int1 gn)l
R
2(1 -7)² + (1 (1-
7 Rmax
2 Rmax
(30)
ER
(yep 4n) Rmax
2(1 - 7)²
This completes the proof.
B. Detailed Description of the Environments and Baselines
B.1. Environments
Ant
HalfChestah
Hopper
Walker
Figure Mujoco environments used in this paper
Gym Mujoco suite: MuJoCo (Todorov et al., 2012) (Figure is high-fidelity physics engine designed for detailed and efficient rigid body simulations with contacts. It is widely used for benchmarking RL algorithms. Agents in MuJoCo environments receive vectorial state inputs and output continuous actions. For comprehensive evaluation in our study, we have selected four tasks from the Gym MuJoCo suite:
Ant: The Ant is 3D robot consisting of one torso with four articulated legs, each with two links. The objective is to coordinate the movements of the legs to navigate towards specified direction.
Hal ahhettah::: The HalfCheetah 2-dimensional robot consisting of links forming a spine, with joints allowing articulation. The challenge is exert torques on the joints to propel the cheetah forward as swiftly as possible.
Hopper: The Hopper is ww--dimessiona one-legged figure that consists of four main body parts the torso at the top, the thigh in the middle, the leg in the bottom, and a single foot on which the entire body rests. The goal to make hops that move the forward direction.
Malker: The Walker is two-dimensional two-legged figure that consists of four main body parts single torso at the top, two thighs in the middle, two legs in the bottom, and two feet attached to the legs. The goal is to coordinate both sets of feet, legs, and thighs to move in the forward direction.
BankHeist
Freeway
Pong
RoadRunner
Figure 7. Atari environments used in this paper
16

---

Learning to Reuse Policies for State Evolvable Environments
proprioception (m)t(m)) o'jonnt (sad/s)
rootz(m) (m) Lup (rad) Ojoint, (rpm)
Addition of Sensor
Axis Rotation
Removal of Sensor
Wajaiin, (rad/s)
Oijiin, (rpm)
Distracting Objects
Removal of Sensor
Addition of Sensor
Raw images
Figure The state space evolution of experimental environments
Atari games: The Atari 2600 emulator (Figure 7) was introduced as an RI platform by Bellemare et al. (2013), offering a high-dimensional, pixel-based image input across a varied and complex set of tasks originally designed to challenge human players. We have selected four Atari games for comprehensive evaluation:
BankHeist: A bank robber wants rob many banks as possible. The police chases him and will appear whenever he robs a bank. The robber controls getaway car and must navigate maze-like cities while avoiding the caught of police, er the dynamite you have previously dropped.
Freeway: The agent's objective is to guide the chicken across lane after lane of busy rush hour traffic.
The agent controls the right paddle, and competes against the left paddle controlled by the computer. The agent each tries to keep deflecting the ball away from the right goal and into the left goal.
Pong:
RoadRunner: The road runner can walk in any direction and jump to outrun the opponent while avoiding the hazards of the desert.
B.2. Evolutions
Table 2. Evolving functions applied in state space of Mujoco and Atari environments, respectively
f2 rotate (30 + 5) add sensors rotate (30 = 5)
f1
fs
fa
Mujaco
remove sensors
rotate (30 + 5)
linear mapping w/ noise
Atari
remove sensors add moving objects
add sensors
linear mapping w/ noise
We here present the detailed evolutions applied our experiments. The process illustrated in Figure B.2 and specific setting of the evolutions are presented Table 2. evaluate the continual adaptation performance different methods under various orders of evolutions.
B.3. Baselines
RL-GAN proposes the analogy-based zero-shot transfer via image-to-image translation. Specifically, it assumes to unpaired data from the source and target domain beforehand, and achieves unaligned image-to-image translation using the Cycle-Consistency GANs. For our experiments, given access to paired data in dataset D.., we implement RL-GAN using the Pix2Pix framework, which has shown better results than the original CycleGAN in our context.
LUSR learns a latent unified state representation that is consistent across multiple domains. It applies Cycle-Consistent VAE ito disentangle the original state s into a latent state representation composed of & and 8, which are nomain-ggener and Nomain-specific respectively. The policy will focus on the Nomain-generrs embeddings to make decisions. In the SERL problem, we learn disentangled latent representations and the offline policy using the collected datatset D,,.
17

---

Learning to Reuse Policies for State Evolvable Environments
PAD makes the use of self-supervision allow the policy continue training after deployment. We decompose the policy into (Te; May ws), where first have intermediate embedding & (sa) and makes decision based on de Ta (ex). (.(:,)) takes and 6 as input to predict the resulting action dę- During the training stage, we optimize the RL and self self-supprvised objectives jointly to learn the policy. During the deployment stage, states are collected to optimize the self-supervised objective alone. In SERI problem, we learn the original policy via optimizing:
((T) Lss(Tee TT) + (Te, ma); (Te mT) = (T (F( Th (8(++))))
(31)
where / is the mean squared error between the ground truth and the model output in Mujoco and cross entropy in Atari. Then in evolving stage, we optimize the policy with the self-supervised objective Lss alone.
Offline learns a new policy for each Ma+1 using data D, collected from the old policy. We choose sota offline RL methods: TD3+BC and CQL for Mujoco and Atari, respectively.
FPT proposes framework for Few-Shot Policy Transfer between two domains through State Mapping and Behavior Cloning. It uses the Cycle-Consistency GANs which aims at map clone the successful source task behavior policy to the target domain. We extend it to SERL problem by replacing alignment.
GANs with Pix2Pix for better
CUP applics a critic-guided mechanism to reuse prior policies. It dynamically selects the policy with maximal one-step improvement at each state, forming theoretically guaranteed guidance policy. The target policy is then regularized to imitate this guidance while preserving exploration. our implementation, pretrained source policies guide the target policy before state evolution. The target policy is subsequently added to the source policy pool for future reuse.
C. Algorithm and Experimental Details
In this section, we describe the detailed designs and techniques used for different modules of Lapse. We also provide the hyperparameters used in Lapse for consistency.
C.1. Robust Training
Lapse requires robust initial policy mo for efficient reuse and offline adaptive learning. We aim to improve the robustness by minimizing:
S
S
= EE
max (e"ETi(e")
(-ss", TT. (-15")))
(32)
To deal with the intractable objective, we apply Wocar-PPO (Liang et al., 2022) and RADIAL-DON (Oikarinen et al., 2021) for Mujoco and Atari, respectively.
Wocar-PPO built on PPO (Schulman et al., 2017) which aims to optimize the policy with:
LRL = E - min (p:A(s,, az). gt Ast, a:))]
(33)
where Pe: := =(a,lss)
[as is importance sampling ratio, g: = clip pr, 1 - 1,,,, 1,,,11111 1), is a hyperparameter, and A(st, az) is an estimate of the advantage function A(s,, az) = Q(st, ae) Vst).. It also includes term that minimizes the loss of the value function and rewards the entropy of the policy. Wocar PPO introduces worst-attack Bellman operator to estimate the lower bound of the policy value under the worst-case attack by minimizing the following estimation loss:
Lea = E((((s))) (ra +
min
Q(s(+1,4)))
(34)
aEA.ndu
where Ando (s, x) = {a E 3 (........) = a). The worst action is selected through Interval Bound Propagation (IBP) (Wong& & Kolter, 2018). With the worst-case value, it optimizes the policy select actions that not only achieve high natural future reward, but also achieve high worst-case reward under adversarial attacks:
Lws R min (p,Q, (st, az). g 9, (8t, at))]
(35)
18

---

Learning to Reuse Policies for State Evolvable Environments
Algorithm recon train(n, Dr: mn))
1: Input: current stage Tn.
2: Initialize In and dn which are parameterized by gg. and dd respectively.
3: for ißer to max do
4: Sample batch ((s(,s(+1)) from Dn--
5: 87 +- (sp
6: Compute GAN according to Equation (1).
7: Optimize d, with 0d, <- dd, + Vos,, C
8: Compute L and fett via Equations (2) and (3).
9: Optimize with 9r. t fg. Vonn Lrcco
end for
Return #++1 TA O 9n-
Furthermore, it regularizes the policy network with a carefully designed state importance weight:
Leg (() max B, (B)
(T))
(36)
where us) is state importance weight defined wss) max E ((, a1) min Q(s, a2). which is approximated by V(s)) (8+ at)- Combining the above objectives together, we have the overall Wocar-PPO objective:
Lwocar LRL + KustCwaat Creg
(37)
where Kwst and Kreg are hyperparameters balancing between natural performance and robustness.
RADIAL-DON is built upon Dueling-DQN which splits Q-values into: Q(st, a4) VQ(s) Aq(s, and optimizes the
Q network with the objective:
RR = E[(Q(st,, at) - (re + max (1+11, d))))
(38)
a'
where Qigi is a periodically updated target network. RADIAL-DON designs regularizer minimize overlap between output bounds of actions with large difference in outcome. If there is overlap, the original action's Q-value is guaranteed to be higher than others even under perturbation, the agent won't 't change its behavior under perturbation. Mcanwhile, it
focuses on actions with different Q-values since taking a different but equally good action under perturbation is acceptable. RADIAL addresses it by adding a weight Qum(s, 01, 02) = max(U Q(s, 02) - Q(s, a1)}. Then, it defines Q(8, a, max;e (3 a) and Q(s, e) = minseB (s) Q33, a), and the overlap between two actions: Oul(s, a1, 02 E) == max (0, ((s, a2, <)) Q(s, 01, E) + where 17 = 0.5 Quir a1, 02). The final loss function is as follows:
Ladv = E
Σ LaEA
Qum(Er, âi, at Oelss, ăe, an)
(39)
Incorporating it with the original DQN loss, we have the overall RADIAL-DON training objective:
CRADIAL KCRL + (1 R)
(40)
where s is hyperparameter controlling the trade-off between standard performance and robust performance with value between and 1.
C.2. State Reconstruction Model
During the th state evolution, we learn a reconstruction model mapping the new state space S.+1 back to S,, using collected paired data, so as to reuse the old policy मn, We propose to use conditional GANs to achieve the state reconstruction as described in Alg.
C.3. Offline Adaptive Policy Learning
To learn new policy which takes actions with new state space M,+1 during state space evolution. we choose TD3+BC for Mujoco and CQL for Atari. Considering that the policy is vulnerable to perturbations of inputs, we make use of
19

---

Update target network with: Learning to Reuse Policies for State Evolvable Environments
conservative smoothing technique introduced Yang al. (2022) make the policy network smoother. The policy will therefore be more robust and facilitate future reuse with the state reconstruction model. Specifically, we generate bounded neighbouring area 1
{ - 222 - e S around s We simply denote it as B in the following. Then compute the upper bound of the variation distance between n (8n+1) and T ($n+1), 8n+1 B according to Equation (5). We use mean square error as the variation distance metric so that the loss function is as follows:
C rohust
E
max | in- n
(-1oat))) nn+1 2 (-18=(1++++11 ')1
(41)
Algorithm 2 off.train(n, Dn, Tay Bn41)
1: Initialize a new policy "n= and Q network Qn-1, parameterized by ox and Φo, respectively.
2: Set target parameters equal to on and ΦQ.
3: Compute neighhouring area bound e according to the scale of states in Da.
4: for iter to T max do
5: Sample batch {(57+1 aa a,+1 .ee (Tt. s1+11) from Dn.
6: Learn Qn+1 and optimize ΦQ via the TD3 algorithm.
7: Compute Ca+1 via Equation (6).
8:
if iler mod policy delay- 0 then
9:
Generate bounded neighbouring area 8 (s"+1).
10:
Compute robust via Equation (41).
11:
Compute L via Equation (7).
12:
Optimize 6n with Φx + oz Vo. La+1
13:
Otarg. f- Troz@targ.r + (1 - TIDs)
Qtarg.Q TrDz@targ. + (1 TTDD)
end if end for 16:
14:
Return 1-
As for CQL applied in Atari games, the loss function optimized as follows:
fn+1 -341 LRL.
log Σ exp ((st( Q(se, ax)
(42)
where CRL the standard DQN objective is defined in Equation (38). We also implement it on Quantile Regression- as defined in Section DQN (Bellemare al., 2017) for effectiveness. We schedule 3a41 3max exp(-7 -n) nn)) 4.2.
C.4. Overall Training Algorithm
In SERL problem, we split into the training phase and deployment phase. During the training phase, To is allowed be learned under Mo in an online manner. To improve the robustness, apply Wocar-PPO and RADIAL-DON into Mujoco and Atari environments, respectively, where the objective is described in Equation (37) and Equation (40). Before evolving into Mo+1, we will learn the adaptive policy Ta+1 following the paradigm of Lapse as is introduced in Alg. 3. Meanwhile, for Atari games that only instantiate Q-function, we do not distinguish я and Q here and make decisions through arg max, Tn+1 (als). Furthermore, to calculate the distance between two policies T T2, we approximate it through [x (m((1)), 22 (-[))))
20

---

Learning to Reuse Policies for State Evolvable Environments
Algorithm 3 Lapse
Input: Environment Env, current stage n 2 O, old policy Vn, size dataset Bn
2: Schedule the coefficient for offline adaptive policy learning: Su+1 f- Pmax (1 exp (-r n)).
3: Collect D,, {(}] + (t, rt, 81+1, (tt1) s } from Env within B, episodes using Th.
4: m recon.train(n, Dns T.). rfff
5: "a+1 < off frfftrain(n1, Dn; Tn. Pn-1).
6: Compute the distance DT mn41) and (TT off "2+1/
7: Compute 1n-1 according to Equation (9).
8: An+1 + Kp+17n+1 decon + (1 Kn+-1 1
9: Mn+1 +- Pruning(TT*: ((Optional))
10: Return In+1-
C.5. The Hyperparameter Choice of Lapse
As To is trained following the paradigm of Wocar-PPO and RADIAL-DON, default parameters in the frameworks are used. introduce the hyperparameter choices of the left parts in Table and omit the subscript n + for simplicity.
Table 3. Hyperparameter choices of Lapse
Env
Hyperparameter
Value
value in Lip
2
À in Lieson Imax in Lost T in Lut rohnet Leet in
10
2.5
0.5
0.1
0.001
Mujoco
policy update interval
2
Trccon
10000
Tooff max
10000
learning rate target update TIDS
0.005
7
0.99
value in Lup A in Creson Braxe in Latt T in Lott mbust in Loet in ema coefficient
10
0.1
0.1
3/255
Atari
0.995
ema interval
100
Trecon Toof! max
20000
40000
learning rate
number of quantiles QRDQN
200
target update interval
400
7
0.99
21

---

Learning to Reuse Policies for State Evolvable Environments
D. Additional Experimental Results
D.I. Complete Adaptation Performance
Figure illustrates the compelete continuous adaptation of various methods to the evolvable state space by comparing the performance over different stages. The result is consistent with the conclusion in the main text. Despite the increased performance degradation over successive evolving stages, Lapse maintains superior performance compared with all other baseline methods.
D.2. Complete Training Curves Lapse
We present the complete training curves in Figure 11, where blue dashed lines represent the performance during new paired data collection. Note that the policy is deployed to the environment only after training for the current stage is complete, thereby preventing costly trial-and-error during adaptation.
D.3. Pruning Strategies
We define 1 Kn TT Teoon (1 - Kr, Taff in recursive way due the reuse of old policies in mron To demonstrate the contribution of each model, introduce occupancy weights co.++1 defined as:
Kip
= 0
G
1, 2, ...; Ka) 11 Ki:
(43)
tn+1,
= n +
Then expand the expression
Tn+1 = COTO go + Σ c.m. o 9i + CnT
(44)
where Ci = 9 O 9i+-2 O In+1- This requires us to store all the learned reconstruction models and the offline adaptive policies, which unbearable memory storage and inference delay with the increase the evolving stages. Take Mujoco environments Ant and Walker as examples, not all components Lapse ms are important as depicted in Figure This way can just use off n recon in Ant and h T0,1,5 #1:5 in Walker. The occupancy weight of one model will not grow in the only store the models that contribute lot. To mitigate the problem, propose prune
future by definition, so that the heavy compound policies based on the decomposed non-recursive expression of xn+11 as shown in Equation (44).
Ant
Walker
7T0
n4
Figure 9. The occupancy weights Lapse policy in Mujoco
Algorithm 4 Pruninn((TTt) (1n+1)
1: Decompose and compute the occupancy weights {(c:) via Equation (43).
2: for = 0 to n n++1 do
3: if c;
then
4: C; 0. {Ignore the model that contributes few)
5: end if
6: end for
7: Normalize the occupancy weights:
&
Gi
0, 1,..,,,..
n1
ビ Cj
8: Return the pruned Lapse policy with {(()) via Equation (44).
22

---

Learning to Reuse Policies for State Evolvable Environments
Lapse
RL-GAN
LUSR
PAD
Offline
FPT
CUP
Ant
HalfCheetah
Hopper
Walker
Normaizzdd Average Retum Nomalized Average Retum
100
100
100
100
80
80
60
60
40
40
20
20
20
20
0
BankHeist
Frocway
Pong
RoadR unner
100
100
100
100
80
80
40
40
20
20
20
20
Evolving Stage
Evolving Stage
Evolving Stage
Evolving Stage
Figure 10. The adaptation performance of the learned policy as the state space evolves
Lapse
Lapse during Exolveing
LUSR
Oftinee
Ant
Normalizad Return
100
HalfCheetah
Normalized Return
100
75
Hopper
Normalized Return
100
75
50
25
Walker
Normaiized Rebarn
100
25
Evolving Stage
Figure 1I. Complete training curves of Lapse
23

---

Learning to Reuse Policies for State Evolvable Environments
D.4. Ablation Studies
Ablation on Different Components We dissect the impact of each component within Lapse as depicted in Figure 12. W/o offfine and W/o recon break down different tasks, illustrating that neither of them can handle continuous adaptation problems alone. Wo robust's poor performance on out 4 tasks again emphasizes the necessity of the robustness. Wo kappa shows inferior adaptation capability compared to Lapse, manifesting the benefit of the automatic ensemble weight adjustment. To analyze the contribution of each component, we conducted a series of experiments in which components were incrementally added, culminating in the complete version of Lapse. Figure 13 shows this ablation study, demonstrating the impact of each added component and verifying their necessity.
Eapec
Wio kappe
Wio offline
Wio neam
robao
Pong
BunkHeist
Aut
Walker
100
100
CandiintArruuu Aveuu ketum
Namaltai Ricians
to
60
60
40
40
40
40
20
Evelving Sagns
Bvclving Stages
Evelving Singes
Evolving Stuga
Figure 12. The overall ablation studies
Ant
Walker
base (recon)
base (offline)
robusteecon
robustatioo
base offline
base
base += robustamhhe
base += robustrecon
base +- kappa (Lapse)
base +- kappa (Lapse)
60
30 Normalized Average Return
90
Normalized Average Return
Figure 13. Demonstration of cach component's 's contrihution
Ablation Ensemble Methods We employ linear mixing because determining weights via Equation (9) and performing linear mixing are cost-free and highly efficient. We evaluated how various ensemble methods affect the performance of Lapse. The results appear in Figure14, 14. We do not plot the learning curve for stacking ensembles, as their returns never exceeded 1000 even after more than one million training steps. Alternative methods, such as hypernetwrkss, also showed no significant improvement.
Ant
HalfCheetah
Hopper
Walker
5600
4500
5000
5400
4000
4000
5200
3500
3500
3400
3000
3000
3300
0
25
50
75
100
25
50
75
100
25
50
75
100
0
25
50
75
100
Steps (K)
Steps (K)
Steps (K)
Steps (K)
Figure 14. Test performance of dynamic ensemble weighting based on a hyper-network. The dashed line represents the performance achieved by linear mixing using the kappa from Lapse, while solid curve demonstrates performance when dynamically learning ensemble weights via online RL using heper-network, which takes the state as input and outputs weight vector used to weight each action dimension for ensemble aggregation.
24

---

Learning to Reuse Policies for State Evolvable Environments
Table Comparison of average performance R, storage N and inference delay T' between vanilla Lapse and pruning strategies across 2 tasks. Results are averaged over 5 distinct seeds. The storage metries measured by thousands of parameters of r and the inference delay metrics are measured by the number of seconds it takes for n; to perform 10" inferences in our experimental environment.
Mo
MI
M2
M3
M4
Ms
Ant
vanilla
100.00
101.70
93.63
86.51
80.62
69.27
R
pruning
100.00
92.38
94.94
90.97
83.52
77.91
vanilla 40.98
156.04
271.10
386.17
501.23
616.29
N
pruning
40.98
156.04
133.61
133.61
133.61
133.61
vanilla 0.36
0.68
1.01
1.33
1.66
1.98
T
pruning 0.36
0.68
0.54
0.54
0.54
0,54
Walker
vanilla 100.00
98.70
102.17
69.33
75.36
70.74
R
pruning
100.00
95.62
87.90
86.71
76.41
78.32
vanilla
5.71
84.07
162.43
240.78
319.14
397.50
N
pruning
5.71
12.12
18.54
24.96
31.38
61.77
vanilla 0.14
0.46
0.78
1.10
1.43
1.75
T
pruning
0.14
0.28
0.43
0.57
0.71
0.91
Ablation on Pruning Strategies We compare average performance, storage and inference delay metrics between vanilla Lapse and pruning strategies with pruming Jhreshold = 0.2 across tasks. As shown in Table 4, we find that our pruning strategies save a lot of storage space and inference delay while ensuring excellent performance. In Ant, the pruning strategy only retains off of the latest stage when i > 1, which means that storage and inference delay remain lightweight and fast. In Walker, the pruming strategy only retains тo and all xę which saves most of the storage and nearly half inference delay. The above experiments show our pruning strategies maintain SERL scalability and performance.
D.5. Sensitivity Studies
We further conduct more experiments in Mujoco to investigate how hyperparameters influence the performance of Lapse. The results can be seen in Figure 15. We selected set of hyperparameters that performed well across tasks.
Additionally, in order to explore the impact of the size of D, on method, we conduct experiments with size ranging from 6 to 18 trajectories. As shown in Figure 16, our method is robust to the various size, which is beneficial to obtain stable and good performance during the deployment stage.
D.6. More evolving stages
To provide more comprehensive evaluation of Lapse's adaptation performance over an extended number of evolving stages, we expanded the experiment to include 15 stages-threr three times the number in our initial tests. These additional stages were simulated using scalar multiplication, with pruning strategies employed to maintain scalability.
Figure 17 showcases the results of this extended evaluation. Lapse demonstrates commendable adaptation performance in three of the four tested environments. However, in the Walker task, Lapse's performance declines after the 7th stage. This drop in performance may be related to the inherent offline learning inefficiencies of TD3+BC within the Walker environment, as noted by Fujimoto& & Gu (2021). It is anticipated that substituting TD3+BC with more efficient algorithm could preserve Lapse's adaptability in this scenario.
25

---

Learning to Reuse Policies for State Evolvable Environments
Aut-T
Ant-dmax
Ant-Aggan
Ani-Tyy
Nomalieed Ronum
Nomol Rauun
Nomalaac Rotur
100
Nomme Ruturn
100
60
80
80
80
80
60
60
60
100
01001
0.005
0.1
Parsmeter Value
Panetcler Vilue
Parameter Valne
Parumnter Valne
Hallt (Cheethh-T
HalfChettah-Orme
HallCheenh Augan
HallChoethh-
100
100
Nomalizos Ranum
Normatins Retunt
Normaludd Netum
Noemalion Batum
80
80
80
80
80
60
60
60
100
0000
Parameter Value
Purmeter Value
Parameter Value
Parameter Value
Hlopper
Hopper Augan
Nomalizal Roram
Nomalizal Ratunn
100
Nomalizz Retun
100
Nomulizal Raturn
100
60
80
80
80
60
60
69
3.5
0.1
Parameter
Parumeeer
Parameter
Parameter Value
Walke2d-T
Walker26-Bmw
Walkes2d-Acpan
Walker2d-Tir
Nommlize Retom
100
Nommiizza Retum
100
Nommlzzz R.tum
100
Nommsizea Retum
100
80
40
80
60
60
60
Parameter
Parsmeler
Value
Figure The overall parameter sensitity studies
Ant
HalfCheetah
Hopper
Walker
100
100
100
100
Normalized Betum
Normalized Retum
Normalized Recum
Normalize Recum
80
801
80
30
60
60
60
Parsmeter Value
Parameter Value
Parameter Value
Figure 16. Sensitity studies of the size of datatsct D,,
100
25
Normalzzed
25
Ant
HalEChestah
Hopper
Walker
Evolving Stage
Figure 17. Continual adaptation under evolving stages
26


Reviews:
[{"review_id": "2DlbSkj4Dz", "reviewer": "Reviewer_Uqfd", "summary": "This paper introduces the problem of state evolvable reinforcement learning (SERL) and proposes a method called Lapse to address it. \nThe key contributions are as follows:\n* Formalization of the SERL problem, where the state space of an environment evolves over time due to sensor changes.\n* Lapse takes a two-pronged approach to policy reuse. On one hand, it directly reuses old policies via state reconstruction to handle vanishing sensors. On the other hand, it reuses behavioral knowledge through offline RL to better utilize new sensors.\n* The method also incorporates automatic adjustment of ensemble weights to combine the two policy reuse approaches effectively. \n* Additionally, the authors provide a theoretical analysis of performance gaps caused by state evolution uncertainty.\n* Empirically, Lapse demonstrates significant performance improvements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and thoughtful comments. We hope our responses can relieve your concern. We present the additional experimental results in https://anonymous.4open.science/r/Lapse-13F8.\n\n### Q1: Prior knowledge about sensor changes\n\nWhile this assumption does not hold universally, it is practical in many real-world systems. For instance: Wear-and-tear or battery depletion in industrial sensors is often predictable, allowing preemptive deployment of replacements.\n\nMeanwhile, the assumption of accessing a large amount of test-domain data is common in domain adaptation [1]. In our work, the prior knowledge about sensor changes only offers a small set of paired test-domain samples. \n\n### Q2: Access to maintenance information\n\nAccess to maintenance information is motivated by the fact that many real-world systems undergo scheduled sensor checks, replacements, or recalibrations [2]. The developers are thus allowed to capture paired data before sensor changes, which is essential for our method to adapt policies effectively with minimal online exploration. For example, large-scale industrial systems often have planned maintenance schedules to update or replace sensors [3]. The system should keep operating during the aforementioned periods.\n\nThe problem pointed by the reviewer is valuable as there are many abrupt changes in the real world, which is also pointed out as the limitation of Lapse in conclusion part. Some works have also focused on dealing with sudden changes in dynamics [4]. However, these methods are only applicable for in-distribution changes and fail to cope with totally unseen scenarios. Online fine-tuning based adaptation is an alternative direction without the need of maintenance information, but it inevitably introduces costly trial-and-error. In future work, combining Lapse with these methods by restricting the policy adaptation in a safe and trustworthy region is of promise.\n\n### Q3: Changes of dynamics and rewards\n\nOur work mainly focuses on the continual evolution of state space, which is built on SERL formulation. Paired states cannot be created if dynamics/rewards change, so we fail to directly apply Lapse to such scenarios. However, to investigate the potential of Lapse, we simply extend it to a more complex multi-agent scenario, SMAC, by combining it with a multi-task MARL method, MaskMA [5]. As the paired transitions do not exist, we assume access to a limited offline dataset when encountering new tasks. **Figure 1** demonstrates Lapse's potential for adaptation to unseen tasks when combined with multi-task/meta learning methods.\n\n### Q4: Real-world experiments\n\nTo further investigate how Lapse performs in real-world applications, we here choose Highway-env as alternatives for real-world applications including autonomous driving. As shown in **Figure 2**,  Lapse consistently outperforms other baselines in adaptation performance.  For more realistic physical embodied scenarios involving robotics, we leave it for future work.\n\n### Q5: Lack of complete training curves, compute analysis and ablation study\n\n1. We list out the complete training curves in the **Figure 3**, where blue dashed lines represent the performance when performing new paired data collection. It should be noticed that the policy is deployed to the environment for the next evolving stage until finishing the training, thus preventing costly trial-and-error during adaptation.\n2. We present the compute analysis during training in **Table 1** by computing the number of parameters required for training and FLOPs during each stage. Our method demonstrates better performance with no significant extra computation cost.\n3. The ablation study that demonstrates what the addition of each component does is presented in **Figure 4**, verifying the necessity of each component.\n\n[1] A Comprehensive Survey on Source-free Domain Adaptation\n\n[2] Open-world machine learning: applications, challenges, and opportunities\n\n[3] A Methodology for Online Sensor Recalibration\n\n[4] Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy\n\n[5] MaskMA: Towards Zero-Shot Multi-Agent Decision Making", "claims_and_evidence": "The claims made in the paper appear to be generally supported by clear evidence. The formalization of SERL is mathematically sound, providing a solid foundation. The Lapse method is described in detail, clearly explaining its components. The theoretical analysis justifies the approach and provides bounds on performance gaps. Empirical results show substantial performance improvements over baselines.\n\nHowever, some claims need further scrutiny. The assumption of access to maintenance information seems strong, and the paper needs more motivation on why this specific setting is important. Without strong motivation, the applicability of the method to real-world scenarios is unclear. The limitation of considering only state evolution without changes to dynamics is also a significant constraint, potentially limiting the generalizability of the approach. \n\nFurthermore, the lack of evaluation in a real-world environment where state evolution occurs as expected weakens the overall evidence for the method's practical utility. These limitations impact the strength of the claims made regarding the method's effectiveness in real-world SERL problems.", "methods_and_evaluation_criteria": "The proposed methods, in general, appear sensible for the SERL problem. Using GANs for state reconstruction is a reasonable approach to handle vanishing sensors, addressing a key challenge. Incorporating robustness regularization addresses potential issues arising from reconstruction errors, enhancing the method's resilience. Leveraging offline RL to learn from limited experience in new state spaces is a clever way to adapt to new sensors, making efficient use of available data.\n\nThe evaluation criteria using MuJoCo and Atari environments are standard benchmarks in RL research. These environments seem appropriate for demonstrating the effectiveness of the proposed method across different types of state spaces, including both vectorial and image-based representations. The baselines are also chosen appropriately.", "theoretical_claims": "The paper includes theoretical analysis in Section 3.4. I did check the Propositions 3.2 and 3.2 and their detailed proofs in Appendix A. The high-level reasoning about bounding performance gaps based on evolution uncertainty and reconstruction errors appears sound.", "experimental_designs_or_analyses": "The experimental design seems reasonably well-structured. Using both MuJoCo (vector based) and Atari (image-based) environments provides a diverse set of test cases, covering different state representations. Testing the policy in environments with multiple unknown state space evolutions evaluates the method's adaptation capability. Comparison against existing adaptation and transfer approaches provides context for the claimed performance improvements.\n\nHowever, the absence of a compute analysis compared to the baselines is a significant omission. Without information on computational costs, it is difficult to assess the practical feasibility of the proposed method. Moreover, the limited presentation of training curves for all environments makes it challenging to fully evaluate the method's performance and stability across different settings. These limitations hinder a complete assessment of the experimental results.", "relation_to_broader_scientific_literature": "The paper positions its work well in relation to several areas of RL research. It builds on work in policy reuse/selection and adaptation to new MDPs. It leverages ideas from robust RL to address uncertainties in state reconstruction and incorporates offline RL techniques for learning from limited experience.", "essential_references_not_discussed": "This is a well-written paper and most of the related works section covers the essential background and research papers in this area.", "other_strengths_and_weaknesses": "Strengths:\n\n* Novel problem formulation (SERL) that addresses a practical challenge in deploying RL systems.\n\n* Creative combination of multiple techniques (GANs, robust RL, offline RL) to address the problem.\n\n* Both theoretical analysis and empirical evaluation are provided.\n\nWeaknesses:\n\n* The assumption of prior knowledge about sensor changes may limit applicability in some scenarios.\n\n* The assumption of access to maintenance information seems strong, and the paper needs more motivation on why this exact setting is important.\n\n* The limitation of only considering state evolution without changes to dynamics/reward structures is a constraint.\n\n* Lack of evaluation in a real-world environment where state evolution occurs naturally.\n\n* Absence of a compute analysis compared to the baselines.\n\n* Limited presentation of training curves for all environments.\n\n* There should be another figure after Figure 10 that demonstrates what the addition of each component does for a complete ablation study.", "other_comments_or_suggestions": "* More details on practical implications and potential real-world applications of SERL would be helpful. \n* A discussion of computational requirements and scalability issues would be valuable. Including a real-world example would greatly enhance the paper's impact. \n* Adding training curves for all environments would provide a more comprehensive view of performance.\n* A compute analysis comparing Lapse to baselines would offer insights into efficiency and scalability.", "questions_for_authors": "* Can you provide more motivation for assuming access to maintenance information and why this specific setting is important? \n* Have you considered extending the method to handle changes in dynamics? \n* Are there plans to evaluate Lapse in a real-world environment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ooe01SkUAn", "reviewer": "Reviewer_cpBN", "summary": "The authors propose the framework Learning to reuse policies in state evolution (Lapse) to solve the issue of model degradation when there is an explicit change in the environment representation that is received by the agent in the form of input. They formalize this scenario as State Evolvable Reinforcement Learning (SERL): a RL environment where there is a sudden change in visual representation. In the majority of RL algorithms, the model would face an extreme degradation in performance when the input representation changes like this, but Lapse avoids this by training a robust RL policy and using the rollouts from this policy as an offline dataset to train a new policy. The old policy and the new policy are then aggregated together to create a new policy for the evolved environment. The authors demonstrate strong results against a number of baselines, demonstrating the ability of their proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for careful review our paper and providing constructive comments and suggestions. We offer some clarification to your questions here, and we would appreciate any further comments you might have.\n\n### Q1 Changes are directly signaled to Lapse\n\nYes, the change in the environment should be directly signaled in our current SERL problem formulation. Access to sudch information is motivated by the fact that many real-world systems undergo scheduled sensor checks, replacements, or recalibrations, where developers can prepare beforehand. For example, large-scale industrial systems often have planned maintenance schedules  for updating or replacing sensors [1]. These scenarios allow developers to prepare for changes in advance, making our signaling assumption reasonable for practical deployment.\n\n### Q2 Lack of essential references\n\nThank you for bringing the references to our attention. The works referenced by the reviewer [2, 3, 4] and our Lapse all focus on the open-world/environment decision making problem. The sudden changes of environments are defined as “novelty” in these works, which emphasizes the abruptness of the change, while the “evolution” of state space in Lapse, although still unpredictable before deployment, can be scheduled beforehand for a **short** period. Specifically, [2, 3] detect the sudden change via knowledge graph or neuro-symbolic world model and achieve policy adaptation with imagination trajectories generated from the updated world model. These methods fail to be directly applied to high-dimensional, continuous and complex environments, where dynamics and rewards are difficult to be depicted with simple rules. [5] addresses this problem by extending it to latent-based novelty detection via DreamerV2 but does not study how to achieve policy adaptation in complex environments, mainly because the accuracy of imagination trajectories cannot be guaranteed. Our method takes a further step for the high-dimensional, continuous and complex environments via a novel solution, and we hope it could fill in gaps for open-environment.\n\nAside from the works mentioned by the reviewer, some other works focus on dealing with sudden changes in decision making, involving environmental dynamics [5] and teammate in multi-agent systems [6]. We will add the corresponding discussions to the paper in the revised version. Thank you again for pointing out the problem.\n\n[1] A Methodology for Online Sensor Recalibration\n\n[2] Detecting and Adapting to Novelty in Games\n\n[3] Neuro-Symbolic World Models for Adapting to Open World Novelty\n\n[4] Novelty Detection in Reinforcement Learning with World Models\n\n[5] Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy\n\n[6] Fast Teammate Adaptation in the Presence of Sudden Policy Change", "claims_and_evidence": "The claims in the paper are supported by proof (Section 3.4) or experimental results (4.2). All results were run and reported over a sufficient number of seeds.", "methods_and_evaluation_criteria": "The primary issue the authors seek to address with their method is the degradation of model performance as a result of changes to the underlying state of the environment. Their approach is a three-step process where overfitting to learned features in the current environment is actively penalized; the now more malleable policy is used to gather offline trajectories in the new/evolved environment; and the offline trajectories are then used to train a new policy for the evolved environment that is aggregated with the old one for the final policy.\n\nI believe this process to be well motivated for the issue the authors are investigating. I do have a question regarding the final step that I will ask later in the review.", "theoretical_claims": "I checked the correctness of equations 1-10, along with the definitions and propositions in section 3.4. I have also checked the expanded proof for Proposition 3.2 in the appendix (Section A) along with training details in Section D. I have found no issues with any of these.", "experimental_designs_or_analyses": "I have checked whether the baselines evaluated against made sense in the context of the author's primary research question, if the experiments were run over enough seeds to be statistically reliable, and if the evolutions within the environment were fair and cherry-picked. I have found no issues with any of the previously mentioned", "relation_to_broader_scientific_literature": "The author's work is relevant to the Reinforcement Learning community as an exploration in how to create robust and rapidly adapting policies in a changing environment. It is also relevant to the larger Machine Learning community as a whole.", "essential_references_not_discussed": "I do believe there is a non-trivial body of work by Mark Riedl's group (Specifically Jonathan Balloch and Xiangyu Peng) in Novelty Detection and Policy reuse that the authors have missed. Notably, how the authors of the former frame a 'novelty' seems extremely similar to the authors definition of an SERL: a \"sudden change in visual or state transitions\".  In the case of the latter, their work with roleplaying agents also touches on reusing policies to adapt to a new environment (in their case, a new character roleplaying 'persona').", "other_strengths_and_weaknesses": "Strengths:\n\n1.) The authors research question is well-motivated and grounded in a very real and serious issue: RL algorithms would not be deployable in the real world if they catastrophically deviated from their policy at state representation perturbations. \n2.) The authors use a straight-forward but well-motivated method (and valid) method for forcing a policy to not be overfit to the current state representation.\n3.) They rigorously prove the effectiveness of the state reconstruction model\n4.) They provide a thorough analysis on each baseline used in their evaluation and ensure that the changes in the environment are fair to all evaluated baselines\n\nWeaknesses:\n1.) It appears the change in state must be directly signaled to the algorithm for it to begin the adaptation process.\n2.) The missing related work mentioned above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1.) Does Lapse require for the change in the environment to be directly signal to it? My impression was yes but please correct me if I am wrong.\n\n2.) I believe the works mentioned in the \"Essential References not Discussed\" merit a reference, if not a slightly longer discussion on the differences between 'evolution' and 'novelty'. However, I am open to being convinced otherwise by the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m6KxD92ODB", "reviewer": "Reviewer_9wCY", "summary": "This work introduces introduces a framework to handle non-stationary state distributions during deployment of an RL agent. The authors are motivated by the deterioration or maintenance of sensors on real-world systems that do not have a constant fidelity. The proposed framework is structured around a State Evolvable RL problem where there is a series of MDPs $\\mathcal{M}_n$ in which the progression is defined by $f_n$. The agent has access to a series of datasets of the evolved data. The agent attempts to learn a policy that can reuse previous behavior as well as adapt to new state distributions. The framework proposes a combination of robust RL algorithms and explicit learning of the state evolution through reconstruction. The authors provide theoretical analysis and extensive experiments in Mujoco (state-based) and Atari (pixel-based) tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and suggestions. We provide our responses below and present the additional experimental results in https://anonymous.4open.science/r/Lapse-7846.\n\n### Q1. Performance of baselines\n\nThe baselines underperform because they cannot anticipate all possible state evolutions during training. Instead, they must adapt to new state space using only a small set of paired transitions collected by $\\pi_n$.  In general, domain adaptation methods require a large amount of data to extract generic representations for adaptation. CUP cannot enumerate all possible evolutions beforehand, leading to failure for direct reuse of source policies. We will expand this analysis in Sec. 4.2.\n\n### Q2. Linear Mixing\n\nLinear mixing is an efficient ensemble method, which has been applied in policy ensemble [1, 2]. We employ linear mixing because determining the weight via Eqn. 9 and performing linear mixing is cost-free and highly-efficient. We conduct experiments on how the way of ensemble affects the performance of Lapse.  Alternatives like hypernetworks showed no significant improvement (**Figure 1**). We also train stacking but it failed in 4 tasks (score < 1000 after 1M steps).\n\n### Q3 TV divergence\n\nSince $D_{TV}^2\\leq \\frac{1}{2}D_{KL}$, both divergences can achieve robustness. On the other hand, the loss is derived from Prop. 3.3, where performance gap is bounded by TV divergence. As discussed in Appx. D.1, the methods we use [4] do not directly optimize over TV divergence but take more efficient ways.\n\n### Q4 New data, cost and algorithms\n\nIn SERL formulation, the policy is deployed after training on the initial MDP $M_0$. During deployment, the state space may evolve for various reasons such as scheduled maintenance. Access to the information is motivated by the fact that real-world systems undergo scheduled sensor checks or replacements. For instance, if a sensor is nearing the end of its lifespan, a replacement can be installed in advance, allowing the agent to observe features from both old and new state spaces. The new dataset $D_n$ is collected by $\\pi_n$ utilizing only state features $s_n$ from the old state space, but paired states $s_{n+1}$ can also be reserved.\n\nOur method will not incur significant cost as the policy for data collection is $\\pi_n$ which performs well in $M_n$. Avoiding costly trial-and-error during policy adaptation is the primary contribution of Lapse.\n\nSince $\\pi_{n+1}^{off}$ is learned using data collected by $\\pi_n$, we choose offline RL algorithms. Online RL including off-policy algrorithms is not suitable because trial-and-error is not allowed in SERL.\n\n### Q5 Continuously learn a robust policy or direct map to evolution 0\n\nWhen the state space evolves to $ S_{n+1}$, the depolyed policy $\\pi_{n+1}$ is composed of both $\\pi_{n+1}^{recon}$ and $\\pi_{n+1}^{offline}$, where the latter does not require state reconstruction and maintains robustness via regularization.\n\nIn Appx. E.2, we employ pruning strategies and verify that the initial policy can be omitted via proper pruning in some scenarios, without the need for iterative map. Meanwhile, we conduct experiments by mapping evolution n to 0 via cycleGAN. **Figure 2** shows our method performs better than directly aligning in most circumstances as it can better utilize the paired data.\n\n### Q6 Paired data and possible enhancement\n\nDespite the limitations in some scenarios, it is still applicable for many real-world systems. Please refer to our response in **Q2 to Reviewer Uqfd for details** due to word limit. \n\nThe idea that automatically pairing data is a valuable direction for further research. Cycle-GAN [6] is widely-used for domain adaptation by introducing cycle consistency loss. It implicitly pairs data from different domains. However, as the experimental analysis presents in **Figure 2**, it fails to achieve satisfying adaptation performance with direct application. In future works, developing relevant methods for automatically paring using a small set of data will help extend our method to more general scenarios.\n\n### Q7 State features, space and observation space\n\nIn our work, state space is the fundamental element $\\mathcal S_n$ in the MDP $\\mathcal M_n$, while state features $s_n \\in \\mathcal S_n$ are the input to the policy for decision making. The change of sensors lead to the evolution of state space, together with state features. There is no distinguishment between “observation” and “state” because we do not consider the partially observable setting. We are sorry for confusion caused by some abuse of these concepts. We will replace observation into state for consistency and clarify two concepts in the revised version.\n\n### Q8 Nitpicks\n\nThank you for pointing out the issues.\n\n[1] Towards Applicable RL: Improving the Generalization and Sample Efficiency with Policy Ensemble\n\n[2] Seerl: Sample efficient ensemble RL\n\n[3] Efficient adversarial training without attacking: Worst-case-aware robust RL", "claims_and_evidence": "The main claim in the paper is that the Lapse method retains better performance through an evolving state space across domains. This claim is supported in the Atari and Mujoco domains in Table 1. The experiments were run over several seeds, and standard deviation is shown. The baselines are discussed in the appendix, but a nuanced understanding of why the baselines do not perform well compared to Lapse cannot be garnered from the paper. Having said that,  in the appendix the authors address more classic methods for training robust agents, such as Domain Randomization.", "methods_and_evaluation_criteria": "The main method is a combination of state evolution modeling, policy regularization, and adaptive policy aggregation. \n## Methods\nThe state evolution modeling is learned through classic GAN-style losses and is reasonable in this setting (Equations 2, 3). The policy regularization for robustness is from Equation 4 and includes a standard TV distance; typically, to regularize to a policy people often use KL divergence and I wonder why these authors opted for TV Distance. \n\nFor offline policy learning, the authors invoke TD3+BC (Fujimoto 2021), which is a method that balances optimism and dataset regularization. I do not understand why the authors opted for an offline RL method when the task is naturally online. Perhaps an off-policy method such as SAC would be more well-suited. \n\nFor adaptive policy aggregation, the authors do linear mixing of the robust policy and the offline RL policy according to a a weight defined in Equation 9. The linear mixing of the policies makes strict assumptions about the policy structure and an action's influence on the environment. It is not always the case that linearly mixing the policies will work; for instance, when the action distributions don't have joint support.", "theoretical_claims": "The theoretical claims (Section 3.4) are minimal in the main paper, but provide bounds on the return of the policy under bounded conditions of the MDP modeling.", "experimental_designs_or_analyses": "The majority of experimental design in shown in Appendix section C. The main paper does not include how the state evolution actually progresses in the experiments, but this information is present in the appendix. Figure 2 and Table 1 demonstrate the effectiveness of the Lapse. In my experience with Mujoco, a small change in state variable can have large impact on the resultant policy; the changes evolutions that the authors design are reasonable.", "relation_to_broader_scientific_literature": "This paper develops a framework for a real-world problem in life long learning. Learning robust policies for lifelong deployment and fine-tuning has been of interest in recent years and this paper fits well into that space. Due to the many components of Lapse, it might make the framework brittle to changes in the real world, and so this work may require testing on real-world data to assess its efficacy.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "In general, this paper is well-motivated and describes a general framework for a real-world problem. \n\n### Weaknesses\n- Assumes that changes due to state-evolution can be mapped iteratively to the original state distribution. I believe this is a restrictive assumption. The authors make claims that they can deal with the deterioration of the iterated mapping by adaptively favoring more recent data. It would be ideal if the framework could continuously learn a robust policy or be able to map directly from evolution n to evolution 0.\n- Requires paired data between evolutions. It is not clear to me when it is reasonable to make this assumption, if ever. Would be more general if the framework could deal automatically pair data based on some estimation of which distribution is came from.\n- Requires knowledge of when the state is evolving. It seems more likely that the evolution is a slow, and continuous process rather than a fast, discrete one. I recognize that maintenance is discrete and fast, but sensor deterioration is not.  \n- The distinction between state features and state space is not clear.", "other_comments_or_suggestions": "Some nitpicks:\n\n\n- the paper uses the phrase \"on the other hand\" 6 times in the paper. \n- The writing in general is somewhat awkward and at times hinders a smooth reading process. \n  - e.g. \"The policy trained by reinforcement learning makes decisions based on state features collect from sensors\" \n  - \n- Section 3.3, betwen lines 203 and 204, the word should be \"specifically\", not \"In specific\"", "questions_for_authors": "In addition to the concerns I voiced in the above sections, I have several explicit questions.\n\n1. Why opt for offline RL instead of off-policy RL for the behavior knowledge compenent?\n2. In general, where is the new data coming from at each stage of the evolution? Why not use an online algorithm to more quickly adapt? It seems that the method would incur significant cost when collecting new data in the evolved state space?\n3. Can you distinguish between evolving state-space, state features, and observation space?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces State Evolvable Reinforcement Learning (SERL), a novel and practical problem formulation addressing the challenge of deploying RL agents in environments where state representations change over time. The proposed Lapse method cleverly combines GANs for state reconstruction, robust RL policies, and offline RL. The authors provide strong theoretical justification for their approach, including performance bounds, and demonstrate substantial empirical improvements. Reviewers highlight the well-motivated methods, sound experimental design, and the paper's relevance to policy reuse and adaptation literature.
Addressing reviewers' concerns, the authors have now included additional experiments in Highway-env, complete training curves, ablation studies, and compute analysis, as well as clarifying the assumptions regarding access to maintenance information. Given these significant improvements and the paper's strong contributions, I recommend acceptance and encourage the authors to fully integrate all proposed changes and address all remaining reviewer comments in the final version.
Decision: Accept (poster)