Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning
Authors: Zhenghai Xue
Lang Feng
Jiacheng Xu
Kang Kang
Xiang Wen
Bo An
Shuicheng YAN
Venue: ICML 2025 spotlightposter (2025)
Keywords: Reinforcement Learning
Policy Transfer
Dynamics Shift
Policy Regularization
Primary area: reinforcement_learning

Abstract: To learn from data collected in diverse dynamics, Imitation from Observation (IfO) methods leverage expert state trajectories based on the
premise that recovering expert state distributions in other dynamics facilitates policy learning in the current one. However, Imitation Learning inherently imposes a performance upper bound of learned policies. Additionally, as the environment dynamics change, certain expert states may become inaccessible, rendering their distributions less valuable for imitation. To address this, we propose a novel framework that integrates reward maximization with IfO, employing F-distance regularized policy optimization. This framework enforces constraints on globally accessible states—those with nonzero visitation frequency across all considered dynamics—mitigating the challenge posed by inaccessible states. By instantiating F-distance in different ways, we derive two theoretical analysis and develop a practical algorithm called Accessible State Oriented Policy Regularization (ASOR). ASOR serves as a general-purpose module that can be incorporated into various RL approaches, including offline RL and off-policy RL. Extensive experiments across multiple benchmarks demonstrate ASOR’s effectiveness in enhancing state-of-the-art cross-domain policy transfer algorithms, significantly improving their performance.

Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning
Zhenghai Xue
Lang Feng' Jiacheng Xu 12 Kang Kang²
Xiang Wen 2 Bo An 12 Shuicheng YAN
Abstract
2019; Fujimoto et al., 2019) continuously updated during training (Lillicrap et al., 2016; Haarmoja et al., 2018). When applied to non-stationary environments with evolving dy- namics, obtaining effective policies necessitates extensive training datasets with sufficient dynamics coverage (Liu et al., 2022; Li et al., 2023). This challenge arises because optimal policies and state-action distributions are inherently dynamics-specific and do not generalize well. As result, trajectory data collected under one dynamics cannot be di- rectly utilized for policy learning in another.
To learn from data collected in diverse dynam- ies, Imitation from Observation (IfO) methods leverage expert state trajectories based on the premise that recovering expert state distributions in other dynamies facilitates policy learning in the current one. However, Imitation Learning inherently imposes performance upper bound of learned policies. Additionally, as the envi- ronment dynamics change, certain expert states may become inaccessible, rendering their distri- butions less valuable for imitation. To address this, we propose novel framework that inte- grates reward maximization with IfO, employ- ing F-distance regularized policy optimization. This framework enforces constraints on glob- ally accessible states-those with nonzero vis- itation frequency across all considered dynam- ics-mitigating the challenge posed by inaccessi- ble states. By instantiating F-distance in different ways, we derive two theoretical analysis and de- velop practical algorithm called Accessible State Oriented Policy Regularization (ASOR). ASOR serves as a general add-on module that can be in- corporated into various RL approaches, including offline RL and off-policy RL. Extensive experi- ments across multiple benchmarks demonstrate ASOR's effectiveness in enhancing state-of-the- art cross- doosin policy transfer algorithms, sig- nificantly improving their performance.
To address inefficient data exploitation, IfO algorithms (Wu et al., 2019; Torabi et al., 201 Jiang et al., 2020) pro- pose synamics-agnosste approach that enables learning from data with varying dynamics. It imitates stationary state distributions of expert policies and relies on the idea that expert state distributions are similar across different dynamics (Gangwani & Peng, 2020; Desai et al., 2020; Ra- dosavovic et al., 2021). Building on this idea, Xue et al. (2023a) further shows that the learning policy can achieve strong cross-dynamies performance it recovers the expert state distribution in least one of the dynamics. Instead, highlight key limitation of this idea: the similarity of state distributions does not always hold, particularly in tasks where state accessibility varies with environmental dynamics. In such cases, certain expert states become in- accessible as the environment changes. For example, an autonomous vehicle may safely navigate intersections at high speed under low traffic densities, but will face a high risk of collisions in dense traffic. Consequently, states repre- senting "safe driving at high speed" become inaccessible in certain dynamics, leading to distinct stationary state distri- butions. In such scenarios, expert trajectories with dynamics shift can be misleading.
1. Introduction
Imitation Learning (IL) and Reinforcement Learning (RL) (Sutton & Barto, 1998) facilitates large-scale policy optimization using datasets that are either static (Wu et al.,
To deal with the issue of distinct state accessibility, ex- pert states that not visited in some dynamics should be excluded during training. We define globally accessible states which maintains the same accessibility across differ- ent dynamics. By restricting imitation to these states, the policy can still leverage expert state trajectories from vari- ous dynamics while avoiding misleading information from inaccessible states. Meanwhile, cannot be naturally inte- grated with datasets containing reward signals and requires
NNnyyang Technological University, Singapore *kkwork AI e National University of Singapore. Correspondence to: Bo An <bean @ edu...
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Policy Regularization on Globally Accessible States in Cross-Dynamies Reinforeement Learning
demonstrations be optimal. To overcome this limitation, propose policy regularization method that incorpo- rates imitation as constraint while optimizing the reward maximization objective in RL. Specifically, the constraint ensures that the F-distance (Arora et al., 2017) between the accessible state distributions of the current and expert policies remains upper-bounded. This approach offers two key advantages: By instantiating the F-distance with JS divergence and network distance, establish lower-bound performance guarantees for policy regularization under dy- namics shift; By instantiating the F-distance with a GAN- like distance measure, we transform policy regularization into a practical reward augmentation algorithm which can be a general add-on module to existing cross-dynamics RL algorithms (Chen et al., 2021; Luo et al., 2022).
under dynamics T if if T(s'sss > 0. y (0,1) is the discount factor and po(s) is the initial state distribution.
Policy optimization under dynamics shift aims at find- ing the optimal policy that maximizes the expected re- turn under all possible 0 E O: x² = arg max, TT())) = EgE,,, 12 ""r(s, ae, St+1)], where the expectation is under su (- at). The Q-value (s, denotes the ex- pected return after taking action at state ()(s,a)= E o 7r((s,, at 8141 |so = 8, ao = =a}. The value function is defined as VF(s) = E (x) 27 (s.a. The optimal policy x² under T is defined as m. arg max, Esrup (s). We also intensely use the station- ary state distribution (also referred to as the state occupation function) (s) = (1->) E 7'p (s; = s π. T). The stationary state distribution under the optimal policy is de- noted as df(s), which is the shorthand for di (s) will be briefly termed as optimal state distribution in the rest of this paper.
Pa. at ~ [(-]s)). and 81+1
In empirical evaluations, we access the proposed algorithm across various environments, including Minigrid (Chevalier- Boisvert et al., 2023), the simulated robotics environment MuJoCo (Todorov et al., 2012), the simulated autonomous- driving environment MetaDrive (Li et al., 2023), and a large- scale fall guys-like game environment. The proposed algo- rithm exhibits superior performance when integrated with multiple sate-of-the-an art algorithms. Our contributions can be summarized as follows: 1) We identify common lim- itation of existing IfO methods under dynamics shift and propose state distribution imitation restricted to globally accessible states; 2) We design an F-distance regularized policy optimization framework that combines expert imi- tation with reward maximization; 3) By instantiating the F-distance in different ways, we conduct theoretical analy- ses and introduce practical algorithm, both validating the effectiveness of policy regularization on globally accessible states.
2.2. Related Work
Cross-domain Policy Transfer Cross-domain policy transfer (Niu et al., 2024) focuses on training policies in source domains and testing them in the target domain. In this paper, we focus on related problem of efficient train- ing in multiple source domains. The resulting algorithm can be combined with any of the following coss-domain policy transfer algorithms to improve the test-time performance. In online RL, VariBAD (Zintgraf et al., 2020) trains a context encoder with variational inference and trajectory likelihood maximization. CaDM (Lee et al., 2020) and ESCP (Luo et al., 2022) construct auxiliary tasks including next state prediction and contrastive learning to train the encoders. Instead of relying on context encoders, DARC (Eysenbach et al., 2021) makes domain adaptation by assigning higher rewards on samples that are more likely to happen in the target environment. Encoder-based (Chen et al., 2021) and reward- (Liu et al., 2022) policy transfer algorithms are also effective in offline policy adaptation and have been extended to offline-to-online tasks (Niu et al., 2022; 2023). VGDF (Xu et al., 2023) use ensembled value estimations to perform prioritized Q-value updates, which can be ap- plied in both online and offline settings. SRPO (Xue et al., 2023a) focus on similar setting of efficient data usage with this paper, but is based on a strong assumption of universal identical state accessibility. We demonstrate that such an assumption will not hold in many tasks and more deli- cate characterization of state accessibility will lead to better theoretical and empirical results.
2. Backgroud
2.1. Preliminaries
To model set of decision- decision-making tasks with different environment dynamics, we consider the Hidden Parame- ter Markov Decision Process (HiP-MDP) (Doshi-Velez & Konidaris, 2016) defined by tuple (S.A, θ, T, r, Y. Po), where S is the state space and A is the bounded action space with actions a E []. θ is the space of hidden param- eters. (a(s'Is (// a) is the transition function conditioned on (s, a). as well as a hidden parameter 0 sampled from e. r(s,a, a, s') is the environment reward function. By taking all s, a, s' into account, the reward function inherently includes the transition information and does not change in different dynamics. We also assume r(s, s') w.r.t. the action a is A-Lisschit.. Discussions on these Lipsehitz properties can be found in Appendix A.3. s' termed as accessible from
'T without subscript 0 refers to the transition function under any of the hidden parameter O.
2

---

Policy Regularization on Globally Accessible States in Cross-Dynamies Reinforcement Learning
Imitation Learning from Observations Imitation Learn- ing from Observation (IfO) approaches obviate the need of imitating expert actions and is suitable for tasks where action demonstrations may be unavailable. BCO (Wu et al., 2019) and GAIfO (Torabi et al., 201 are two natural modifications of traditional Imitation Learning (IL) meth- ods (Ho & Ermon, 2016) with the idea of IfO. IfO has also been found promising when the demonstrations are collected from several environments with different dynamics. Usually an inverse dynamics model is first trained with samples from the target environment by supervised learning (Wu et al., 2019; Radosavovic et al., 2021), variational inference (Liu et al., 2020), or distribution matching (Desai et al., 2020). It then used to recover the adapted actions in samples from the source environment. The recovered samples can be used to update policies with action discrepancy loss (Gangwani & Peng, 2020; Radosavovic et al., 2021; Liu et al., 2020). To our best knowledge, only HIDII (Jiang et al., 2020) consid- ered state distribution mismatch across different dynamics, where policies are allowed to take extra steps to reach the next state specified in the expert demonstration. We con- sider in this paper a more general setting where states in expert demonstrations may even be inaccessible.
Figure 1. Lava world example with dynamics shift.
the movable lava block is at Row 2 and Row 4. Taking state (1,3,0) as an example, the same action of "moving down" gives rise to different next state distributions due to distinct lava positions, leading to environment dynamics shift.
The state trajectories of the optimal policies on two example lava environments are plotted with black lines. The optimal state distributions are different under distinct environment dynamics. For example, state (3,3,0) has non-zero probabil- ity under d' (x) in Fig. 1 (above), but cannot be visited by the optimal policy in Fig. I (bottom). Existing state-only policy transfer algorithms will therefore not be suitable for such seemingly simple task, as demonstrated by the empirical results in Appendix C.2. The main cause of this distribution difference is the break of accessibility. State (4.2) is accessi- ble from (3,2) in Fig. (lower), but is inaccessible in Fig. I (upper). The inaccessible states will certainly have zero visitation probability and make the optimal state distribution different. Such break of accessibility can also happen in various real-world tasks, as discussed in Appendix B.
3. A Accssible States Oriented Policy Learning
In this section, we first provide a motivating example in See. 3.1, where expert state trajectories can mislead the learning policy in cross-dynamics training due to different state uccessibility. Based on the globally accessible states defined in Sec. 3.2, we then propose a general approach of F-distance regularized policy optimization. In Sec. 3.3 and Sec. 3.4, we propose two instantiations of the F-distance and respectively provide infinite-sample and finite-sample performance analyses for the policy regularization method. In Sec. 3.5, we propose a GAN-like instantiation of the F-distance, giving rise to the practical algorithm.
3.2. F-distance Regularized Policy Optimization with Accessible State Distribution
3.1. Motivating Example
Previous IfO approaches imitate expert policies through state distribution matching (Torabi et al., 2018;; Desai et al., 2020; Jiang et al., 2020) for cross-dynamics policy training. Fig. 1 demonstrates lava world task with dynamics shift where expert state distributions are no longer reliable. Con- sidering a three dimensional state space including agent row, agent column, and a 0-1 variable indicating whether there is an accessible lava block within one step reach of the agent. The agent starts from the blue grid and targets at the green grid with positive reward. It also receives a small negative reward on each step. The red grid stands for the dangerous lava area which ends the trajectory on agent entering. One while the other may appear at Row 2, 3, 4, and 5. Fig. demonstrates two examples where
Motivated by examples in Sec. 3.1, we propose to ignore inaccessible states and focus states that are accessible under all possible dynamics. We term the latter as globally accessible states with the following formal definition.
Definition 3.1 (Globally Accessible States). In an HiP- MDP (S, A, θ T, r, 7, Po), a state a S is called an glob- ally accessible state if for all E + there exist a policy T, such that d7, (s) > 0. S+ C S is denoted as the set of globally accessible states. The accessible state distribution of policy x under dynamics T is defined as dr:+(ss) (s)
(1 7) Ego r'p(s: S, E S T) /Z(T). where ((T) = Di E ESt S is the normalizing term.
lava block is fixed at Row 1,
In the lava world example in Fig. 1, the intersection of
3

---

Policy Regularization Globally Accessible States in Cross-Dynamics Reinforeement Learning
optimal state trajectories and globally accessible states is marked with yellow stars. These states more likely to be visited by the expert policies across both dynamics, effectively filtering out misleading states that are optimal in one dynamics and suboptimal in others. Consequently. expert state distributions can be safely imitated on globally accessible states under dynamics shifts.
N & M, 50 - 8, ON =8'. I. T2(8n 1n an-1) > 0.
and
N-1 Σ 1=1
7
r(8ns r On, sn+1) + (1 - - (50)
& Rs,
M2 is referred to M-R, accessible from MM.
In this definition, M is the number of extra steps required in M2 to reach the state s' from 8, compared with in Mi. Rg constrains the reward discrepancy in these extra steps. One special case is when M2 and MM are 1-0 accessible from each other, all states between (s. s') will have the same state accessibility. It is identical to the property of "homomorphous MDPs" (Xue al., 2023a), based on which theorem about identical optimal state distribution can be proved. Most of the previous approaches in IfO is built upon such assumption of 1-0-accessible MDPs, which is an over- simplification in many tasks. For example, the Minigrid environment in Sec. 3.1 contains 3-0.03 accessible MDPs. Instead, our derivations are based on the milder assumption M-R,, accessible MDPs that fits for broader range of practical scenarios.
Meanwhile, IL relies on access to expert trajectories, limit- ing its applicability in broader scenarios. In more general RL setting, where the dataset consists of trajectories with mixed policy performance and reward labels, policy opti- mization to maximize environment reward is often more preferable. To simultaneously leverage the expert state dis- tribution, we propose to regularize the training policy to generate stationary state distribution that closely aligns with the expert accessible state distribution (s). The regularization is exerted with an upper-bound constraint on the F-distance.
Definition 3.2 (F-distance, Definition 2 in (Arora et al., 2017)). Let F be a class of functions from Rdd to [0, 1] such that if f E F,1- - f E F. Let Φ be a concave measuring function. Then the F-divergence with respect to ø between two distributions μ and v is defined as
We first consider the case where policy regularization is performed with infinite samples, so that the constraint on F-distance is followed during both training and validation. We utilize IS divergence with the following instantiation of the F-distance.
d = sup E [+([[)))]+ E war
ssu
where ((0) is irrelevant to F, p4, and
Proposition 3.4. When Φ(t) log(t) and F == fall func- tions Rd to [0, I]}, dy,s is the JS divergence.
The resulting regularized policy optimization problem is formulated as follows:
We show that the learning policy & will have a performance lower-bound given a bounded JS-divergence with the opti- mal accessible state distribution.
max E Σ Y'r (st, de, 8141)
t=0
Theorem 3.5. Consider the HiP-MDP (S,A,O,T, Pa) with its MDPs M-R, accessible from each other. For all policy â, if there exists one certain dynamics To such that max Das(d ('ll, (-) BEG
s.t. max dy (d7 di ()) <E,
where To is an arbitrary environment dynamics. In the fol- lowing, we will demonstrate the effectiveness of this policy regularization framework both theoretically and empirically.
(()) & E, we have
((+) > max (TT,) 2R. + +6Λ + 2Rmax
(2)
1-7
3.3. Infinite Sample Analysis with JS Divergence Instantiation
Previous approaches (Xu et al., 2023; Janner et al., 2019; Xue et al., 2023b) also provide policy performance lower- bounds, but these bounds have quadratic dependencies on the effective planning horizon I- By accessible state- based policy regularization, we obtain a tighter discrepancy bound with linear dependency on the effective horizon.
In Sec. 3.3 and See. 3.4, we conduct theoretical analysis on how the policy regularization method in Eq. (1) influences policy performance under dynamies shift. We start with the definition of M-R, accessible MDPs, which formally characterizes the required accessibility property of MDPs so that expert state distributions can be imitated in a dynamics- agnostic approach.
3.4. Finite Sample Analysis with Network Distance Instantiation
Definition 3.3. Consider MDPs M1 = (,,,,,T, If for all transi-
and M₂ = AS T2, 7.P)). tions (8, 8') with (s' a. s) > 0, there exists states 8o, '1,'1 8N and actions ao 01, "0."",,NN- .N-1 such that
In this subsection, we derive a performance lower-bound of A if it is regularized with finite samples from the optimal ac- cessible state distribution. Due to the limited generalization
4

---

Policy Regularization on Globally Accessible States in Cross-Dynamies Reinforcement Learning
ability of JS distance (Xu al., 2020), we characterize the regularization error in Eq. (1) with the network distance. Proposition 3.6 (Neural network distance (Arora et al., 2017)). When <(t) = t and F is the set of neural net- works, F-distance is the network distance: dz(u,,) - suprer (Fzxu(((s) - Eeww [Fs))}
Algorithm et al., 2022).
The workflow of ASOR on top of ESCP (Luo
1: Input: Training MDPs (Mo..... ,Mn-1);: Context encoder Φ; Policy network иτ; Value network V; Dis- criminator Network w; Rollout horizon H; State parti- tion ratio pl, P2; Regularization coefficient X; Replay Buffer R.
Given m samples and the network distance bounded by ex, we analyze the generalization ability of the policy regular- ization in Eq. (1) with the following theorem.
2: for step = 0, 0, 2.... do
3: Sample MDP M, from (Mo, MRn11
4: for t= 1., H do
Theorem 3.7. Consider the HiP-MDP (S, O,,, po) with its MDPs M-,, accessible from each other. Given { sampled from dip²" for policy ßे regularized by To with the constraint max - < Ex, we 6EG
5: Sample from Φb (= | st, 11-1; Ft-1) 1) and then
sample a from π (a St, st, as in ESCP.
6:
data ($1+1, Tt, d(+1, St, Qt, 24) from Mg Add the data to the replay buffer R.
Rollout and
get
transition
have?
Sample a batch Doaehh Add pL Drechhl states with top P1 portion of high values and (2 portion of high proxy visitation counts to Dp; Add other states to DQ-
2R. + 8A
m 1/vm EF
1/
EF
n(++) > max ((7,)
O 1 1-7
1 1
(3)
with probability at least 1 - 8, where dž and a are the empirical version of distributions dy and d7,"" on {s0))m
8:
Train w as a GAN discriminator, regarding Dp as the real dataset and DQ as the generated dataset. For one-step transition in Dhachh update ry with r + Alogw(ss) w(z)..
Thm. 3.7 shows that with finite samples, the policy regular- ization still leads to tight performance lower-bound with linear horizon dependency. The lower-bound is stronger than the sample complexity analysis of Behavior Cloning with quadratic horizon dependency (Xu et al., 2020) and has the same horizon dependency with GAIL (Ho & Ermon, 2016).
10:
Use the updated Drutch to update o, n, and V.
G ( (8t) can therefore be written as dF. (s (0.0o))), which is the distribution of the states generated by a certain n, given these states are optimal. With the Bayes' rule, we have
3.5. Practical Algorithm with GAN-like Objective Function
+ To d7
(s)00) To
(8)
(7())
In this subsection, we design practical algorithms to solve the policy regularization problem in Eq. (I). The first chal- lenge is to estimate the value of certain instantiation of the F-distance. With the following proposition, we manage to do so with GAN-like optimization process.
pl p(Oo:ools, |s, n,, To To)d + (s)
To
1
p(O T, To)
(s
p(O n, To) (s)
1
To
(s)
p(O T, To)
di(())
(5)
Proposition 3.8. When F is a set of neural networks and <(t) = log t, the F-distance d v) is equivalent to the discriminator's objective function in a GAN (Goodfellow et al., 2014), where y is the real data distribution D real and is the generated data distribution De-
di (s)
d To (s)
(4)
(3)
where the last equation is also obtained with the Bayes' rule. According to Xue et al. (2023a), state s will be more likely to be sampled from dz (-) than from (-) if it has a higher state value (()) than average. Meanwhile, d7, (s) will be higher if 8 falls in the set of globally accessible states S+ and is most likely to be visited by various policies under dy- namics shift. Therefore, any of the pseudo-count approaches of visitation frequency can be used to measure whether is more likely to be sampled from d=; (s). In simulated environments with small observation spaces (Sec. 4. and 4.2), disagreement in next state predictions of ensembled environment models has shown to be a good proxy of visita- tion frequency (Yu et al., 2020). In large-scale real-world
According to the proposition, dF (), (-) in Eq. (1) can be obtained by training GAN discriminator to classify two datasets, one sampled from d-)(-)) and the other sampled from di (.). While (-) can be related to data newly collected in the replay buffer (Liu et al., 2021; Sinha et al., 2022), sampling from d;: (-) is still challenging. We introduce the binary observation state O with O, = de- noting 8 is the optimal state at timestep & (Levine, 2018).
The O notation omits constants irrelevant to m, Er, and 7- The full version in Thm. A.4.
5

---

Policy Regularization on Globally Accessible States in Cross-Dynamies Reinforeement Learning
tasks (Sec. 4.3), next state predictions can be unreliable, so we adopt Random Network Distillation (RND) (Burda et al., 2019) and use the error of predicting random mapping as the proxy visitation measure.
2020), MuJoCo (Todorov et al., 2012), and a Fall Guys-like Battle Royal Game. Dynamics shift in these environments comes from changes in navigation maps, evolvements of environment parameters, and different layouts of obstacles. To train RI. policies in these environments, ASOR is im- plemented on top of algorithms including PPO (Schulman et al., 2017), MA (Chen et al., 2021), and ESCP (Luo et al., 2022), which are all state-of-the corresponding field. To answer question (3), we visualize how the learned discriminator and the pseudo state count behave in different environments. Moreover, ablation stud- ies are conducted to examine the role of the discriminator and the influence of hyperparameters. Detailed descriptions of baseline algorithms are in Appendix C.I.
With the trained discriminator (x) in the F-distance, Eq. (1) can be transformed into an unconstrained optimiza- tion problem with the following Lagrangian to maximize". To
approaches in the
As
y (8t, Ce, 11+1) + A log (8+))
1
(5)
where A > 0 is the Lagrangian Multiplier. The only dif- ference between Eq. (5) and the standard RL objective is that logw"(ss) (st) is augmented to the environment reward r((e, Gz: 8t41). Therefore, the proposed approach can work as an add-on module to a wide range of RL. algorithms with reward augmentation.
4.1. Results in Offline RL Benchmarks
For offline RL benchmarks, we collect the static dataset from environments with three different environment dynam- ies in the format of D4RL (Fu al., 2020). Specifically, data from the original MuJoCo environments, environments with 3 times larger body mass, and environments with 10 times higher medium density are included. For baseline algorithms, we inlude IfO algorithms BCO (Torabi et al., 201 8a) and SOIL (Radosavovic et al., 2021), standard offline RL algorithms CQL (Kumar et al., 2020) and MOPO (Yu et al., 2020), offiine cross-domain policy transfer algorithms MAPLE (Chen et al., 2021), MAPLE+DARA (Liu et al., 2022), and MAPLE+SRPO (Xue al., 2023a).
Summarizing previous derivations, we obtain a practical re- ward augmentation algorithm termed as ASOR (Accessible State Oriented Regularization) for policy optimization un- der dynamics shift. We select the ESCP (Luo et al., 2022) algorithm, which is one of the SOTA algorithms in online cross- dooss-dynamics policy training, as the example base algo- rithm. The detailed procedure of ESCP+ASOR shown in Alg. 1. After the environment rollout and obtaining the replay buffer (line 6), we sample batch of data from the buffer, obtain a portion of P1P2 states with higher values and proxy visitation counts, and add them to the dataset Dp. Other states are added to Do. Then a discriminator network w is trained (line 8) to estimate the logarithm of the optimal discriminator output llgg (s), which is added to the reward r. (line 9). The effects of hyperparameters Pl. P2, and À are investigated in Sec. 4.1. The procedure of the offline algorithm MAPLE (Lee et al., 2020)+ASOR is similar to ESCR+ASOR, where the datasets are built with data from the offline dataset rather than the replay buffer.
The comparative results are exhibited in Tab. 1. IfO ap- proaches have the worst performance because they ignore the reward information (BCO) or cannot safely exploit the offline dataset (SOIL). Without the ability of cross- domain policy learning. CQL and MOPO cannot learn from data with dynamies shift and show inferior perfor- mances. Cross-domain policy transfer algorithms MAPLE, MAPLE+DARA, and MAPLE+SRPO show reasonable per- formance enhancement, while our MAPLE+ASOR algo- rithm leads to the highest performance. This highlights the effecttivneess of policy regularization on accessible states. We discuss the conceptual advantages of ASOR compared with DARA and SRPO in Appendix A.4.
4. Experiments
In this section, we conduct experiments to investigate the fol- lowing questions: (1) Can ASOR efficiently learn from data with dynamics shift and outperform current tftte-tf algorithms? (2) Is ASOR general enough when applied to different styles of training environments, various sources of environment dynamics shift, and when combined with distinet algorithm setup? (3) How does each component of ASOR (c.g., the reward augmentation and the pseudo- count of state visitations) and its hyperparameters perform in practice? To answer questions (1)(2), we construct cross- dynamics training environments based on tasks including Minigrid (Chevalier-Boisvertt et al., 2023)*, D4RL (Fu et al.,
The results of ablation studies are shown in Tab. 2. They show that larger value of reward augmentation coefficient X can give rise to performance increase. Meanwhile, using fixed values of X, P1 and P2 will not lead to a large drop of performance scores, so ASOR is robust to hyperparam- eter changes. ASOR's will have degraded performance if training datasets Dp and Da are improperly constructed, e.g., with random data partition, or without considering state values and visitation counts, i.e., with fixed øi = 0 or fixed P2 = 0.
'The derivations are in Appendix A.I. Wee leave the results in Appendix C.2.
6

---

Policy Regularization Globally Accessible States in Cross- -ross-Dynammcs Reinforeement Learning
Table 1. Results of offline experiments on MuloCo tasks. Numbers before 1 are scores normalized according to D4RL (Fu et al.. 2020)
and averaged across trials with four different seeds. Numbers after + normalized standard deviations. ME, M. MR and R correspond to the medium-expert, expert, mepium-replay and random dataset, respectively.
MAPLE +DARA
MAPLE +SRPO
MAPLE +ASOR
BCO
SOIL
CQL
MOPO
MAPLE
Walker2d-ME
0.25+0.04
0.14+0.08
0.63+0.13
0.06+0.05
0.1440.08
0.31
0.2240.07
0.2940.12
Walker2d-M
0.17+0.07
0.1640.01
0.75+0.02
0.15+0.22
0.41
0.4620.10
0.3240.17
0.4920.04
Walker2d-MR
0.01
0.04+0.01
0.06#0.00
-0.00+0.00
0.13+0.01
0.1240.00
0.13+0.01
0.1420.01
Walker2d-R
0.00#0.00
0.00+0.00
0. 0.00=0.00
-0.00#0.00
0.22+0.00
0.16#0.01
0.22+0.00
O 0.22
Hopper-ME:
0.08+0.02
0.01
0.20+0.07
0.01
0.45+0.07
0.49
0.43+0.06
0.51
Hopper-M
0.00#0.00
0.08+0.00
0.29#0.06
0.01=0.00
0.38+0.09
0.26+0.02
O. 0.48+0.04
0.71#0.14
Hopper-MR
0.00#0.00
0.00+0.00
0. 0.08
0.01+0.01
0.55+0.17
0.75+0.10
0.73+0.16 16
0.76
Hopper-R
0.00+0.00
0.00+0.00
0.10#0.00
0.01
0. 0.12+0.00
0.1240.00
0.25+0.08
0.32+0.00
HalfCheetah-Me ME
0.43=0.00
0.0000.00
0.03=0.04
-0.03+0.00
0.53+0.07
0.39:0.00
0.58+0.04
0.61#0.02
HalfChectah-N M
0. 0..422..2
0.39+0.00
0,42+0.01
0.36++0227
0.61#0.01
0.66+0.03
0.62+0.00
0.62#0.01
HalfChcetah-
0.16+0.01
0.25+0.00
0.46+0.00
-0.03+0.00
0.52+0.01
0.53+0.02
0.54+0.00
0.56:0.01
HalfCheetah-R
0.1420.01
0.3540.01
-0.01
-0.03+0.00
0.20+0.02
0.1920.01
0.22+0.00
0.21
Average
0.11
0.11
0.25
0.04
0.36
0.37
0.40
0.45
Table 2. Results of ablation studies in Offline MuJoCo tasks. The scores are averaged on cach environment with different expert levels.
Fixed
Fixed
Random partition
Fixed
Fixed
Fixed
Fixed
MAPLE +ASOR
0.1
0.3
Pi =
P2
P1-P2
0.5
P1-P2
0.3
Walker2d
0.22
0,25
0.26
0.22
0.29
0.30
0.26
0.28
Hopper
0.31
0.54
0.30
0.47
0.38
0.46
0.54
0.58
HalfCheetah
0.48
0.49
0.47
0.49
0.50
0.49
0.50
0.50
Average
0.34
0.43
0.34
0.40
0.39
0.42
0.43
0.45
4.2. Results in Online Continuous Control Tasks
the effect of the accessible state-based policy regularization. We also compare in Fig. 3 (left) the augmented reward with the enviromment reward on different states in Walker-2d. On states where the agent is about to fall over, the augmented reward drops significantly while the environment reward does not change much, demonstrating the effectiveness of the reward augmentation.
In online continuous control tasks, we explore other dimen- sions of dynamics shift, namely environment non-stationary and the continuous change of environment parameters. Such tasks are far more complicated than offline tasks with 3 different dynamics, but are within the capability of current approaches thanks to the existence online interactive train- ing environments. We consider the HalfCheetah, Walker2d, and Ant environments in the MuJoCo simulator (Todorov et al., 2012) and the autonomous driving environment in the MetaDrive simulator (Li et al., 2023). Sources of dynamics change include wind, joint damping, and traffic densities. For baselines we include the IfO algorithm GARAT (Desai et al., 2020), standard online RL algorithm SAC (Haarnoja et al., 2018), online cross-domain policy transfer algorithm OSI (Yu et al., 2017), ESCP (Luo et al., 2022), CaDM (Lee al., 2020), and SRPO (Xue et al., 2023a).
4.3. Results in A Large-Scale Fall Guys-Like Battle Royal Game Environment
In the large-scale fall guys-like game environment, we focus on highly dynamic and competitive race scenarios, char- acterized by myriad of ever-changing obstacles, shifting floor layouts, and functional items. The elements within the game exhibit both functional and attribute changes, result- ing in dynamics shift and evolving state accessibility. As shown in Fig. 4, the effects of trampolines (e.g., height and orientation) vary across different maps and the agent's inter- action with the trampoline will therefore result in different environment transitions depending on the specific configura- tion. The resulting dynamics shift has high stochasticity and cannot be effectively modelled by context encoder-based algorithms (Luo et al., 2022; Lee et al., 2020). We train the agent on 10 distinct maps, each presenting unique chal- lenges and configurations. The training step is set to 6M. Metries except policy entropy were averaged over the final
Comparative results in online continuous control tasks are shown in Fig. 2, where our ESCP+ASOR algorithm has the best performance all environments. Specifically, it only makes marginal improvements the HalfCheetah en- vironment, in contrast to large enhancement in other envi- ronments. This is because the agent will not "fall over" in the HalfCChetah environment, and the state accessibility will not change a lot under dynamics shift, undermining
7

---

Policy Regularization Globally Accessible States in Cross-Dynamics Reinforcement Learning
HaliChestah-NS
We Ii
Ant-NS
MetaDrive-NS
1000
6000
5000
5.00
8000
5000
4000
7000
4000
3000
0000
3000
2000
200
5000
2000
4000
1000
1000
3000
1000
600
800
1000
400
600
800
1000
4.0 Episodes
600
800
1000
Epicodas
SAC
GARAT
Ca
ESCP
ESCP4 SRPO
ESCP+ASOR
Figure 2. Results of online experiments on MuloCo and MetaDrive tasks. "NS" refers to tasks with non-stationary environment dynamics.
Extra Augmented
0.70
0,10
-0.08
Augmented Rewardd
0.60
0.00
0
0.50
00.0
0.45
0.00
Augmented Reward
0.29
3.11
.88.95
-336.37
0.35
Environment Reward
4.73
Timesteps (M)
Figure Left: Comparisons of the logarithm of the discriminator output, i.c., the augmented reward, and environment reward on different states in the Walker-2d environment. The augmented reward can better reflect the state optimality. Right: Curves for average extra loss and augmented reward in the fall-guys like game environment.
in maps with diverse altitudes, but unnecessary and hin- der the fast goal reaching in other maps. Fig. 3 (right) shows the curve of the augmented reward and the extra loss, includ- ing the discriminator training loss and the RND training loss. The loss curve drops smoothly and the average augmented reward remains stable, which means that the discriminator network is easy to train and has stable performance.
(a) Vertical and low
(b) Vertical and high
(b) Upper left and high
Figure 4. Demonstrations of dynamics shift caused different trampoline effect. Colors and textures are only for visual enhance- ment and not part of the agent's observations.
5. Conclusion
In this paper, we focus on efficient policy optimization with data collected under dynamics shift. We demonstrate that despite widely utilized IfO algorithms, the idea of similar expert state distribution across different dynamics can be unreliable when some states are no longer accessible the environment dynamics changes. We propose a policy regu- larization method that only imitates expert state distributions globally accessible states. By formally characterizing the difference of state accessibility under dynamics shift, we show that the accessible state-based regularization approach provides strong lower-bound performance guarantees for efficient policy optimization. We also propose practical algorithm called ASOR that can as add-on reward augmentation module to existing RL approaches. Extensive experiments across various online and offline RL bench- marks indicate ASOR can be effectively integrated with several state-of-the-arr cross-domain policy transfer algo- rithms, substantially enhancing their performance.
IM steps and the policy entropy is averaged in the initial IM steps. More experiment details listed in Appendix C.3, including additional map demonstrations, MDP setups, and the network structure.
As demonstrated in Tab. 3, PPO+ASOR achieves the highest scores in all five performance- related metrics. To be specific, the high total reward, unweighed goal reward, and success rate demonstrate the overall effectiveness of ASOR when ap- plied to complex large-scale tasks. Low trapped rate, small distance from cliff, and high policy entropy demonstrate the strong exploration ability of ASOR since it is better at getting rid of low-reward regions and has higher policy stochasticity. The low unnecessary jump rate demonstrates the effectiveness of policy regularization only on accessible states. Jumping states may appear in the optimal trajectories
8

---

Policy Regularization Globally Accessible States in Cross-Dynamics Reinforeement Learning
Table Experiment results in fall guys-like game environment. Metries with the up arrow (f) are expected to have larger values and vice versa. Metrics with (~) have no specific tendencies.
Total Reward (1)
Goal Reward
Success Rate
Trapped Rate (4)
Unnecessary Jump Rate (4)
Distance from Cliff (r)
Policy
(1)
(1)
Entropy (~)
PPO PPO+SRPO PPO+ASOR
0.329
0.361 1.154:1.0855
0.012.10.009
0.064
0.152.10.025
5.725:10.185
0.337=0.257
0.013
0.376=#0.006
0.038=00015
0.040=0.003
0,148=00018
5.859:10.098
0.554=0.336
1.781=0.053
0.387=#0.005
0.005=00.005
0.029=0.003
0.143=0.021
6.358++..122
Limitations This paper focuses the setting of HiP- MDP with evolving environment dynamics and static ward function. The resulting ASOR algorithm will not be applicable to tasks with multiple reward functions. Mcan- while, the theoretical results will be weaker on some adver- sarial HiP-MDPs with large Rg. Details will be discussed in Appendix A.3.
Chevalier-Boisvert, M., Dai, B., Towers, M., de Lazcano, R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306. 138006,1133,, 2023.
Desai, S., Durugkar, I., Karnan, H., Warnell, G., Hanna, J., and Stone, An imitation from observation approach to transfer learning with dynamics mismatch. NeurIPS, 2020.
Impact Statement
The ASOR approach enables better adaptation of RL mod- els in dynamic environments, making RL applications more resilient across diverse real-world conditions. The concept of avoiding misleading inaccessible states may improve the safety of autonomous agents, ensuring that they do not make catastrophic decisions when transitioning between environments. Therefore, ASOR's method can enhance cross-domain policy transfer, making RL algorithms more efficient for applications such as robotics, industrial automa- tion, healthcare, and finance.
Doshi-Velez, F. and Konidaris, G. D. Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. In IJCAI, 2016.
Eysenbach, B., Chaudhari, S., Asawa, S., Levine, S., and Salakhutdinov, R. Off-dynamicc reinforcement learning: Training for transfer with domain classifiers. ICLR, 2021.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Acknowledgement
This research is supported by the National Research Foun- dation, Singapore under its Industry Alignment Fund - Pre- positioning (IAF-PP) Funding Initiative. Any opinions, find- ings and conclusions or recommendations expressed this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. This research is also supported by the Joint NTU-WeBank Re- search Centre Fintech, Nanyang Technological Univer- sity, Singapore.
Fujimoto, S., Meger, D., and Precup, D. Off Off-policy deep reinforcement learning without exploration. In ICMI., 2019.
Gangwani, T. and Peng, J. State-only imitation with transi- tion dynamics mismatch. In ICLR, 2020.
Goodfellow, I. J., Pouget-Abadic, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial networks. CoRR, abs/1406.2666 2014.
References
Arora, S., Ge, R., Liang, Y., Ma. T., and Zhang, Y. Gener- alization and equilibrium in generative adversarial nets (GANs). In ICML, 2017.
Haamoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Burda, Y., Edwards, H., Storkey, A. J., and Klimov, O. Exploration by random network distillation. In ICLR, 2019.
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. NIPS, 2016.
Chen, X., Yu, Y., Li, Q., Luo, F., Qin, Z. T., Shang, W., and Ye, Offline model-based adaptable policy learning. In NeurlPS, 2021.
Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In NeurIPS, 2019.

---

Policy Regularization Globally Accessible States in Cross-yynamies Reinforcement Learning
Jiang, S., Pang, J., and Yu, Y. Offline imitation learning with misspecified simulator. In NeurIPS, 2020.
Niu, H., Hu, J., Zhou, G., and Zhan, X. A comprehen- sive survey of cross-domain policy transfer for embodied agents. CoRR, abs/2402.04580, 2024.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative Q-learning for offline reinforcement learning. In NeurIPS, 2020.
Radosavovic, I., Wang, X., Pinto, L., and Malik, J. State- only imitation learning for dexterous manipulation. In IROS, 2021.
Lee, K., Seo, Y., Lee, S., Lee, H., and Shin, J. Context- aware dynamics model for generalization in model-based reinforcement learning. /CML, 2020.
Schulman, J.. Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, 0tt//177.003344 2017.
Levine, S. Reinforcement learning and control as probabilis- tic inference: Tutorial and review. CoRR, 2018.
Sinha, S., Song. J., Garg, A., and Ermon, Experience replay with likelihood-frea importance weights. In L4DC, 2022.
Li, Q., Peng, Z.,, Feng, L., Zhang, Q., Xue, Z., and Zhou, B. Metadrive: Composing diverse driving scenarios for gen- eralizable reinforcement learning. IEEE Trans. Pattern Anal. Mach. Intell., 2023.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An
introduction. IEEE Trans. Neural Networks, 1998.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: physics
engine for model-based control. In IROS, 2012.
Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Gold- berg, K., Gonzalez, J., Jordan, M., and Stoica, RIlib: Abstractions for distributed reinforcement learning. In ICML, 2018.
Torabi, F,, Warnell, G., and Stone, Behavioral cloning from observation. In IJCAI, 2018a.
Torabi, F., Warnell, G., and Stone, Generative adversar- ial imitation from observation. CoRR, 0obs/1807.061555 2018b.
Lillicrap. T. P.. Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In ICLR, 2016.
Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. CoRR, abs/1911.11361, 2019.
Liu, F, Ling, Z., Mu, T., and Su, H. State alignment-based imitation learning. In ICLR, 2020.
Xu, K., Bai, C., Ma, X., Wang. D., Zhao, B., Wang. Z., Li, X., and Li, W. Cross-domain policy adaptation via value- value-guided data filtering. In NeurlPS, 2023.
Liu, J., Zhang, H., and Wang, D. DARA: Dynamics-awart reward augmentation in offline reinforcement learning. In ICLR, 2022.
Xu, T., Li, Z., and Yu, Y. Error bounds of imitating policies and environments. In NeurlPS, 2020.
Liu, X., Xue, Z., Pang. J., Jiang, S., Xu, F, and Yu, Y. Regret minimization experience replay in off- off-policy rein- forcement learning. In NeurlPS, 2021.
Xue, Z., Cai, Q., Liu, S., Zheng, D., Jiang, P., Gai, K., and An, B. State regularized policy optimization on data with dynamics shift. In NeurIPS, 2023a.
Luo, F. Jiang, S., Yu, Y., Zhang, Z., and Zhang, Y. Adapt to environment sudden changes by learning context sensitive policy. In AAAI, 2022.
Xue, Z., Peng, Z., Li, Q., Liu, Z., and Zhou, B. Guarded policy optimization with imperfect online demonstrations. ICLR, 2023b.
Nachum, O., Chow, Y., Dai, B., and Li, L. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In NeurIPS, 2019.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. MOPO: Model based offline policy optimization. In NeurIPS, 2020.
Niu, H., Sharma, S., Qiu, Y., Li, M.. Zhou, G., Hu, J., and Zhan, X. When to trust your simulator: Dynamics-aware hybrid offline-and-onling reinforcement learning. CoRR, abs/2206. 3464, 2022.
Yu, W., Tan, J., Liu, C. K., and Turk, G. Preparing for the unknown: Learning a universal policy with online system identification. In RSS, 2017.
Niu, H. Ji, T., Liu, B., Zhao, H.. Zhu, X., Zheng. J., Huang, P., Zhou, G., Hu, J., and Zhan, X. H20+: an improved framework for hybrid ics gaps. CoRR, abs/2309.12777 2023.
Zintgraf, L. M., Shiarlis, K., Igl. M., Schulze, S., Gal, Y., Hofmann, K., and Whiteson, S. VariBAD: very good method for adaative deep RL via meta-learning. In ICLR, 2020.
RL with dynam-
10

---

Policy Regularization Globally Accessible States in Cross-ynamices Reinforcement Learning
A. Additional Derivations and Proofs
A.1. Derivations of the Lagrangian
For expression convenience, we denote df (-) with u(()) and &. with v(-). We also omit the maximization over T in Eq. (1) as it can be obtained by following all policy constrains in different dynamies. We start from the optimization problem
max Be 11 1IN0
7'r (0t, (ot, 8641)
s.t. d (u()), <e.
(6)
The F-distance term can be transformed as:
d (.(,)())))) = Blogllooww ( (s)]= ) (lowo
/ (p(s) log (s)ds / v(()) w (s))ds
/ (1-1) 7)
Σ
7 p(ss = s| log
1=0
(7)
+ / (1 7)
2
n "((o = 3, ST To) log(1 (s))dss
(1 7)
yd [ BRSTT logw + E log(1 w (s:))]
where wt the trained discriminator. So the optimization problem can be written as the following standard form
min Egg Be 8141
-2'r (1,
t=
(8)
s.t. (11-7) Σ
7' [En,T, log (8t) + BEETTE+ log(1 - w*())))]
I- 7 < 0.
s
t=0
Neglecting items irrelevant n,, we get the Lagrangian as
Ae
L E 84
2' (r(se, 8441) + "loow"(ss))
(9)
1 1-7
A.2. Proofs of Theorems
Lemma A.1 (Value Discrepancy). Considering MDPs M1 = (S, (... and M₂ = (S, A. r, 7.P.))) Po) which are M-R, accessible from each other, for all we have
R₂ 2X
|V (s) - [1)(s)] <
(10)
1-7
where X is the action coefficient in the reward function. Detailed definition are in Sec. 2.1.
Proof. Without the loss generality, we consider the state s with ViA (s) > Vi, (s). Under the optimal policy (s). the next state of in Mi will be T's,, a²). As M₂ is M-R, accessible from Mi, there exists N & M such that in M2, s' can be reached from with action sequence a1, .., ,av. We then borrow the idea of iteratively computing V (s) Vi,(s) from Xue et al. (Xue et al., 2023a). According to the optimistic Bellman equation
Vi(() (8) = max r(s,a, 8') + VV
(11)

---

Policy Regularization Globally Accessible States in Cross-Dynamies Reinforcement Learning
have
VA (s) (s) (8) -V,,
V7, (s)
N-1
< <r(s,"", ,6) + +VVi (6')
Σ
(41)) (8')
(50 = 8, oy = s for brevity)
=r(s,a",s) - +-)
(5')
(12)
Σ r Ong Sn+1 7 (5) -V7) V7,
N-2
< - (00,"="((0,,,,, + 21
Σ
7 (r((8n Ons On, 8n41) 7VV (8') - [5')
N-2
<11 (1-=,_))((((s'0") + (s')) +V7
+ 22
Σ
n"n(sn On Sn+1) + + (6') - v7,(5'))
m=0
< Rg + 21 + 7| (s') V (s))]
lteratively computing | (8) - V7 (8)]. we have
R, 21
(6) <
(13)
1-7
Theorem A.2 (Thm. 3.5 in the main paper.). Consider the MDP M1 = (S,A,Ti.M,% (oP)) which is M-B Rs acces- sible from the MDP M2 = (S,A, A, T2, PsPr;), Po). For all policy ú, if there exists one certain dynamics To such that
max (dt ('llg, (-)) < e, we have
m(+) > maxn(T;) 2R, + + 2Rmax
(14)
1 1-7
Proof. (T),) VT, (TT,)) can be bounded with Thm. A.I:
Rs 22
11,,, |1,,(( (T7,) - 177, (TT,)) = |Boeno V7, (8) - ExEro 1// (8)
(15)
1-7
With a slight abuse of notation, we define the transition distribution di a, (s))aa ( a) and the accessible- state transition distribution d7; s') (s)(((a)))] a). Consider से such that d) = di (s) for all E S. The accumulated return of policy म under transition Ti can be written Wr, ( (1 7) - (r(s,,,s')). We also consider the accumulated return of the optimal policy under transition T2 including only accessible states: TH, (m)))) = (1 - - "),'',,,,''dd; s,a,s'dd;,
[r(s, a, s')], where TTa the optimal policy making transitions among accessible
states. Consider the Lipschitz property of the reward function:
|r(s, a1, s') - r(s, 8')l & Alaa a2ll
(16)
Taking expectation w.r.t. di, (.) on both sides, we get
Egndy |r(s, a1 s') - ((s s')l < Bondi A|[a 0zllı.
(17)
Letting p(Ar, Az|s) be any joint distribution with marginals and n conditioned on s E S+. Taking expectation w.r.t. μ
12

---

Policy Regularization Globally Accessible States in Cross-Dynamies Reinforcement Learning
on both sides, we get
Eag, r(s, a's rs, < Bondy Ba ser.s,s r(s, a2)
< ABgndj Eplla1 02ll1
(18)
max AE la1 02|11
2A 22
According to the definitions of rys (f) and ni, (m ), the L.H.S. of Eq. (18) is exactly the difference the two accumulated returns. Therefore, we get
21
Mr, (a) 12, TT +
)
(19)
1 1-
Then we will compute the discrepancy between mt. and 1T2 mr, can be computed with
Tra (TTz)) BoNpp Vr, (s)
N-1
(20)
E
7 r((( On snt1) + +. VTz
Tin (sx),
n=U
where eN is the accessible state accessible from so with Tテ2
According to the definition of M-R, accessible MDPs,
N
mB (xTT))
E,
2 In
7
VT (SN) Rsg + R.
N
N-2
< E,
Σ
r'
Σ
Tn - -(1 - 1)r(su, + (su)s SN)
n=0
n=0
(21)
+ N\ V ==x=((x)) (8N) - (7 V7)
112 (SN) + Rs
< Eso Tra (so) SN) + >Va23 न.. (sN) + *~ o (SN) - Via (SN)) + Rs +2 (Vis
= nソ. (+)) N (V The
(SN)
VT= क (sN)) + R. + 2A,
where is the short for r88n, ans + Iteratively scaling the value discrepancy between THi and TT= +.- we get
Rs + 2A
RS + 2A
(=1,)) Tig (
<
<
(22)
1- x
1-7
According to results in imitation learning (Lem. 6 in (Xu et al., 2020)), we have
2mmaa
Im (5) TT, (f) <
(23)
1-
Combining Eq. (15)(1))11)223,, we have |m (#) Wn (M
& Im (+) WT, (F)l + 1127 (7) n (mT5")
+ 117 (TT²2 +. ) 7Ta (TTT)) + + (mig)) m7,, (m/))
(24)
2R. + 6Λ + 2.Rmax
1-7
Taking expectation with respect to all T in the HiP-MDP concludes the proof.
Lemma A.3 (Lemma in Xu et. al (Xu et al., 2020)). Consider network class set P with A-bounded value functions,
i.e. [P(s) & A, for all s E S.P E P. Given an expert policy me and an imitated policy Ti with dp (d (
infzem dp dd) S sp, then vo E (0, 1), with probability at least 8, we have
dp (d d S inf dp (jmm d²))
2 (m) (P) + 27 (m) +
4F (P) + 12A
log(2/0) + Ep.
(25)
fell
m
13

---

Policy Regularization Globally Accessible States in Cross-Dynamies Reinforcement Learning
Proof. See Appendix B.3 of (Xu et al., 2020).
Theorem A.4. Consider the MDP M1 (1. which is M-R, accessible from the MDP M2
A,,,,, 7,Po) po) and the network P bounded by A, i.e. IP(s) & A. Given { sampled from diz"" if + P and the reward function T (8) = Ea (ss s') lies in the linear span of P. for policy it regularized by
TT
+ according to Eq. (1) with dp (d), d+:²) < Ep, we have
TE (f) > TE, (T7,)
2 + 82 2||r|1
(P) (P)
R
(m) d
R ((m) (P) + V
1og(2/0)
+ 2)
(26)
7 1 1-y
with probability at least
Proof. As M1 is M-R, accessible accessible from M2, there exists policy such that d7 (s) = di (8) for all 8 E E
With Thm. A.2, we have
Mm (T) > mT (Ti 2R 6
(27)
Then we compute the performance discrepancy nh (+) 7r, (f) given that dp (d d < Ep. The following derivations borrow the main idea from Xu et al. (Xu et al., 2020) and turn the state-action occupancy measure P into the state-only occupancy measure We start with the network distance of the ground truth state occupancy measures. According to Lem. A.3, we have
log(2/8) + Ep
dp dp(dT, of, (m)) or
(P) (P) 22(2)))) (m) (P) + 12A dg
(28)
m
with probability at least 1 8. Meanwhile,
Mm, (ft
- 72, (+)
1
E enddf
[ (8) - Egndy [(a,T]
1
1
1
(29)
E Bondj [ (s) - E Bandy [ra,Ts (8)]
+
E Bandy [Pa,2s (8)) - Bondd [s)
1
1-
1
+ 21
Bendy [A..T (5) - E Boddy [ra,T [+)]
As assume that the reward function ra,7: (s) lies in the linear span of P, there exists (P E PP such that r co + >x C.P. So we obtain that
N, (cr E R) and
21
mm (ft m2, (5)
E Bandi [r
-Bxndy (r (s)]
+
1-7
1
2x
Σ IE Eandd
[P](s, - Σ c. B.Badd (P((
+
1-7
21
[ci E BBndi [P( - E Egndf [P((s, + 1-7
(30)
1=1
21
Σ [cs dp (dty. of,)) + 1-7
1
21
||ll (di, dF.) + 1 I-
Combining Eq. (28) and Eq. (30), we have
m (+) > Tr,, (7)
2||/|1
R
(mm) (P) + R.mi) (P) 64,
(m)
log(2/0)
+
+
2x
(31)
1 1-7
m
2
1
14

---

Policy Regularization Globally Accessible States in Cross-Dynamies Reinforcement Learning
Table 4. Comparison between the
coefficient À coeffiiin
the maximum reward Amax in practical environments.
Environment
Action-related Reward
A
Rmax
CartPole-vO
1.00
InvertedPendulum-v2
0
1.00
Lava World
0
0
1.00
MetalDrive
0
0
Fall-guys Like Game
0
0
>>
Swimmer-v2
-0.0001 ||a|
0.0001
0.36
HalfCheetah-v2
|
0.1
4.80
Hopper-v2
-0.001
0.001
3.80
Walker2d-v2
-0.001 ||
0.001
Ant-v2
-0.5
0.5
6.00
Humanoid-v2
-
-0.1
0.1
> 8
with probability at least - 8. Combining Eq. (31) and Eq. (27), we have
MT (f) > mT (T), )
2 + 8A
2||||p
R
(m)
(P)
R di
(m) +6
(P) +
log(2/5)
(32)
with probability at least
6. Taking expectation with respect all T in the HiP-MDP concludes the proof.
A.3. Discussions on the Theorems
Lipschitz Assumption The Lipschitz assumption in Sec. 2.1 requires that s and s' keep unchanged, the deviation of the reward will not be larger than À times the deviation of the action
|r(s, a1, s') - r(s, az, s')l < Alaa1 - a2ll1.
(33)
Therefore, the Lipschitz coefficient À is only depends action-related terms in the reward function. In Tab. 4, we list the action-related terms of the reward functions for various RL evaluation environments, along with the corresponding values of A derived from these terms. As indicated in the table, the action-related terms in reward functions exhibit reasonably small coefficients all environments compared with the maximum environment reward Rmax- Therefore, the Lipschitz coefficient À will not dominate the error term in Thm. 3.5 and Thm. 3.7.
Failure Cases Apart from the Lipschitz assumptions that can easily be realized, Thm. 3.5 and Thm. 3.7 depend on the formulation of M-R, accessible MDPs. Potential failure cases will therefore include tasks with high Rg, that the performance lower bounds become weak. This will happen if states with lowest rewards exist in the optimal trajectory of some. and not all dynamics. In the example of lava world in Fig. 1, if a reward of -100 is assigned to grid (3,4), R, will be as large as 100, leading to a weak performance lower bound when the bottom lava block is at Row 2 with a best episode return of 1. Nevertheless, issues with theoretical analyses will not negatively influence the practical performance of ASOR.
A.4. Comparisons with Previous Approaches
Intuitively, the practical algorithm procedure of ASOR share some insights with some offline RL algorithms including AWAC (Nachum et al., 2019), CQL (Kumar et al., 2020) and MOPO (Yu et al., 2020). For example, ASOR prefers states with high values similar to AWAC and states with high visitation counts similar to CQL and MOPO. The advances of ASOR include: 1) By restricting the considered states accessible states, ASOR can be applied in offline datasets collected under dynamics shift, where the aforementioned offline RL algorithms can only learn from the dataset with static dynamics. Thm. 3.5 and Thm. 3.7 demonstrate the effectiveness of such procedure. 2) ASOR modifies the original policy optimization process by reward augmentation. This enables the easy combination of ASOR with other cross-domain algorithms to enhance their performance.
ASOR also share the approach classifier-based reward augmentation with DARA (Liu et al., 2022) and SRPO (Xue et al., 2023a). The classifier input in DARA (s, a, 8') from the source and target environments. Compared with ASOR, the
15

---

Policy Regularization Globally Accessible States in Cross-Dynamies Reinforcement Learning
traffic density:0.1
traffic density:0.2
traffic density:0.3
traffic deñgity:0.4
Figure MetaDrive environments with different traffic densities.
classifier in DARA exhibits higher complexity and is harder to train. It also requires the access to the information of target environments. Therefore, DARA has poor performance as demonstrated in Sec. 4.1 and cannot be applied to tasks with no prior knowledge on the target environments. The algorithm and theories of SRPO are based on the assumption of the same state accessibility, which is an over-simplifin werrsimpliificaton of some environments, as demonstrated in Sec. 3.1 and Sec. B. Comparative results in Sec. demonstrate the inferior performance SRPO compared with ASOR, in correspondence with the flaw in the assumption. Also, the theoretical analysis in the SRPO paper built on the assumption called "homomorphous MDPs" which is stronger than the M-R, accessible MDPs used in this paper and is special of the latter.
B. Examples of Distinct State Distributions
We claim in the main paper that previous assumption of similar state distributions under distribution shift will not hold in many scenarios. Apart from the motivating example of lava world in Sec. 3.1, we demonstrate more examples in the MetaDrive (Li et al., 2023) and the fall-guys like game environment. Examples of MetaDrive environments with different traffic densities shown in Fig. 5. The dynamics shift lies in that the ego vehicle will have different probabilities to detect other vehicles nearby. In environments with low traffic densities, there is enough space for some vehicles with optimal policies to drive in high speeds. But in environments packed with surrounding vehicles, fast driving will surely lead to collisions. So the vehicles can only drive in low speeds. As the vehicle speed is included in the agent's 's state space, difference in traffic densities will lead to distinct optimal state distributions.
Visualizations the fall- fall-guys like game environment used Sec. 4.3 are shown in Fig. 7, where map components including the conveyor belt speed, the balloon reaction, the floor reaction, and the hammer distance will work together, giving rise example, in the left environment the optimal
dynamics shift. Taking the variation of hammer distance (Fig. 7 (d)) trajectory will contain states where the hammer is near the agent. But in the right environment, there trajectories that keep the hammer far away to avoid being hit out of the playground. Blindly imitating optimal states collected in the left environment will lead to suboptimal performance in the right environment.
C. Experiment Details
C.1. Baseline Algorithms
In experiments with four different tasks, we compare ASOR with the following baseline algorithms:
PPO (Schulman al., 2017): The widely used, off-the-shelf online RL algorithm with on-policy policy update.
SAC (Haarnoja et al., 2018): The widely used off-policy RL algorithm with entropy maximization for better exploration.
BCO (Torabi et al., 2018a): Learn agent-specific inverse dynamics model infer the experts' missing action information.
GAIfO (Torabi et al., 2018b): A state-only version of the GAIL algorithm.
GARAT (Desai et al., 2020): Use the action transformer trained with GAlL-like imitation learning to recover the experts' next states the original environment.
16

---

Policy Regularization Globally Accessible States in Cross-Dynamis Reinforcement Learning
Training Performance
Reward with Different Lava Rows
Estimated Uncertainty on Different Rows
0.7
1.75
0.6
Average Reward
Average Reward
0.6
1.50
0.5
Uncertainty
0.4
0.4
0.3
0.75
0.2
0.50
0.1
0.25
0.0
500
1000
1500
2000
3
5
(a) Training Steps
(b) Lava Row
(c) Row
PPO
PPO+SRPO
SOIL
GAIO
PPO+ASOR (Ours)
Column
Column
Figure 6. Results in the Minigrid environment. (a) Performance comparison between PPO+ASOR and baseline algorithms. (b) The average reward the environment with different position of the second lava PPO and PPO+SRPO has very low rewards when the bottom lava is on row 5; (c) The state uncertainty estimated by ASOR on different rows of the lava environment.
SOIL (Gangwani & Peng. 2020): An algorithm combining state-only imitation learning with policy gradients. The overall gradient consists of policy gradient term and auxiliary imitation term.
CQL (Kumar et al., 2020): The widely used offline RL algorithm with conservative Q-learning.
MOPO (Yu al., 2020): A model-based offline RL algorithm subtracting disagreements in next-state prediction from environment rewards.
MAPLE (Chen et al., 2021): The offline RL algorithm based on MOPO with an additional context encoder module for cross-dynamics policy adaptation.
OSI (Yu et al., 2017); An algorithm using context encoders for online system identification.
CaDM (Lee et al., 2020): The online RL algorithm with context encoders for cross-dynamics policy adaptation.
DARA (Liu et al., 2022): Make reward augmentations with importance weights between source and target dynamics.
SRPO (Xue et al., 2023a): Make reward augmentations with the assumption of similar optimal state distributions under dynamics shift.
C.2. Results in Minigrid Environment
For experiments in the Minigrid environment Sneevller-Boiiveed et al., 2023), the row number of the bottom lava randomly sampled from (2, 3, 4, 5), leading to dynamies shift. By including lava indicator part of the state input, the policy is fully aware of environment dynamies changes and the need of context encoders (Luo et al., 2022; Lee et al., 2020) is excluded. The categorical action space includes moving towards four directions. The reward function for each environment step is -0.02 and reaching the green goal grid will lead to an additional reward of 1. The episode terminates when the red lava or the green goal grid is reached. For baseline algorithms we select online RL algorithms PPO (Schulman et al., 2017) and PPO+SRPO (Xue et al., 2023a), as well as IfO algorithms SOIL. (Gangwani Peng. 2020) and GAIfO (Torabi et al., 2018b).
We demonstrate the experiment results in Fig. 6 (a). Our ASOR algorithm can increase the performance of PPO by large margin, while SRPO can only make little improvement. This because the optimal state distribution different lava world environments will not be the same. SRPO will still blindly consider all relevant states for policy regularization, leading to suboptimal policies. Fig. 6 (b) demonstrates the average reward with each possible position of the bottom lava block. PPO and PPO+SRPO have low performance when the bottom lava block is at Row 5. They mistakenly regard grids at (5,4) and (5,5) as optimal, but ASOR will recognize these grids as -ccesscess states. We also demonstrate in Fig. (c) the disagreement in next state predictions used to compute pseudo count. States far from the starting point have higher prediction disagreements and lower pseudo counts.
17

---

Policy Regularization Globally Accessible States in Cross-ynamiies Reinforcement Learning
Up
Down
Low Speed
High Speed
(a) Conveyor Belt Speed
(b) Balloon Reaction
Near
Far
Up
Disappea
(c) Floor Reaction
(d) Hammer Distance
Figure 7. More demonstrations of dynamics shifts in fall guys-like game. Colors and textures are only for visual enhancement and are not part of the agent's observations.
C.3. Additional Setup of the Fall-guys Like Game Environment
Additional Environment Demonstrations Next, we present additional examples dynamic shifts within the fall-guys like game environment demonstrate the diverse and variable nature of in- in-game elements. As shown in Fig. 7, the game environment features range of dynamic shifts which contribute to the complexity and unpredictability of the gameplay, Specifically, observe the following scenarios: Fig. 7 (a): The speed of conveyor belts changes across different game settings, leading to varied transitions in the agent's position and momentum when it steps onto these belts. Fig. 7 (b); Balloons exhibit different reactions upon interaction with the agent. This variation can significantly affect the agent's s subsequent trajectory. Fig. (c): The behavior of floors under the agent's 's influence varies significantly. Some floors may collapse, disappear, or shift unexpectedly, introducing further complexity to the environment. Fig. (d): The distance and direction in which the agent is ejected when struck by hammers can vary widely. This variability depends on the unpredictable environmental dynamic shifts, for example, the force and angle of the hammer's swing.
MDP Setup
Below, we provide definitions of state space, action space, and rewards in the fall-guys like game environment.
State space S:
Terrain Map (dim=(16 x 16 x 2) with granularities of [1.0, 2.0)): The relative terrain waypoints the agent's surrounding area. Various granularities capture different details and perceptual ranges effectively.
Item Map (dim=(16 16 1) with granularities of [1.0, 2.0)): Map of nearby items objects. Multiple maps focus different item types, with granularities for varying spatial scales.
Target Map (dim=(16 x 16 x 1) with granularities of [1.0, 4.0, 16.0)): Map of archive points locations. Various granularities capture different details and perceptual ranges at different spatial scales.
Goal Map (dim=(16 16 x 1) with granularities of [1.0, 4.0, 16.0)): Map of intermediate goal locations. Various granularities capture different details and perceptual ranges at various spatial scales.
Agent Info (dim=32): Details about agent's own state, including position, rotation, velocity, animation state, and forward direction.
18

---

Policy Regularization Globally Accessible States in Cross-Dynamies Reinforcement Learning
Destination Info (dim=9): Details about the destination, including position, rotation, and size of the destination, providing crucial details for navigation and goal achievement.
Action space A::
MoveX (dim=3): Move along the X-axis. The three discrete options typically represent movement in the positive direction, negative X direction, or no movement.
MoveY (dim=3): Move along the Y-axis, with three discrete options for movement in the positive Y direction, negative Y direction, or no movement.
Jump (dim=2): Represents jump behavior. Options are to initiate jump or not.
Sprint (dim=2): Represents sprint behavior. Options are to start sprinting or not.
Attack (dim=2): Executes attack. The two discrete options initiate attack or not.
UseProp (dim=2): Utilizes prop. The two discrete options indicate whether the prop is used or not.
UsePropDir (dim=8); Determines the direction for prop usage. The eight discrete options offer various directional choices for prop utilization.
Idle (dim=2): Represents idle behavior. Options are to remain idle not.
Reward r:
Arrive Target (value=1.0): Rewards the agent for successfully reaching the archive point, with a positive reward of 1.0 upon achievement.
Arrive Goal (v)luc=0.3)): Rewards the agent for reaching intermediate goal locations within the environment, with positive reward of 0.3.
Arrive Destination (value=1.0): Rewards the agent for reaching the final destination endpoint within the environment, motivating task completion.
Goal Distance (decay rate=0.05): Offers distance- rewards, varying based on proximity to specific goal locations. Rewards diminish as the agent moves away from the goal, with distinct values for different distance ranges.
Fall or Wall (value=-1 1.0): Penalizes the agent for continuously hitting the wall or falling off cliff with a penalty of -1.0.
Stay (value=-0.00) Penalizes the agent for remaining stationary for extended periods, encouraging continuous exploration and movement.
Time 0.02): Penalizes each time step, encouraging efficient decision maaiinn and timely task completion.
Network Architecture The network architecture is structured as follows: The Terrain Map. Item Map, Target Map, and Goal Map are each fed into a convolutional neural network (CNN) with ReLU non-linearity, followed by a fully connected network (FCN). This process yields four separate 32-dimensional vector representations for each respective map. The Destination Info and Agent Info are independently input into attention layers, generating 32-dimensional vectors for each. Subsequently, all 32-dimensional vectors (from the CNNs and attention layers) concatenated into single feature vector. The concatenated feature vector undergoes processing by multi-head FCN to yield various output actions. Additionally, the concatenated feature vector is processed by another FCN to produce value as the value function estimator.
Training Setup We utilized the Ray RLlib framework (Liang et al., 2018), configuring 100 training workers and 20 evaluation workers. The batch size was set to 1024, with an initial learning rate of x 10-3, which linearly decayed to 3 x 10 10-4 over 250 steps. An entropy regularization coefficient of 0.003 was employed to ensure adequate exploration during training. The training was conducted using NVIDIA TESLA V100 GPUs and takes around 20 hours to train 6M steps.
19


Reviews:
[{"review_id": "LNEXIJBElE", "reviewer": "Reviewer_yDbB", "summary": "This paper studies policy learning in cases where the environment dynamics may vary during training. Even when expert demonstrations or replay buffers are provided, some states within them may be inaccessible. In such situations, the policy should identify which states are globally accessible and consider only the information from them.\n\nTo address this, the paper proposes ASOR, an add-on GAN-like module that identifies states with high values or proxy visitation counts. ASOR is integrated with existing RL methods and achieves superior or competitive performance compared to baselines on various benchmarks, including Minigrid, D4RL, MuJoCo, and Fall Guys-like battle royale games.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and insightful comments. Responses are provided as follows. The linked file is available [here](https://anonymous.4open.science/api/repo/ICML_ASOR_Rebuttal-01CB/file/rebuttal_append_file.pdf).\n\n**Q1: Practical estimation concerns**\n\nA: We discuss in Sec. 3.5 on how to obtain practical estimations. Instead of directly estimating the intractable accessible state distribution $d\\_{T_0}^{\\*,+}(s)$ , we estimate the its likelihood ratio with $d_T^\\pi(s)$. The key property used here is Prop. 3.8, which shows that the likelihood ratio can be obtained by optimizing a classifier similar to the GAN discriminator, as long as the real data are sampled from the state distribution in the molecular of the ratio and the fake data from the denominator. According to Eq. (4), the likelihood ratio $\\frac{d_{T_0}^{\\*,+}(s)}{d_T^\\pi(s)}$ can be decomposed into the production of state optimality and accessibility. We therefore use the state value function and state visitation pseudo-count approaches to split the training batch and create classifier training data. In this way, we manage to obtain an estimation of the likelihood ratio, which serves as the augmented reward in practice.\n\nFor the amount of transitions required, we refer the reviewer to Fig. (3)(right), where the red line (extra loss) contains the loss of training the estimation network. It takes roughly 1M steps to obtain a decent estimation, which is 1/6 of the total policy training steps. In other words, a relatively accurate likelihood ratio estimation is easier to obtain compared with a good RL policy, and can provide reasonable guidance in the policy training process.\n\nWhen there are safety issues, safe RL techniques can be integrated as ASOR does not change the training pipeline of the base algorithms. One may also increase the coefficient of the augmented reward because it encourages the policy to stick to states that are more likely to be visited by expert policies in different dynamics.\n\n**Q2: Dynamics identification**\n\nA: Different base algorithms have their own approach of identifying environment dynamics, and ASOR as a general reward augmentation algorithm is agnostic to these approaches. For example, MAPLE and ESCP use context encoders trained with auxiliary loss to detect dynamics change. In the fall-guys like game environment, we include transformers in the policy network to automatically determine environment dynamics from history state-actions. In the lava scenario, we add environment information in the state space for simplicity. Apart from location coordinates, the state space has a 0-1 variable indicating whether there is a lava block nearby (Lines 153-158). In Fig. 1 (up), the agent at (2,3) will have (2,3,1) as the state; In Fig. 1 (bottom), the state will be (2,3,0). The agent can therefore be informed which situation it is facing.\n\nWe will add the aforementioned discussion to the revision.\n\n**Q3: Confusing notations**\n\nA: Sorry for the confusing notations.  $d_T^*(s)$ is the brief version of $d_T^{\\pi^*}(s)$ and means the same distribution. For the accessible state distribution such as $d^{\\pi,+}_T(s)$, we plan to use a new distribution notation $\\delta^\\pi_T(s)$ for better readability.\n\n**Q4: Experimental designs**\n\nA: Thanks for recommending diffusion policies as baseline. We consider the Diffusion-QL algorithm and refer the reviewer to Tab. 3 of the linked pdf for comparative analysis. It exhibits better performance than CQL especially in the Hopper environment, but cannot outperform the average performance of MAPLE+ASOR. This can be because Diffusion-QL tries to recover the state-action joint distribution in the offline dataset. Such distribution will be unreliable under dynamics shift since given the same state, the optimal action may be different in different dynamics. For maximum entropy policies, we included SAC as baseline in online RL experiments. Meanwhile, ESCP and ESCP+ASOR are built upon SAC themselves and will benefit from the maximum entropy training objective.\n\nFor experiments with deviated dynamics during training and testing, i.e., testing with out-of-distribution dynamics, we refer the reviewer to Fig. 1 of the linked pdf. We utilize the OOD setting in ESCP, where test environment parameters, including the damping coefficient and the wind speed, are sampled from distributions that are 33% broader than the training distribution. In this scenario, HalfCheetah and Ant witness performance drops across different algorithms.  Algorithms show almost no performance changes in the Walker2d environment on unseen test environments. This may be because dynamics shift exerts less influence on the walker agent. In both scenarios, ESCP+ASOR still achieves the best results, demonstrating its ability to efficiently train a more robust policy.\n\n\n**Q5: Literature completeness**\n\nA: Thanks for bringing LMDP and diffusion RL research into our attention. We will add discussions on these related works.", "claims_and_evidence": "This paper highlights the potential impact of changes in the environment's dynamics and their consequences—specifically, the accessibility of states must be considered when learning from collected transitions. Otherwise, the policy may be misled, as it was trained under different dynamics and scenarios.\n\n[+] In my opinion, the claim appears sound, and the theoretical support—ranging from infinite samples and finite samples to a practical GAN-like implementation—seems accurate. Moreover, the paper conducts comprehensive experiments to empirically validate its claims.\n\n[-] However, one concern I have is that even if we determine that a given task and environment can be suitably defined in the form of HiP-MDP, how can we obtain the required information in practice? For example, how do we estimate the accessible state distribution? How many orders of magnitude of transitions are required to obtain a reliable estimate? What should be done if there is a safety issue?", "methods_and_evaluation_criteria": "The ASOR method has the following major components or strategies, including a GAN-like training method, tracking state values and proxy visitation counts, and using a Lagrangian multiplier to augment rewards with the discriminator's output.\n\n[+] These designs make sense to me. Intuitively, they would make the policy more cautious when visiting areas that are likely to change over time. These designs are also supported by the ablation studies (Table 2, Figure 3).\n\n[+] The results for each environment are tested over multiple rounds, and the standard deviation of the method's performance is reported as well.\n\n[-] One question I have: while ASOR may encourage the policy to focus on areas that are not affected by changes in dynamics and have high values, how can it determine which dynamics it is in during inference? For example, taking the lava scenario in Figure 1, the policy may know that there is uncertainty in the intersection areas of rows 2, 3, and 4 and columns 4, 5, and 6. But when it stands at (2, 3), how does it know which situation it is facing now?", "theoretical_claims": "[+] As mentioned above, the theoretical claims are sound, and their proof seems accurate. Those toy examples and illustrations play a critical role, helping the reader follow the derivation even though it is somewhat cumbersome.\n\n[-] Some notations use superscripts and subscripts to represent different meanings, which can easily be confused while reading, such as $d_{T}^{\\pi^{*}}(s)$ , $d_{T}^{\\*}(s)$, $d_{T_{\\theta}}^{\\pi}(s)$, and $d^{\\pi, +}_{T}(s)$. If a clearer and more concise way of expression exists, the theoretical paragraphs will be easier to read.", "experimental_designs_or_analyses": "[+] ASOR is compared with multiple baseline methods in environments across different fields. Although ASOR achieves sub-optimal results in some cases, I acknowledge that it performs the best overall.\n\n[+] This paper analyzes the experimental results in depth, rather than simply reporting the results of each method, which is commendable.\n\n[-] I believe diffusion and maximum entropy (max-ent) policies should be compared in the experiments, as these policies are effective at handling sudden changes in the environment. Comparing with them and still achieving the best performance would make ASOR's effectiveness and superiority even more convincing.\n\n[-] Another experiment worth exploring is the scenario when the dynamics during training and testing deviate. Since in Algorithm 1, the dynamics are randomly sampled, it may happen that a particular dynamic is rarely sampled, and thus, the policy may not learn well from transitions of that dynamic. One possibility is that a dynamic is excluded during training but may be sampled during inference. In this case, what would be the difference in performance between each method?", "relation_to_broader_scientific_literature": "[+] Although HiP-MDP is a relatively new and niche research direction, I find it more aligned with practical scenarios and believe it has the potential to enhance RL applications in real-world settings. I appreciate efforts to explore this direction and propose novel methods to push its boundaries.", "essential_references_not_discussed": "[+] Since there is little literature related to HiP-MDP, I believe this study sufficiently cites and discusses the literature in this field.\n\n[-] However, it would make the literature review more complete if HiP-MDP were discussed alongside other potentially related MDP settings, such as LMDP [R1]. Additionally, diffusion and max-ent policies are known to be good at dealing with environmental (dynamics) changes, and I believe they should be discussed and examined in this work.\n\n[R1] Zhou et al., \"Horizon-Free and Variance-Dependent Reinforcement Learning for Latent Markov Decision Processes,\" ICML 2023.", "other_strengths_and_weaknesses": "[+] The proof, descriptions of the baseline method, and details of the evaluation environments provided in the appendix allow readers to more fully examine how the study was conducted and provide sufficient reproducibility.\n\n[-] A minor typo is found at line 383: \"ASOR's will have ...\".", "other_comments_or_suggestions": "All my comments and suggestions are listed in the appropriate fields above. Since I have rarely engaged in HiP-MDP research before, I am giving a relatively conservative initial recommendation of 'weak acceptance.' However, I am happy to raise my recommendation if my concerns are addressed during the discussion phase and no significant or widely shared concerns are raised by other reviewers.", "questions_for_authors": "My concerns and questions have been summarized in the above fields, please consider addressing those points marked with [-].", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H09wiLP9yY", "reviewer": "Reviewer_ZkD1", "summary": "The paper pinpoints a flaw in existing IfO methods where state inaccessibility due to changing environment dynamics can disrupt the similarity of expert state distributions. To tackle this, it presents a policy regularization approach centered on globally accessible states. The proposed framework combines reward maximization and IfO via F-distance regularized policy optimization, leading to the development of the ASOR algorithm through different F-distance instantiations. Experiments across multiple benchmarks demonstrate its effectiveness in enhancing cross-domain policy transfer algorithms, outperforming baselines, and theoretical analysis offers performance guarantees for policy regularization during dynamics shift.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and insightful comments. Responses are provided as follows.\n\n**Q1: Determining accessibility in practice**\n\nA: Tasks with potentially empty globally accessible state set are indeed extreme cases where ASOR may degrade into the base RL algorithm. An example proposed by Reviewer UQdV is when we want to learn a walking policy for robots with either normal legs or crippled legs, and with different heights. In this case, as the robots may never perfectly achieve the same status, the globally accessible state will become an empty set. But two practical factors can help alleviate such concern.\n- One may regard the state dimension that remains distinct across dynamics as hidden environment parameter and do not include such information when identifying globally accessible states. For example, the robot height in the mentioned example can be excluded in the state space. States that are visited by expert policies with different robot height, e.g., smooth walking with moderate speed, can be regarded as globally accessible and preferred when training new policies.\n- In the practical algorithm procedure of ASOR, a relative accessibility detection mechanism is applied. States that are most likely to be accessible in a training batch are regarded as preferable (Line 7 in Algorithm 1). Even though there are no states that are perfectly accessible across dynamics, we can rely on the discriminator network to automatically find the most appropriate states to assign higher rewards (Line 8 in Algorithm 1).\n\n**Q2: Sampling from the accessible distribution**\n\nA: According to Sec. 3.5 of the paper, instead of directly sampling from the intractable accessible state distribution $d_{T_0}^{\\*,+}(s)$ , we estimate the its likelihood ratio with $d_T^\\pi(s)$. The key property used here is Prop. 3.8, which shows that the likelihood ratio can be obtained by optimizing a classifier similar to the GAN discriminator, as long as the real data are sampled from the state distribution in the molecular of the ratio and the fake data from the denominator. According to Eq. (4), the likelihood ratio $\\frac{d_{T_0}^{\\*,+}(s)}{d_T^\\pi(s)}$ can be decomposed into the production of state optimality and accessibility. We therefore use the state value function and state visitation pseudo-count approaches to split the training batch and create classifier training data. In this way, we manage to obtain an estimation of the likelihood ratio, which serves as the augmented reward in practice.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes.  For example, Theorem 3.5 shows the lower bound of the learning policy's performance when using JS divergence for F-distance regularization.", "experimental_designs_or_analyses": "The experimental designs are sound. In offline RL benchmarks, the authors collect static datasets from environments with different dynamics and compare ASOR with multiple baseline algorithms. In online continuous control tasks, they explore different sources of dynamics shift. In the fall guys-like game environment, they consider highly dynamic and competitive scenarios. The ablation studies in the MuJoCo tasks help to understand the role of different components and hyperparameters in ASOR.", "relation_to_broader_scientific_literature": "The paper builds on the existing literature of cross-domain policy transfer and Imitation Learning from Observation. It addresses the limitations of previous IfO methods that assume similar state distributions across different dynamics. \n\nThe paper defines globally accessible states, providing a new way of looking at state spaces in the context of learning from diverse dynamics.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths:\n- The paper is well-written and the concepts are clearly explained.\n- Policy regularization on globally accessible states is a novel way to handle dynamics shift in RL.\n\n### Weaknesses:\n- Although the definition of accessible states is clear in theory, it may be extremely difficult to accurately determine whether a state is globally accessible in a real-world environment. Real systems may have noise, unmodeled factors, or overly complex dynamic changes, making it hard to determine whether there exists a policy that can access the state with a non-zero probability for all values of the dynamic parameters. \n- Sampling from and estimating distributions from different distributions are crucial steps in the algorithm. However, in practice, accurately sampling from these distributions can be challenging, especially sampling from $d^{*, +}_{T_0}(\\cdot)$. If the sampling is inaccurate, it may lead to a deviation in the estimation of the state distribution, thereby affecting the training of the discriminator and the learning effect of the policy.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "JzNVgQ4piL", "reviewer": "Reviewer_2QKH", "summary": "The paper addresses the challenge of learning policies when the environment dynamics vary such that expert state trajectories may not always be accessible under dynamics shifts. To overcome this, the authors present the Globally Accessible States with a formal definition and the  F-Distance, a measure of the discrepancy between the accessible state distributions of the current and expert policies. Building on these concepts, they present the ASOR Algorithm, which integrates these ideas as a reward augmentation module. Designed as a plug-and-play component, ASOR can be applied to both online and offline RL approaches. \n\nThe method is evaluated across diverse benchmarks, including MuJoCo, MetaDrive, and a large-scale Fall Guys-like environment. Combining theoretical insights with extensive empirical validation, the paper demonstrates that regularizing policies based on accessible states enhances robustness across dynamic shifts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and insightful comments. Responses are provided as follows. The linked file is available [here](https://anonymous.4open.science/api/repo/ICML_ASOR_Rebuttal-01CB/file/rebuttal_append_file.pdf).\n\n**Q1: Robustness to hyperparams**\n\nA: Thanks for mentioning the hyperparameter table which is indeed missing in the original paper. We refer the reviewer to Tab. 1 and Tab. 2 in the linked file for detailed hyperparameters. In offline RL, we tried two values of $\\lambda$: 0.1 and 0.3. For datasets with higher level of optimality, $\\lambda=0.1$ is preferred so that less policy regularization is exerted. $\\lambda=0.3$ is preferred on datasets with higher randomness to add more policy regularization based on optimal accessible state distributions. $\\rho_1$ and $\\rho_2$ can be similarly chosen based on offline data optimality with possible values (0.5, 0.5) and (0.3, 0.3). In medium-expert datasets, states are more likely to be sampled from the optimal accessible state distribution, so higher values $\\rho_1$, $\\rho_2$ are chosen. In online RL, $\\lambda$ is set to 0.03 in MuJoCo environments and 0.1 in the Fall Guys-like environment, mainly to fit the reward scale of different RL environments. $\\rho_1$, $\\rho_2$ are kept to 0.5 in online RL experiments.\n\nIn Tab. 2, a fixed $\\lambda=0.1$ has poor performance compared with MAPLE. This is because the augmented reward adds additional variance to environment reward. In some datasets its scale may not be large enough to exert reasonable policy regularization, so the reward variance dominates the training process and leads to a performance drop. Meanwhile, random partition denotes that the real data and fake data to train the GAN-like discriminator in Algorithm 1 are randomly chosen from the training batch. Its poor performance compared with MAPLE demonstrates the importance of constructing discriminator training data according to the accessible state distributions.\n\nIn Tab. 4 of the linked file, we conduct additional hyperparameter tuning where $\\lambda$ ranges from 0.1 to 0.4 and $\\rho_1,\\rho_2$ equal to 0.3, 0.5, and 0.7. The results demonstrate that ASOR's performance is relatively stable with $\\lambda=0.2, 0.3, 0.4$. $\\rho_1,\\rho_2=0.7$ may include too many states in the real dataset for discriminator training, so the performance drops accordingly.\n\n**Q2: Marginal improvements in HalfCheetah**\n\nA: We agree with the reviewer that ASOR is most beneficial in environments where accessibility varies, such as the fall-guys like game environment. As discussed in Sec. 4.2 (Lines 380~406), the agent will not fall over in the HalfCheetah environment, so the state accessibility will be more likely to remain unchanged under dynamics shift. This undermines the effectiveness of ASOR and leads to the marginal performance improvements.\n\n**Q3: More training steps in MetaDrive**\n\nA: We clip at 1M episodes to make training steps consistent with other environments. We will add the full training log in the revision.\n\n**Q4: The scale of fall guys-like game environment**\n\nA: The fall guys-like environment has a 1065-dimensional state space containing terrain, map, target, goal, agent, and destination information. Although there are no visual inputs, the state space contains diversified information that is challenging to integrate. Meanwhile, due to the highly dynamic environment and complex interactions between the agent and the environment components, a large number of environment steps is needed to train a decent policy. That is why we call the fall guys-like environment \"large scale\". The scale of the action space will not influence the effectiveness of ASOR, as the proposed policy regularization objective relies solely on the state distribution and do not take the action or policy distribution into account.\n \n**Q5: Restrictiveness of the assumption**\n\nA: As discussed after Def. 3.3 (Lines 208~219), the assumption is weaker than those in existing methods and less restrictive. It requires state $s'$ can be accessed from $s$ after dynamics shift, as long as it is accessible from $s$ in the original dynamics. It does not require a specific policy to do so and does not limit the number of intermediate states. Such assumption still holds in two of the mentioned extreme cases in that the state accessibility does not change.\n\n**Q6: Different optimal policy**\n\nA: According to Eq. (2,3), regularized policy optimization achieves performance lower-bounds without assumptions on how close the optimal policies are. Intuitively, the regularization provides a reasonable starting point for policies to explore more efficiently, but the starting point is not necessarily optimal. The walker2d environment  are known to have many different near-optimal policies, while our ASOR algorithm can still achieve the best performance (Fig. 2).\n\n**Q7: Stability of GAN training**\n\nA: Due to the limited word count, we refer the reviewer to Q4 in the rebuttal to Reviewer UQdV.", "claims_and_evidence": "Most of the paper’s claims are supported by a mix of rigorous theoretical analysis and extensive empirical experiments.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally well-aligned with the problem of cross-dynamics reinforcement learning. The core idea of focusing on globally accessible states addresses a real issue in cross-dynamics scenarios. The method directly targets the shortcomings of traditional IfO approaches by regularizing policies to only mimic these reliable states. The theoretical framework (using F-distance with both JS divergence and neural network distance) is solid, even though it relies on certain assumptions that might limit its application in highly adversarial settings.\n\nFor the evaluation criteria, the paper evaluates its methods on a variety of benchmark datasets and environments, including MuJoCo, Minigrid, MetaDrive, and a large-scale Fall Guys-like game. These benchmarks cover a range of tasks (both online and offline) and dynamics shifts, providing a comprehensive test bed that is well-suited to assess the proposed method’s performance.", "theoretical_claims": "I examined several of the theoretical proofs provided in the paper, particularly those supporting the performance lower bounds (Theorem 3.5 and Theorem 3.7).", "experimental_designs_or_analyses": "I examined several aspects of the experimental designs and analyses presented in the paper. Overall, while the chosen benchmarks and metrics are appropriate for cross-dynamics RL, some issues remain on certain experiments. Addressing these problems would strengthen the evaluation of the results:\n- Some experimental details, such as the exact procedures for constructing state partitions (DP and DQ) and the calibration of pseudo-counts, are described briefly. This might make it harder for a reader to reproduce the results or fully understand the sensitivity of the approach to these choices.\n- Although the authors have provided ablation studies in Table 2, I remain unconvinced of ASOR's robustness to the reward coefficient and state partitioning hyperparameters. First, in the first and third columns, the average result (0.34) is lower than that of the original MAPLE (0.36). The authors should provide more detailed results to illustrate the sensitivity of a broader range of hyperparameter choices. Second, in the final column, it is unclear whether the results across different domains were obtained using the same set of hyperparameters. If not, how were the hyperparameters tuned for each domain? Lastly, perhaps I overlooked it, but I could not find the exact values of the final chosen hyperparameters.\n- For the online RL experiments in Figure 2, the authors are encouraged to follow the experimental setups in the SRPO paper more closely, ensuring alignment in iteration numbers and experimental domains. This would help readers better assess whether the proposed method is fairly compared with SRPO. Additionally, in HalfCheetah, ASOR shows only marginal improvements over the baselines. Is there a fundamental reason for its reduced effectiveness in this setting? If the limitation stems from state accessibility not changing significantly in HalfCheetah, this would suggest that ASOR is most beneficial in environments where accessibility varies.\n- Regarding the experiments in MetaDrive, the training process appears to be unstable and not fully converged for the compared methods. Could the authors include results with additional training steps for better convergence? \n- I find it unclear why the authors describe the Fall Guys-Like Game experiments as large-scale, given that no visual inputs are used and the action space appears to be small. It remains uncertain whether the proposed method would be effective in high-dimensional state and action spaces, such as humanoid or environments with visual observations. Could the authors clarify these points?", "relation_to_broader_scientific_literature": "The key contributions of the paper relate to multiple areas within reinforcement learning (RL), particularly in cross-dynamics policy transfer, imitation learning from observation (IfO), and policy regularization. For cross-dynamics RL, it extends SRPO by redefining state accessibility and improving transfer in dynamic environments.", "essential_references_not_discussed": "The paper builds on SRPO (Xue et al., 2023a) but does not cite works on meta-learning for RL adaptation, such as MAML (Finn et al., 2017), which are highly relevant in learning adaptable policies across diverse dynamics.", "other_strengths_and_weaknesses": "Strengths:\n- The paper identifies a critical limitation in many IfO approaches—the assumption of identical expert state distributions across dynamics—which is often violated in real-world scenarios. This insight is both timely and relevant for cross-dynamics RL.\n- The derivation of performance bounds using two different instantiations (JS divergence and network distance) provides solid theoretical grounding.\n- ASOR is designed as a modular add-on that can be integrated with existing state-of-the-art RL algorithms. Its practical implementation via a GAN-like objective for reward augmentation makes it appealing for both online and offline settings.\n\nOther weaknesses:\n- While the paper is well-structured overall, some sections (especially those covering theoretical analyses and the accessible states) could be improved with additional intuitive explanations or visual aids. This would help the readers better understand the paper.", "other_comments_or_suggestions": "For the methodology part, the construction of DP and DQ datasets (used for state accessibility filtering) is described briefly but is critical to the method. Adding more details on how the partitioning is implemented and its computational overhead would strengthen reproducibility. Besides, the construction of DP and DQ datasets (used for state accessibility filtering) is described briefly but is critical to the method. Adding more details on how the partitioning is implemented and its computational overhead would strengthen reproducibility.", "questions_for_authors": "1. In Theorem 3.5 and Theorem 3.7, the performance lower bounds rely on the assumption that the HiP-MDP’s MDPs are M-Rs accessible from each other. How restrictive is this assumption in practice? If it fails in many real-world cases (e.g., highly stochastic environments or POMDPs with high-dimensional observations such as visual inputs), then the practical significance of the theoretical guarantees would be weakened.\n2. In some cases, the globally accessible states between the current environment and the expert trajectories may not share the same optimal policy. How do the authors account for this situation? Could ASOR have a negative effect in such cases?\n3. How stable is the discriminator training for F-distance estimation? Are there cases where the optimization struggles (e.g., mode collapse, vanishing gradients)? Since GAN training can be unstable, an analysis of failure cases or stability techniques (e.g., spectral normalization, gradient penalties) would be valuable. If the discriminator fails in some environments, addressing this limitation (or providing mitigation strategies) would be important for practical applications.\n4. Please see our questions in the \"Experimental Designs Or Analyses\" section above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b91tHmbnJi", "reviewer": "Reviewer_UQdV", "summary": "This paper proposes Accessible State Oriented Policy Regularization (ASOR), a reward shaping technique for offline and online,  off-policy RL with dynamic shift. The work is inspired by the dynamic-agnostic state distribution matching in Imitation from Observation (IfO), but points out that naively imitating expert states could be suboptimal with changed dynamics. Thus, the paper focuses on expert states that are always accessible throughout dynamic changes (\"globally accessible\"), and tries to maximize reward in RL with a constraint of state occupancy difference ($\\mathcal{F}$-distance) between state occupancy on all states and state occupancy on globally accessible states. With Lagrangian derivation, the final implementation is to train a GAN discriminating states with high values or high proxy visitation counts (by RND) with other states, and add a reward shaping term from the GAN onto the original reward. On several offline RL and online RL environments, the proposed method is proved to outperform baselines.\n\n## Update After Rebuttal\n\nI have updated my score accordingly during the author-discussion period; no further update.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and insightful comments. Responses are provided as follows. The linked file is available [here](https://anonymous.4open.science/api/repo/ICML_ASOR_Rebuttal-01CB/file/rebuttal_append_file.pdf).\n\n**Q1: Possibly empty accessible state set**\n\nA: Tasks with no globally accessible states are indeed extreme cases where ASOR may degrade into the base RL algorithm. But two practical factors can help alleviate such concern.\n- One may regard the state dimension that remains distinct across dynamics as hidden environment parameter and do not include such information when identifying globally accessible states. For example, the robot height in the mentioned example can be excluded in the state space. States that are visited by expert policies with different robot height, e.g., smooth walking with moderate speed, can be regarded as globally accessible and preferred when training new policies.\n- In the practical algorithm procedure of ASOR, a relative accessibility detection mechanism is applied. States that are most likely to be accessible in a training batch are regarded as preferable (Line 7 in Algorithm 1). We can rely on the discriminator network to automatically find appropriate states to assign higher rewards (Line 8 in Algorithm 1).\n\nMeanwhile, the mentioned robotics scenario with different robot leg conditions can also be challenging for base RL algorithms which ASOR is built on, such as cross-domain RL methods. A considerable amount of training data may be needed to train decent policies in such scenarios.\n\n**Q2: Overlooked IfO papers**\n\nA: Thanks for bringing these papers into our attention. We will add discussions in the related work of the revised paper. We also agree with the reviewer that we should give IfO papers [1,2] enough credit for also considering state distribution mismatch. IfO methods are helpful when enough near-expert trajectories or unlabeled data are available and have better training efficiency. The advantage of ASOR lies in that its reward augmentation mechanism can be effectively integrated with both offline and online RL methods and adds minimal changes to their original training pipeline.\n\n**Q3: Additional baselines**\n\nA: Thanks for introducing alternative ways of constructing baselines. We refer the reviewer to Tab. 3 of the linked page, where comparative analysis on SOIL with 10% data and IQL is included. SOIL indeed benefits from the 10% data fix and shows increased performance, while IQL has similar performance with CQL. Both approaches cannot outperform MAPLE+ASOR. This can be because the offline datasets do not have enough state coverage for SOIL to imitate. Meanwhile, IQL and CQL still focus on state-action joint distributions in the offline dataset. Such distribution can be unreliable under dynamics shift since given the same state, the optimal action may be different in different dynamics. We are unable to reimplement the mentioned DICE-related methods due to the limited rebuttal time and will add them in the revision. \n\n\n**Q4: Potentially fragile training stability**\n\nA: Both the RND module and the GAN module are essentially supervised learning on given datasets. Their training will be more stable than the actor-critic module that involves RL loss. The red loss curve in Fig. 3 (right) also indicates that the RND loss and the GAN discriminator loss drop smoothly, indicating a relatively stable training process. One might relate the GAN module to the unstable training of GANs. But in fact, the GAN module only utilizes a GAN-like objective function. Its training procedure is different from GAN in that the data to train the discriminator is directly constructed from the replay buffer, instead of being adversarially generated.\n\n\n**Q5: Contradicting augmented reward**\n\nA: Fig. 3 (left) demonstrates the augmented reward of a single step, while in Fig. 3 (right) we show rewards averaged in a batch and multiplied by the coefficient. The extreme values of the augmented reward only exist when the walker agent is about to fall down near the end of the trajectory, so they only make up a small fraction of a training batch. The average augmented reward will therefore remain relatively stable during training.\n\n**Q6: Repeated $\\lambda$**\n\nA: Thanks for pointing out this issue. We will change the Lipschitz coefficient to $\\mu$ in the revision. The Lagrange multiplier $\\lambda$ is the the same with the coefficient for the augmented reward according to Eq. (5), so we remain the same notation. For the values of $\\lambda$ in different experiments, we refer the reviewer to Tab.1 of the linked pdf page.\n\n**Q7: Number of GPUs**\n\nA: We only use one V100 GPU for centralized training in Ray. Distributed environment rollouts are carried out with ~300 CPU threads.", "claims_and_evidence": "Yes in general. This paper made several claims: empirically, the paper claims that matching expert states similar to IfO is a good way to learn cross-dynamics RL, but naively matching expert states without considering accessibility is bad; theoretically, the paper claims that one can convert the problem into a constrained optimization with $\\mathcal{F}$-distance limited between state occupancies of the policy and the policy on globally accessible states, and the framework has several theoretical guarantees. With Lagrangian multiplier, the objective can be turned into (as a surrogate) an unconstrained one and optimized with a GAN. I agree with most of the claims, except that:\n\n1. not all IfOs are ignorant to dynamic changes;\n 2. there are cases in cross-dynamics RL where globally accessible states are empty sets which this method does not seem to apply. \n\nSee \"methods and evluation criteria\" for details.", "methods_and_evaluation_criteria": "Yes. Overall, the proposed method make sense for the motivation proposed by the paper that one should follow states that are successful and visited by different dynamics. However, there is a concern remaining: The globally accessible state might be an empty set in cross-dynamics RL. This is common in cross-embodiment learning; for example, we want to learn a walking policy for robots with either normal legs or crippled legs, and with different heights. in this case, as the robots may never perfectly achieve the same status, the globally accessible state will become an empty set. Such scenario is also considered in some IfO papers which this paper overlooked [1, 2], and thus the statement \"only HIDIL considered state distribution mismatch across different dynamics (in IfO) ...\" does not hold.\n\n**References**\n\n[1] Y. J. Ma et al. Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching. In ICML, 2022.\n\n[2] K. Yan et al. A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories. In NeurIPS, 2023.", "theoretical_claims": "There are two theoretical claims in this paper, which are about finite and infinite sample analysis of the proposed method. I briefly checked both of them and they look sensible and correct to me.", "experimental_designs_or_analyses": "Yes, and I found the results to be sufficient to prove the effectiveness of the proposed method. The empirical success and detailed analysis on visual environments Is particularly convincing. However:\n\n1. Even if many methods are tested, I still feel some IfO methods are missing; for example, the IfO papers I mentioned in \"Methods and Evaluation Criteria\" also considers cross-dynamics generalization. \n\n2. The authors claim in Sec. 4.1 that \"IfO approaches have the worst performance because they ignore the reward information ...\" . However, there is an easy fix (which is widely used in decision transformer papers [1] as \"10%BC\") to make use of the reward labels: one can simply select the few trajectories with the highest reward as \"expert trajectory\", and the rest as \"auxiliary unlabeled data\" (or simply discard them). There are many IfO papers that can learn from a few expert trajectories with auxiliary unlabeled data, such as SMODICE [2], LobsDICE [3], TAILO [4], MAHALO [5] (which can also do offline RL), etc.\n\n3. Some baselines are a bit out-of-date. For example, CQL is usually considered to be worse than IQL [6], which is a more recognized offline RL algorithm. There are also other newer methods such as XQL [7].\n\n**References**\n\n[1] L. Chen et al. Decision Transformer: Reinforcement Learning via Sequence Modeling. In NeurIPS, 2021.\n\n[2] Y. J. Ma et al. Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching. In ICML, 2023.\n\n[3] G-H Kim et al. LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation. in NeuriPS, 2023.\n\n[4] K. Yan et al. A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories. In NeurIPS, 2023.\n\n[5] A. Li et al. MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations. In ICML, 2023.\n\n[6] I. Kostrikov et al. Offline Reinforcement Learning with Implicit Q-Learning. In ICLR, 2022.\n\n[7] D. Garg et al. Extreme Q-Learning: MaxEnt RL Without Entropy. In ICLR, 2023.", "relation_to_broader_scientific_literature": "This paper is beneficial to the Reinforcement Learning (RL) / Imitation Learning (IL) community, and potentially helpful for the robotics and computer vision community (as it contains visual environments). It does not affect much to the scientific literature beyond the communities above.", "essential_references_not_discussed": "As mentioned in the \"Methods and Evaluation Criteria\" section, I feel some IfO works are missing, and the authors should also discuss cross-embodiment works where the globally accessible states are empty sets.", "other_strengths_and_weaknesses": "**Other Strengths**\n\n1. The paper is overall well-written and easy to follow; the motivation from IfO that \"one should focus on expert states, but not all expert states\" seems natural. \n\n**Other Weaknesses**\n\n1. The method seems to have many moving parts. For this method, one need to train a RND for proxy of visitation frequency, a GAN, a context encoder (from ESCP), an actor and a critic together. The training stability of this method could potentially be fragile. The authors mention this in Fig. 3, but the augmented reward seems to be very close to 0, which contradicts with the left hand side of Fig. 3 where augmented reward has a very large absolute value. Can the authors explain this?", "other_comments_or_suggestions": "1. the title of Sec. 2, \"Backgroud\" -> \"Background\";\n\n2. In this paper, $\\lambda$ is used as the Lagrange multiplier, regularization coefficient, and Lipschitz coefficient in the same time. This is confusing, as i cannot find the value of $\\lambda$ regularization coefficient in this paper (there is an ablation in Tab. 2, but I am unable to find the the value adopted in the main results). The $\\lambda$ in Tab. 4 seems to refer to the Lipschitz coefficient.\n\n3. I would suggest the authors to also provide the procedure of ASOR+MAPLE in the appendix besides ASOR+ESCP.", "questions_for_authors": "I have a question: how many GPUs does the training use? The authors claim that \"The training was conducted using NVIDIA TESLA V100 GPUs and takes around 20 hours to train 6M steps\", but did not specify the number of GPUs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors present an approach to learn from environments with varying dynamics by combining reward maximization with imitation from observations by using f-divergence between state distributions for policy optimization. This allows them to only consider globally accessible states, with the assumption that different dynamics across environments can render some states inaccessible. The theoretical framework is a strong contribution, showing how different f-divergences lead to different analyses. The empirical contributions are also strong, evaluating on a variety of benchmark datasets and environments, including MuJoCo, Minigrid, MetaDrive, and a large-scale Fall Guys-like game. These benchmarks cover a range of tasks (both online and offline) and dynamics shifts, providing a comprehensive test bed that is well-suited to assess the proposed method’s performance.
Decision: Accept (spotlight poster)