SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we analyze some properties of MDPs with different dynamics and provide theoretical justifications for the SRPO algorithm in Sec. 3. The notations are introduced in Sec. 2.1 and proofs can be found in Appendix A.4. We first show in Thm. 4.2 that the performance of a policy can be lower-bounded when its stationary state distribution is close to a certain optimal state distribution. In accordance with the intuition in Sec. 3.1, it is also demonstrated in Thm. 4.3 that optimal policies can induce the same stationary state distribution in different dynamics under mild assumptions. We start the analysis with the definition of homomorphous MDPs.
Definition 4.1 (homomorphous MDPs). In an HiP-MDP  $(\mathcal{S},\mathcal{A},\Theta ,T,r,\gamma ,\rho_0)$ , consider hidden parameters  $\theta_{1},\theta_{2}\in \Theta$ . Let  $T_{i}(s^{\prime}|s,a) = T(s^{\prime}|s,a,\theta_{i}),\forall (s,a,s^{\prime})\in \mathcal{S}\times \mathcal{A}\times \mathcal{S},i = 1,2$ . If  $\sum_{a\in \mathcal{A}}T_{1}(s^{\prime}|s,a) > 0\Leftrightarrow \sum_{a\in \mathcal{A}}T_{2}(s^{\prime}|s,a) > 0$  for all  $s,s^{\prime}\in \mathcal{S}$ , MDPs  $(\mathcal{S},\mathcal{A},T_1,r,\gamma ,\rho_0)$  and
$(S, \mathcal{A}, T_2, r, \gamma, \rho_0)$  are referred to as homomorphous MDPs.
In this definition,  $\sum_{a\in \mathcal{A}}T(s'|s,a) > 0$  means state  $s^\prime$  can be reached from  $s$ , so the equivalence of non-zero transition probabilities refers to the same reachability from  $s$  to  $s^{\prime}$ . Such condition holds in a wide range of MDPs differing only in environment parameters. For example, pendulums with different lengths can all reach the upright state from an off-center state, with longer pendulums exerting a larger force. Apart from the homomorphous property, we also require the reward and dynamics functions of MDPs to have Lipschitz properties. We assume reward function  $r(s,a,s^{\prime})$  w.r.t. the action  $a$  is  $\lambda_{1}$ -Lipschitz and the dynamics function  $T(s,a)$  w.r.t. the action  $a$  is  $\lambda_{2}$ -inverse Lipschitz. Discussions on these Lipschitz properties can be found in Appendix A.5.
With these preliminaries, we first analyze the discrepancy of accumulated returns of two policies with similar stationary state distributions. The analysis is related to our SRPO algorithm in that the state regularized policy optimization formulation in Eq. (2) also constrains the learning policy to have a similar stationary state distribution with the optimal policy. Specifically, we derive a theorem as follows.
Theorem 4.2. Consider two homomorphous MDPs with dynamics  $T$  and  $T'$ . If  $T' \in (T, \varepsilon_m)$ , for all learning policy  $\hat{\pi}$  such that  $D_{\mathrm{KL}}(d_T^{\hat{\pi}}(\cdot) \| d_{T'}^*(\cdot)) \leqslant \varepsilon_s$ , we have
$$
\eta_ {T} (\hat {\pi}) \geqslant \eta_ {T} \left(\pi_ {T} ^ {*}\right) - \frac {\lambda_ {1} \lambda_ {2} \varepsilon_ {m} + 2 \lambda_ {1} + \sqrt {2} R _ {\max } \sqrt {\varepsilon_ {s}}}{1 - \gamma}. \tag {7}
$$
The theorem implies that if a policy  $\hat{\pi}$  has a similar stationary state distribution with the optimal policy in one MDP  $M$ ,  $\hat{\pi}$  will have a lower-bound performance guarantee in all MDPs that are homomorphous with the MDP  $M$ . Therefore, the learning policy can benefit from the state regularized policy optimization in Sec. 3.2.
More specifically, Eq. (7) shows that the gap in accumulated return of  $\hat{\pi}$  and  $\pi_T^*$  is related to the dynamics shift  $\varepsilon_{m}$ , the KL-Divergence of two stationary state distributions  $\varepsilon_{s}$ , and the effective planning horizon  $\frac{1}{1 - \gamma}$ . With respect to the dynamics shift  $\varepsilon_{m}$ , it is related to a "uniform" constraint on the dynamics  $T'$ . We further show in Appendix A.4 that constraining the dynamics shift on a certain state-action pair is enough to derive Eq. (7). Unlike the dynamics shift  $\varepsilon_{m}$  that is determined by a pre-defined RL task, the discrepancy between stationary state distributions  $\varepsilon_{s}$  is determined by the learning policy  $\hat{\pi}$  and can be optimized during training to obtain a better performance lower-bound. We also discuss in Appendix A.5 how tight Eq. (7) is in terms of the effective planning horizon  $\frac{1}{1 - \gamma}$ , compared with some similar performance bounds.
With an additional assumption on the action gap  $\Delta$  (defined in Sec. 2.1), we further demonstrate that the optimal policy of two homomorphous MDPs can have the same stationary state distribution, which verifies the intuition in Sec. 3.1.
Theorem 4.3. Consider two homomorphous MDPs with dynamics  $T$  and  $T'$ . If  $T' \in (T, \varepsilon_m)$  and the action gap  $\Delta$  follows  $\Delta > \frac{(2 - \gamma)\lambda_1\lambda_2\varepsilon_m}{1 - \gamma}$ , for all  $s \in S$  we have  $d_T^*(s) = d_{T'}^*(s)$ .
The assumption is mild and holds in many scenarios. For example, in autonomous driving it can be very dangerous to deviate from the optimal policy. Such suboptimal actions have low rewards, leading to a large action gap  $\Delta$ . In recommendation tasks we are hardly concerned with what items we recommend (the action), as long as the recommendation outcome (the state), i.e., the users' experiences are good enough, leading to a small  $\lambda_{1}$ . The condition of large enough action gap holds in these situations.
In this section, we conduct experiments to investigate the following questions: (1) Can SRPO leverage data with distribution shift and outperform current SOTA algorithms in the setting of HiP-MDP, in both online and offline RL? (2) How does each component of SRPO (e.g., use state regularization rather than behavior regularization) contribute to SRPO's performance? To answer question (1), we use the MuJoCo simulator [41] and generate environments with different transition functions. We train the CaDM+SRPO and the MAPLE+SRPO algorithm proposed in Sec. 3.4 and make comparative analysis with baseline algorithms. To answer question (2), we do ablation studies to examine the role of different modules in SRPO. We also examine how the discriminator  $D_{\delta}$  works in complex environments and the effect of regularizing with state distributions in different performance levels.
We alter the simulator gravity to generate different dynamics in online experiments. Possible values of gravity are  $\{1.0\}$ ,  $\{0.7, 1.0, 1.3\}$ , and  $\{0.4, 0.7, 1.0, 1.3, 1.6\}$  in experiments with 1, 3, and 5 kinds of different dynamics, respectively. When the simulator resets, the gravity is uniformly sampled from the set of all possible values. The number of training steps is in proportion to the number of environment parameters. Therefore, the agent has access to the same amount of training data on a
Figure 4: Results of online experiments on MuJoCo tasks. The comparison is made between CaDM+SRPO and baseline algorithms PPO, CaDM. Our CaDM+SRPO algorithm has the best overall performance in experiments with 3 and 5 different environment dynamics. The curves show the average return on 4 random seeds and the shadow areas reflect the standard deviation.
Table 1: Results of offline experiments on MuJoCo tasks. Numbers are the normalized scores according to the D4RL paper [42]. ME, M, MR and R correspond to the medium-expert, expert, medium-replay and random dataset, respectively. The evaluation is done on policies at the last iteration of training, averaged over four random seeds. The number after  $\pm$  is the standard deviation. Our proposed MAPLE+SRPO algorithm has the best performance in 8 of 12 tasks and the highest overall performance.
CQL (Single Env)	GAIL	CQL	MOPO	MAPLE	MAPLE +DARA	MAPLE +SRPO(Ours)
Walker2d-ME	1.11	0.21±0.03	1.03±0.10	0.25±0.18	0.55±0.21	0.80±0.02	0.66±0.08
Walker2d-M	0.79	0.15±0.06	0.78±0.01	0.23±0.34	0.82±0.01	0.83±0.03	0.84±0.03
Walker2d-MR	0.27	0.00±0.00	0.07±0.00	0.00±0.00	0.16±0.02	0.17±0.01	0.17±0.02
Walker2d-R	0.07	0.00±0.00	0.03±0.01	0.00±0.00	0.22±0.00	0.22±0.00	0.22±0.00
Hopper-ME	0.98	0.04±0.01	0.32±0.14	0.01±0.00	0.96±0.14	0.96±0.06	0.98±0.02
Hopper-M	0.58	0.00±0.00	0.57±0.16	0.01±0.00	0.78±0.28	0.40±0.05	1.03±0.09
Hopper-MR	0.46	0.00±0.00	0.14±0.02	0.01±0.01	0.91±0.11	1.02±0.01	1.02±0.01
Hopper-R	0.11	0.00±0.00	0.11±0.00	0.01±0.00	0.13±0.00	0.13±0.01	0.32±0.02
HalfCheetah-ME	0.62	0.36±0.06	0.03±0.04	-0.03±0.00	0.50±0.06	0.50±0.00	0.63±0.01
HalfCheetah-M	0.44	0.25±0.02	0.43±0.03	0.38±0.28	0.62±0.01	0.67±0.03	0.63±0.01
HalfCheetah-MR	0.46	0.18±0.11	0.46±0.00	-0.03±0.00	0.52±0.00	0.53±0.01	0.55±0.00
HalfCheetah-R	0.35	0.14±0.02	0.01±0.02	-0.03±0.00	0.22±0.03	0.21±0.00	0.24±0.01
Average	0.52	0.11	0.33	0.068	0.53	0.54	0.61
certain value of simulator gravity. We also consider the shift of medium density and body mass in offline experiments to show SRPO's robustness to different forms of dynamics shift.
To perform comparative analysis, we choose CaDM [9] and PPO [7] as baseline algorithms in online experiments. In offline experiments, DARA [5] also exploits large amount of data with dynamics shift. Its algorithm relies on Importance Sampling and will be used as a baseline method. Apart from that, we choose MOPO [40], MAPLE [12] and CQL [26] as baseline methods. More information on the setup of experiments is shown in Appendix B.1.
Online Experiments The results of online experiments are shown in Fig. 4. With the context encoder and conditional policy, CaDM is able to outperform PPO in all environments. However, it fails to take advantage of the increase in the amount of data with dynamics shift. Its performance with 5 different dynamics is lower than that with 3 dynamics. In contrast, our proposed SRPO
Table 2: Results of ablation studies in offline experiments.
MAPLE +SRPO	Behavior Regularizing	Random Partition	Fixed λ = 0.3
Walker2d	0.47	0.45	0.39	0.40
Hopper	0.83	0.68	0.56	0.79
HalfCheetah	0.51	0.50	0.45	0.40
Average	0.61	0.54	0.47	0.53
Table 3: Performance comparison of differently regularized policies.
Original Hopper Env	10x Density
Random	121.3	44.15
Medium	2178	913.3
Expert	3819	3748
algorithm leads to better performance on top of CaDM in accordance with more training data. It significantly outperforms the original CaDM algorithm in environments with 5 different dynamics. The performance comparison in the Pendulum environment is also in accordance with the motivating example in Sec. 3.1. More results of online experiments are shown in Appendix B.2.
Offline Experiments The results of offline experiments are shown in Tab. 1. The column of "CQL Single" refers to the evaluation score in the CQL [26] paper, where the policy is with data from a single static environment. Without the mechanism of context-based encoders, GAIL [43], CQL and MOPO [40] cannot handle data with distribution shift and show a performance drop. MAPLE [12] and MAPLE+DARA [5] only achieve marginal performance improvement with respect to CQL single. On the other hand, MAPLE+SRPO shows significant performance improvement over CQL single, which means that SRPO can efficiently leverage the additional data with dynamics shift to facilitate policy training. The MAPLE+SRPO algorithm also has a  $15\%$  higher evaluation score than MAPLE, achieving the best performance in 8 out of 12 tasks. Apart from MAPLE, the meta-RL algorithm PEARL [44] also has an context encoder for fast adaptation. We compare PEARL with PEARL+SRPO and leave the results in Appendix. B.3.
Ablations We conduct ablation studies in offline environments to analyze the role of each algorithm component in SRPO. The results are shown in Tab. 2. We first investigate the outcome of regularizing with state-action distribution rather than state distribution in Eq. (2). The resulting policy has a lower evaluation score on average than policies trained with the original SRPO algorithm in all environments. This is because environments with different dynamics do not have a similar optimal policy. The action distribution in the mixed dataset can be misleading when training new policies. According to Sec. 3.3, SRPO trains a classifier to discriminate states with higher values from lower values. We also train another classifier discriminating a random binary partition of states. Ablation results show a huge performance drop, which verifies the effectiveness of the classification-based surrogate mechanism. We also evaluate MAPLE+SRPO with a fixed value of the hyperparameter  $\lambda$ .  $\lambda = 0.1$  is more suitable for Walker2d and HalfCheetah environments, while in the Hopper environment  $\lambda = 0.3$  is better. This is in accordance with previous analysis that Hopper agents can benefit more from regularizing with the stationary state distribution.
Effectiveness of Discriminators We first train a discriminator  $D_{\delta}$  according to Alg. 1. Then a set of states is sampled from the D4RL [42] dataset and classified into two sets according to the output of  $D_{\delta}$ . The average values of the two sets of states are compared in Fig. 5. As shown in the figure, states with higher  $D_{\delta}$  outputs also have higher values in all three environments. It means that the trained discriminator  $D_{\delta}$  can successfully identify states with high values from those with low values. Therefore, its output can be a good surrogate for the density ratio  $\frac{\zeta(s)}{d_{\pi}(s)}$  in Sec. 3.
Figure 5: Comparison of values on states with high and low output of the discriminator  $D$ .
Effectiveness of the Regularization We also study the effect of policy regularization with different performance levels of stationary state distributions. Random, medium and expert policies in the original Hopper environment are
used to estimate the stationary state distributions, which regularize the learning policies in a new environment with different dynamics. The results are shown in Tab. 3, where the expert policy is the most effective in regularizing. This verifies the practice in Sec. 3.2 and the theoretical analysis.
In this work, we focus on the problem of leveraging data with dynamics shift to efficiently train RL agents. Based on the intuition that optimal policies can lead to similar stationary state distributions, we give a constrained optimization formulation that regards the state distribution as a regularizer. After discussions on a sample-based surrogate, we propose the SRPO algorithm which can be an add-on module to context-based algorithms and improve their sample efficiency. The resulting CaDM+SRPO and MAPLE+SRPO algorithms show superior performance when learning on data sampled from environments with different dynamics. Theoretical analyses are also given to analyze some properties of MDPs with different dynamics. They provide justifications for the intuition of the dynamics-invariant state distribution, as well as the constrained policy optimization formulation.
Limitations and Future work The theoretical analyses of this work require the assumption of homomorphous MDPs, i.e., the same state reachability in different MDPs. It would be interesting to discuss whether similar conclusions on the stationary state distribution can be derived without such assumption.
Acknowledgements We thank Wanqi Xue and Yanchen Deng for helpful discussions. This research is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative and Ministry of Education, Singapore, under its Academic Research Fund Tier 1 (RG13/22). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
[2] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):484-489, 2016.
[3] Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, and Bo An. PrefRec: Preference-based recommender systems for reinforcing long-term user engagement. CoRR, abs/2212.02779, 2022.
[4] Zhenghai Xue, Qingpeng Cai, Tianyou Zuo, Bin Yang, Lantao Hu, Peng Jiang, and Bo An. AdaRec: Adaptive sequential recommendation for reinforcing long-term user engagement. CoRR, abs/2310.03984, 2023.
[5] Jinxin Liu, Hongyin Zhang, and Donglin Wang. DARA: Dynamics-aware reward augmentation in offline reinforcement learning. In ICLR, 2022.
[6] Fan-Ming Luo, Shengyi Jiang, Yang Yu, Zongzhang Zhang, and Yi-Feng Zhang. Adapt to environment sudden changes by learning a context sensitive policy. In AAAI, 2022.
[7] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
[8] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.
[29] Qiang Liu, Lihong Li, and Ziyang Tang and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In NeurIPS, 2018.
[30] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In NeurIPS, 2019.
[31] Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. In ICLR, 2020.
[32] Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning. In ICLR, 2020.
[33] Tianwei Ni, Harshit S. Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-IRL: Inverse reinforcement learning via state marginal matching. In CoRL, 2020.
[34] Shentao Yang, Yihao Feng, Shujian Zhang, and Mingyuan Zhou. Regularizing a model-based policy stationary distribution to stabilize offline reinforcement learning. In ICML, 2022.
[35] Paul F. Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model. CoRR, abs/1610.03518, 2016.
[36] Shengyi Jiang, Jing-Cheng Pang, and Yang Yu. Offline imitation learning with a misspecified simulator. In NeurIPS, 2020.
[37] Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathematical statistics, 33(3):1065-1076, 1962.
[38] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1406.2661, 2014.
[39] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In NIPS, 2016.
[40] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In NeurIPS, 2020.
[41] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
[42] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
[43] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS, 2016.
[44] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In ICML, 2019.